Dataset Card for "QQP_triplets"

Dataset Summary

This dataset will give anyone the opportunity to train and test models of semantic equivalence, based on actual Quora data. The data is organized as triplets (anchor, positive, negative).

Disclaimer: The team releasing Quora data did not upload the dataset to the Hub and did not write a dataset card. These steps were done by the Hugging Face team.

Supported Tasks

• Sentence Transformers training; useful for semantic search and sentence similarity.

Languages

• English.

Dataset Structure

Each example is a dictionary with three keys (query, pos, and neg) containing a list each (triplets). The first key contains an anchor sentence, the second a positive sentence, and the third a list of negative sentences.

{"query": [anchor], "pos": [positive], "neg": [negative1, negative2, ..., negativeN]}
{"query": [anchor], "pos": [positive], "neg": [negative1, negative2, ..., negativeN]}
...
{"query": [anchor], "pos": [positive], "neg": [negative1, negative2, ..., negativeN]}

This dataset is useful for training Sentence Transformers models. Refer to the following post on how to train them.

Usage Example

Install the 🤗 Datasets library with pip install datasets and load the dataset from the Hub with:

from datasets import load_dataset
dataset = load_dataset("embedding-data/QQP_triplets")

The dataset is loaded as a DatasetDict and has the format:

DatasetDict({
    train: Dataset({
        features: ['set'],
        num_rows: 101762
    })
})

Review an example i with:

dataset["train"][i]["set"]

Curation Rationale

More Information Needed

Source Data

Initial Data Collection and Normalization

More Information Needed

Who are the source language producers?

More Information Needed

Annotations

Annotation process

More Information Needed

Who are the annotators?

More Information Needed

Personal and Sensitive Information

More Information Needed

Considerations for Using the Data

Social Impact of Dataset

More Information Needed

Discussion of Biases

More Information Needed

Other Known Limitations

Here are a few important things to keep in mind about this dataset:

• Our original sampling method returned an imbalanced dataset with many more true examples of duplicate pairs than non-duplicates. Therefore, we supplemented the dataset with negative examples.
• One source of negative examples were pairs of “related questions” which, although pertaining to similar topics, are not truly semantically equivalent.
• The distribution of questions in the dataset should not be taken to be representative of the distribution of questions asked on Quora. This is, in part, because of the combination of sampling procedures and also due to some sanitization measures that have been applied to the final dataset (e.g., removal of questions with extremely long question details).
• The ground-truth labels contain some amount of noise: they are not guaranteed to be perfect.

Additional Information

Dataset Curators

More Information Needed

Licensing Information

More Information Needed

Citation Information

More Information Needed

Contributions

Thanks to Kornél Csernai , Nikhil Dandekar , Shankar Iyer for adding this dataset.