IndicSBERT

这是一个基于NLI数据集训练而成的MuRIL模型（google/muril-base-cased），用于十种印度主要语言。该单一模型适用于英语、印地语、马拉地语、卡纳达语、泰米尔语、泰卢固语、古吉拉特语、奥里亚语、旁遮普语、马拉雅拉姆语和孟加拉语。该模型还具有跨语言能力。作为MahaNLP项目的一部分发布： https://github.com/l3cube-pune/MarathiNLP

这里分享了一个更好的句子相似度模型（该模型的微调版本）： https://huggingface.co/l3cube-pune/indic-sentence-similarity-sbert

有关数据集、模型和基准结果的更多详细信息可以在我们的[论文]( https://arxiv.org/abs/2304.11434 )中找到。

@article{deode2023l3cube,
  title={L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT},
  author={Deode, Samruddhi and Gadre, Janhavi and Kajale, Aditi and Joshi, Ananya and Joshi, Raviraj},
  journal={arXiv preprint arXiv:2304.11434},
  year={2023}
}

monolingual Indic SBERT paper multilingual Indic SBERT paper

其他单语Indic句子BERT模型如下： Marathi SBERT Hindi SBERT Kannada SBERT Telugu SBERT Malayalam SBERT Tamil SBERT Gujarati SBERT Oriya SBERT Bengali SBERT Punjabi SBERT Indic SBERT (multilingual)

其他单语相似度模型如下： Marathi Similarity Hindi Similarity Kannada Similarity Telugu Similarity Malayalam Similarity Tamil Similarity Gujarati Similarity Oriya Similarity Bengali Similarity Punjabi Similarity Indic Similarity (multilingual)

使用（Sentence-Transformers）

如果已经安装了 sentence-transformers ，使用这个模型将变得很容易：

pip install -U sentence-transformers

然后您可以像这样使用模型：

from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('{MODEL_NAME}')
embeddings = model.encode(sentences)
print(embeddings)

使用（HuggingFace Transformers）

如果没有 sentence-transformers ，可以这样使用该模型：首先，将输入通过变换器模型，然后必须在上下文化的单词嵌入之上应用正确的汇聚操作。

from transformers import AutoTokenizer, AutoModel
import torch


#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
model = AutoModel.from_pretrained('{MODEL_NAME}')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, mean pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)