Unknown task sentence-similarity
#20
by BenTouss - opened
Hello,
I am trying to use transformers library to run this model, I am using the code snippet provided:
# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline("sentence-similarity", model="nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)
But I got the following error:
Unknown task sentence-similarity
I will need to do a little more digging but it seems that you're right there is not sentence-similarity pipeline and it's suggested to use SentenceTransformers instead: https://github.com/huggingface/transformers/issues/22923#issuecomment-1517996334
let me know if you still run into issues running this code
import torch.nn.functional as F
from sentence_transformers import SentenceTransformer
matryoshka_dim = 512
model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)
sentences = ['search_query: What is TSNE?', 'search_query: Who is Laurens van der Maaten?']
embeddings = model.encode(sentences, convert_to_tensor=True)
embeddings = F.layer_norm(embeddings, normalized_shape=(embeddings.shape[1],))
embeddings = embeddings[:, :matryoshka_dim]
embeddings = F.normalize(embeddings, p=2, dim=1)
print(embeddings)
That's working!
Thanks!
Maybe there is a way to update the snippet when you select "Use this model" in the model page..
I was not aware of the package, sentenceTransformers..
Thank you!
BenTouss changed discussion status to closed