Using Python to Call OpenAI Library to Compute Embedding

Practical foundation preparation for designing an item merging algorithm later.

Knowledge Points

Setting Environment Variables

Set $OPENAI_API_KEY as the key for v1 verification. Set $OPENAI_BASE_URL to use self-deployed OneAPI as a relay.

Calling the OpenAI Library

Import the library and create an instance.

from openai import OpenAI
client = OpenAI()

Call Embedding, specifying the string and the model to call.

client.embeddings.create(
            input=s, model="text-embedding-3-small"
            ).data[0].embedding

The output is a vector. Debug output of the vector content and len() length (manually omitted):

[-0.007801616098731756, -0.026818055659532547, ……, -0.019145511090755463, 0.008654918521642685]
1536
[Finished in 3.3s]

Switching to nomic-embed-text also works. Our nomic runs on the server side, so it takes a bit longer. Actually, there still seems to be a normalization issue here. When calling nomic directly in FastGPT, the similarity can be greater than $1$. But no normalization issue was encountered when running locally.

[-0.007801616098731756, -0.026818055659532547, ……, -0.019145511090755463, 0.008654918521642685]
1024 
[Finished in 10.1s]

Defining Functions

The pattern is:

def fuc_name(var):
    return 0    

The function body is completed within an indented block.

Specifying Parameter Types When Building Functions

Attach the type after the variable name, such as: fuc(s:str,n:int)

Checking Whether a string Is Empty

Call the len() function; not sure which library it is from. Such as: len(s)!=0

Calculating Cosine Similarity

Import the library

from sklearn.metrics.pairwise import cosine_similarity
s = cosine_similarity([a, b])

When calling the function, note that an array must be passed in, and it will generate a two-dimensional array.

Output

[[1.         0.58309876]
 [0.58309876 1.        ]]
[Finished in 4.4s]

The input length can be greater than $2$, and it will return the cosine similarity of each pair.

[[1.         0.58309876 0.69302698]
 [0.58309876 1.         0.6850905 ]
 [0.69302698 0.6850905  1.        ]]
[Finished in 5.3s]

Code

import os
os.environ["OPENAI_API_KEY"]= "sk-"
os.environ["OPENAI_BASE_URL"]= "http://x.x.x.x:3000/v1/"

from openai import OpenAI
client = OpenAI()

from sklearn.metrics.pairwise import cosine_similarity

def embedding(s:str):
	if len(s)==0:
		return
	else:
		return client.embeddings.create(
    		input=s, model="text-embedding-3-small" # nomic-embed-text text-embedding-3-small
			).data[0].embedding

a=embedding("I tend to draw fine distinctions between similar feelings (e.g., depressed and blue; annoyed and irritated).")
b=embedding("我喜欢界定两种相似的情绪（如沮丧和忧伤，烦恼和被激怒）。")
c=embedding("I like to define two similar emotions (e.g., frustration and sadness, annoyance and irritation).")

# print(a,"\n",len(a))

s = cosine_similarity([a, b, c])
print(s)

The three input strings come from the Emotion Complexity Scale, mxr612's Chinese translation of it, and Coco's back-translation of the Chinese translation.