perf(embedding): always request embedding creation as base64 #1312
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Requesting base64 encoded embeddings returns smaller body sizes, on average ~60% smaller than float32 encoded. In other words, the size of the response body containing embeddings in float32 is ~2.3x bigger than base64 encoded embedding. We always request embedding creating encoded as base64, and then decoded them to float32 based on the user's provided encoding_format parameter. Closes openai#1310
manekinekko
force-pushed
the
perf/wassim-chegham-issue-1310
branch
from
7702d54 to
270861b
Compare
1 task
RobertCraigie
requested changes
Feb 11, 2025
Comment on lines
+20
to
+22
| // Force base64 encoding for vector embeddings creation | ||
| // See https://github.com/openai/openai-node/issues/1310 | ||
| encoding_format: 'base64', |
There was a problem hiding this comment.
I don't think we want to always use base64, if the user explicitly asked for a different format we should have the exact same behaviour as we do prior to this PR which is to just let them.
| console.log(embeddingBase64Obj); | ||
| const embeddingBase64Str = embeddingBase64Obj.embedding as unknown as string; | ||
| embeddingBase64Obj.embedding = Array.from( | ||
| new Float32Array(Buffer.from(embeddingBase64Str, 'base64').buffer), |
There was a problem hiding this comment.
Buffer is a Node.js specific API, we need to use something that is available everywhere, would be great to add a generic helper function in core.ts.
| return base64Response._thenUnwrap((response) => { | ||
| if (response && response.data) { | ||
| response.data.forEach((embeddingBase64Obj) => { | ||
| console.log(embeddingBase64Obj); |
There was a problem hiding this comment.
nit
Suggested change
| console.log(embeddingBase64Obj); |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Requesting base64 encoded embeddings returns smaller body sizes, on average ~60% smaller than float32 encoded. In other words, the size of the response body containing embeddings in float32 is ~2.3x bigger than base64 encoded embedding.
Closes #1310
Changes being requested
We always request embedding creating encoded as base64, and then decoded them to float32 based on the user's provided encoding_format parameter.
Additional context & links
After running a few benchmarks, requesting base64 encoded embeddings returns smaller body sizes, on average ~60% smaller than float32 encoded. In other words, the size of the response body containing embeddings in float32 is ~2.3x bigger than base64 encoded embedding.
This performance improvement could translate to:
This is the result of a request that creates embedding from a 10kb chunk, run 10 times (the number are the size of response body in kb):
Read more #1310