Serve Embedding Model

[RamishSiddiqui]

Hi guys,

I am trying to serve an embedding model through langserve is this possible?.

So far i have tried this but stuck at the error.

from langchain_community.embeddings import HuggingFaceEmbeddings

app = FastAPI(
    title="LangChain Server",
    version="1.0",
    description="A simple api server using Langchain's Runnable interfaces",
)

current_path = os.path.dirname(os.path.realpath(__file__))
model_name=os.path.abspath(os.path.join(current_path, r"../llms/all-MiniLM-L6-v2"))

model_kwargs = {
        'device': 'cuda'
}
encode_kwargs = {'normalize_embeddings': False}

add_routes(
    app,
    HuggingFaceEmbeddings(
        model_name=model_name,
        # model_kwargs=model_kwargs,
        encode_kwargs=encode_kwargs,
        multi_process=True
    ),
    path="/minilm_embed",
)

Error:

Traceback (most recent call last):
  File "/home/gpu-titan/Desktop/Ramish/Seer/langchain_server/main.py", line 31, in <module>
    add_routes(
  File "/home/gpu-titan/anaconda3/envs/Seer/lib/python3.9/site-packages/langserve/server.py", line 362, in add_routes
    api_handler = APIHandler(
  File "/home/gpu-titan/anaconda3/envs/Seer/lib/python3.9/site-packages/langserve/api_handler.py", line 553, in __init__
    runnable.get_input_schema(), "Input", model_namespace
AttributeError: 'HuggingFaceEmbeddings' object has no attribute 'get_input_schema'

[eyurtsev]

Any runnable object can be exposed. Embeddings are not runnables, so you need to re-wrap them into a runnable.

Here's the embeddings interface (not a runnable):

https://github.com/langchain-ai/langchain/blob/bf0b3cc0b5ade1fb95a5b1b6fa260e99064c2e22/libs/core/langchain_core/embeddings.py#L7-L7

The simplest way to do is is using a RunnableLambda.

from langchain_core.runnables import RunnableLambda
embedder = HuggingFaceEmbeddings(...)
runnable_embedder = RunnableLambda(afunc=embedder.aembed_documents)
add_routes(app, runnable_embedder)

That will expose an API around it.

LangServe doesn't do anything to optimize or manage hardware by anything that does local computations. So you should verify to see what kind of throughput you can get etc.

[shubhankar1477]

Hi ,

I am trying the same thing but getting error that Runnable Lambda has no attribute embed documents

code :

from langchain_core.runnables import RunnableLambda
from langchain_community.embeddings import HuggingFaceEmbeddings
emb_model_name, dimension, emb_model_identifier = "all-MiniLM-L6-v2", 384, "all-minilm-l6"
embeddings = HuggingFaceEmbeddings(model_name=emb_model_name, encode_kwargs={'normalize_embeddings': True,})
runnable_embedder = RunnableLambda(afunc = embeddings.aembed_documents)
vectorstore = DocArrayInMemorySearch.from_texts(
    ["harrison worked at kensho", "bears like to eat honey"],
    embedding=runnable_embedder,
)
retriever = vectorstore.as_retriever()

error:
AttributeError: 'RunnableLambda' object has no attribute 'embed_documents'

Thanks for reply