Invoke through forwarding message

[ezelanza]

Hi

My scenario is FRONT_END(:5005/openaiapi)--> BACK_END(:8000/api_openai )-INVOKE -->OPENAI (API)

In my front end, I'm using Fastapi listening to /api_openai, to forward the request to openai wrapper

FRONT_END

from pydantic import BaseModel

class Data(BaseModel):
    question: str

openai_llm = RemoteRunnable("http://localhost:8000/openai_api")
@app.post("/openaiapi")
async def process_text_data(question: Data):
    try:
        print("Entering try block...")
        user_question= str(question.question)
        
        prompt = ChatPromptTemplate.from_messages(
            [
                (
                    "system",
                    "You are a highly educated person who loves to use big words. "
                    + "You are also concise. Never answer in more than three sentences.",
                ),
                ("human", user_question),
            ]       
        ).format_messages()     
        
        # Pass the extracted question
        result=openai_llm.invoke(prompt)

        print(result.content)
        
        return result.content

BACKEND*

model = ChatOpenAI(openai_api_key="sk-xx")
#prompt = ChatPromptTemplate.from_template("Answer user question please :{question}")

add_routes(
    app,
    model,
    path="/openai_api",
)

The funny thing is that the request works when I run on my Jupyter notebook

from langserve import RemoteRunnable
from langchain.prompts.chat import ChatPromptTemplate

openai_llm = RemoteRunnable("http://localhost:8000/openai_api")
question = ("Tell me about k8s")
prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are a highly educated person who loves to use big words. "
            + "You are also concise. Never answer in more than three sentences.",
        ),
        ("human", question),
    ]
).format_messages()
save=openai_llm.invoke(prompt)
print(save.content)

K8s, short for Kubernetes, is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. It provides a highly scalable and flexible infrastructure for running and managing applications in a distributed environment.

BUT It doesn't work when I curl the front end

curl -X POST -H "Content-Type: application/json" -d '{"question": "Tell me about kubernetes"}' http://localhost:5005/api_openai
{"detail":"Error: [Errno 99] Cannot assign requested address"}% 

Doing some troubleshooting I found that the problem is in this part of the front_end code. The front end is a fast api implementation intended to fw requests to .invoke. Which works when is directly accessed (Jupyter) but it doesn't work when is curl by the fe

openai_llm.invoke(prompt)

I'm using the same envs and nothing is different. Is there anything I should be aware when forwarding requests? Does .invoke need any additional detail on the request?

[eyurtsev]

curl -X POST -H "Content-Type: application/json" -d '{"question": "Tell me about kubernetes"}' http://localhost:5005/api_openai
{"detail":"Error: [Errno 99] Cannot assign requested address"}% 

Try looking for the error message online -- I don't think this is LangServe related

e.g.,

https://support.prodi.gy/t/errno-99-error-while-attempting-to-bind-on-address-1-8080-0-0-cannot-assign-requested-address/3110