How to expose LLMS over a REST endpoint

[rajnikhil-netapp]

This module contains the functions to create the language model manager (LLM) instances.
The LLMs are used to interact with the language models (LMs) to generate responses, embeddings, etc.

The module contains the following functions:
* get_langchain_llm - returns a language model manager for the chat model.
* get_embedding_llm - returns a language model manager for the embeddings model.

import os
from langchain_openai import ChatOpenAI
from langchain_openai import OpenAIEmbeddings

def get_langchain_llm(
                model_name:str=os.environ.get("OPENAI_MODEL"),
                temperature:float=float(os.environ.get('OPENAI_TEMPRATURE')),
                request_timeout:float=os.environ.get("OPENAI_REQUEST_TIMEOUT"),
                user:str|None=None,
                **kwargs) -> ChatOpenAI:
    """Returns a language model manager for the chat model."""
    user = os.environ.get('DEFAULT_USER','test') if not user else user
    return ChatOpenAI(
            model_name=model_name,
            openai_api_key=os.environ['OPENAI_API_KEY'],
            openai_api_base=os.environ['OPENAI_API_BASE'],
            temperature=temperature,
            request_timeout=request_timeout,
            model_kwargs={'user': user},
            **kwargs)


def get_embedding_llm(
                model_name=os.environ.get("OPENAI_EMBEDDINGS_MODEL", default="text-embedding-ada-002"),
                user: str|None=None,
                max_retries=10,
                timeout=100) -> OpenAIEmbeddings:
    """Creating OpenAI Embedding LLM."""
    user = os.environ.get('DEFAULT_USER','test') if not user else user
    return OpenAIEmbeddings(
                model=model_name,
                openai_api_key=os.environ['OPENAI_API_KEY'],
                openai_api_base=os.environ['OPENAI_API_BASE'],
                max_retries=max_retries,
                timeout=timeout,
                model_kwargs={'user': user})

How can I expose these two llms over a REST endpoint using langserve.

[eyurtsev]

Simplest: https://github.com/langchain-ai/langserve/blob/main/examples/llm/server.py

If you need to support configuration, you can use configurable fields:
https://github.com/langchain-ai/langserve/blob/main/examples/configurable_chain/server.py#L47

If you need to pick up user information from the request itself:
There are different ways to do it (look at the examples), the most flexible is using the APIHandler:
https://github.com/langchain-ai/langserve/blob/main/examples/api_handler_examples/server.py

Use the examples for reference:

https://github.com/langchain-ai/langserve/tree/main?tab=readme-ov-file#examples