bedrock configurable fields with model_id doesn't work

[yw791772]

I'm following one of the examples:

async def config_model(config: Dict[str, Any], req: Request) -> Dict[str, Any]:
    request_body = await req.json()
    if "model_name" in request_body:
        config["configurable"]["model_id"] = request_body["model_name"]
    return config
   
add_routes(
      app,
      BedrockChat(model_id="meta.llama2-13b-chat-v1").configurable_fields(
          model_id=ConfigurableField(
              id="model_id",
              name="beckrock API model id",
              description=("The id of the model to use for bedrock interaction"),
          )
      ),
      path="/bedrock",
      per_req_config_modifier=config_model,
  )

this works fine with open ai(note for open ai, it is model_name), but when I change it to bedrockChat, I keep getting the following error:

pydantic.v1.errors.ConfigError: field "config" not yet prepared so type is still a ForwardRef, you might need to call BedrockChat.update_forward_refs().

Note: when I remove the configurable_fields logic, and just use
add_routes(
app,
BedrockChat(model_id="meta.llama2-13b-chat-v1"),
path="/bedrock",
)

it works.

Wondering if anyone could share some light

[eyurtsev]

pydantic.v1.errors.ConfigError: field "config" not yet prepared so type is still a ForwardRef, you might need to call BedrockChat.update_forward_refs().

Looks like an issue due to pydantic type annotations not being resolved.

Have you tried typing BedrockChat.update_forward_refs() prior to add_routes?

[yw791772]

thanks for the reply, I haven't, will try that today. weird that I didn't run into this issue when I use the roughly the same code on openAIChat, with configurable model_name field.
Will let you know what I find

[yw791772]

I've tried add BedrockChat.update_forward_refs(), but hitting a Config not defined error. I go back to look at the stack trace, seems like previous error message I copied is not root. Paste the first error that showing up:

INFO:     127.0.0.1:55460 - "POST /bedrock/stream HTTP/1.1" 200 OK
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/vscode/.local/share/virtualenvs/langserve-3zV0jjiH/lib/python3.10/site-packages/sse_starlette/sse.py", line 281, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/home/vscode/.local/share/virtualenvs/langserve-3zV0jjiH/lib/python3.10/site-packages/sse_starlette/sse.py", line 270, in wrap
    await func()
  File "/home/vscode/.local/share/virtualenvs/langserve-3zV0jjiH/lib/python3.10/site-packages/sse_starlette/sse.py", line 221, in listen_for_disconnect
    message = await receive()
  File "/home/vscode/.local/share/virtualenvs/langserve-3zV0jjiH/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 542, in receive
    await self.message_event.wait()
  File "/usr/local/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f7f8ac3f1f0
During handling of the above exception, another exception occurred:
   the field "config" error I paster earlier show up.

[eyurtsev]

This is likely some bug associated with type annotations on the LangChain side. I'll mark as bug right now, will try to investigate when i have free time.

You will probably be able to work-around by manually specifying .with_types on the model (not 100% sure), I suspect it'll also work if composed as part of the chain (e.g., add a prompt in front of it).

The issue is very likely from the inputs and output schema that's associated with the model not being resolved for whatever reason.

[yw791772]

sorry for the late reply, I will give it a shot. thank you for look into it