How to use document loader in chain?

[magick93]

Hello

I'm working from the extract-openai-functions. And I'm wondering how I can modify this to pass in a url, and using something like the WebBaseLoader to fetch a url, rather than passing in a large body of text.

Any help is appreciated.

[eyurtsev]

Document loaders aren't runnables at the moment, but you can always wrap non runnables in a Runnable Lambda.

from langchain_core.runnables import RunnableLambda

asycc def get_text_to_analyze(url: str) -> str:
    loader = WebBaseLoader(..)
    documents = loader.aload()
    as_text = format_to_text(documents)
    return as_text


chain = RunnableLambda(get_text_to_analyze) | ... rest of extraction chain

Follow good security practices if you expose such chain as an endpoint on a server.

Since it allows an end user to extract things from a URL, a malicious user could also direct the server to access internal network resources that are supposed to be only accessible by the server (and not by users).

https://python.langchain.com/docs/security

[magick93]

Thanks @eyurtsev