Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add webhooks modal example #173

Merged
merged 6 commits into from
Apr 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 115 additions & 1 deletion docs/monitoring/faq/webhooks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -76,4 +76,118 @@ When delivering events to your webhook endpoint we follow these guidelines
- If your endpoint takes longer than 5 seconds to reply we declare the delivery failed and do not .
- If your endpoint returns a 5xx status code in less than 5 seconds we retry up to 2 times with exponential backoff.
Copy link

@charlesfrye charlesfrye Apr 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some cases where a Modal web endpoint might not spin up and ACK in less than 5 seconds. To avoid that, you can put keep_warm on the stub.function so you don't hit cold boots, but then you lose autoscaling. You can also reduce the chance of by putting less work in the path of the request -- e.g. defining a separate stub.function to handle a run and using Function.spawn to launch it as a background task, which means you can get to the ACK faster.

I'll include that in my example, so no need to put it in here, just wanted to raise the issue.

- If your endpoint returns a 4xx status code, we declare the delivery failed and do not retry.
- If your endpoint returns any body do not read it
- Anything your endpoint returns in the body will be ignored

## Example with Modal

### Setup

For an example of how to set this up, we will use [Modal](https://modal.com/). Modal provides autoscaling GPUs for inference and fine-tuning, secure containerization for code agents, and serverless Python web endpoints. We'll focus on the web endpoints here.

First, create a Modal account. Then, locally install the Modal SDK:

```shell
pip install modal
```

hwchase17 marked this conversation as resolved.
Show resolved Hide resolved
To finish setting up your account, run the command:

```shell
modal setup
```

and follow the instructions

### Secrets

Next, you will need to set up some secrets in Modal.

First, LangSmith will need to authenticate to Modal by passing in a secret.
The easiest way to do this is to pass in a secret in the query parameters.
To validate this secret, we will need to add a secret in _Modal_ to validate it.
We will do that by creating a Modal secret.
You can see instructions for secrets [here](https://modal.com/docs/guide/secrets).
For this purpose, let's call our secret `ls-webhook` and have it set an environment variable with the name `LS_WEBHOOK`.
hwchase17 marked this conversation as resolved.
Show resolved Hide resolved

We can also set up a LangSmith secret - luckily there is already an integration template for this!
hwchase17 marked this conversation as resolved.
Show resolved Hide resolved

![LangSmith Modal Template](../static/modal_langsmith_secret.png)

### Service

After that, you can create a Python file that will serve as your endpoint.
An example is below, with comments explaining what is going on:

```python
hwchase17 marked this conversation as resolved.
Show resolved Hide resolved
from fastapi import HTTPException, status, Request, Query
from modal import Secret, Stub, web_endpoint, Image

stub = Stub("auth-example", image=Image.debian_slim().pip_install("langsmith"))


@stub.function(
secrets=[Secret.from_name("ls-webhook"), Secret.from_name("my-langsmith-secret")]
)
# We want this to be a `POST` endpoint since we will post data here
@web_endpoint(method="POST")
# We set up a `secret` query parameter
def f(data: dict, secret: str = Query(...)):
# You can import dependencies you don't have locally inside Modal funxtions
from langsmith import Client

# First, we validate the secret key we pass
import os

if secret != os.environ["LS_WEBHOOK"]:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Incorrect bearer token",
headers={"WWW-Authenticate": "Bearer"},
)

# This is where we put the logic for what should happen inside this webhook
ls_client = Client()
runs = data["runs"]
ids = [r["id"] for r in runs]
feedback = list(ls_client.list_feedback(run_ids=ids))
for r, f in zip(runs, feedback):
try:
ls_client.create_example(
inputs=r["inputs"],
outputs={"output": f.correction},
dataset_name="classifier-github-issues",
)
except Exception:
raise ValueError(f"{r} and {f}")
# Function body
return "success!"
```

We can now deploy this easily with `modal deploy ...` (see docs [here](https://modal.com/docs/guide/managing-deployments)).

You should now get something like:

```
✓ Created objects.
├── 🔨 Created mount /Users/harrisonchase/workplace/langsmith-docs/example-webhook.py
├── 🔨 Created mount PythonPackage:langsmith
└── 🔨 Created f => https://hwchase17--auth-example-f.modal.run
✓ App deployed! 🎉

View Deployment: https://modal.com/apps/hwchase17/auth-example
```

The important thing to remember is `https://hwchase17--auth-example-f.modal.run` - the function we created to run.
NOTE: this is NOT the final deployment URL, make sure not to accidentally use that.

### Hooking it up

We can now take the function URL we create above and add it as a webhook.
We have to remember to also pass in the secret key as a query parameter.
Putting it all together, it should look something like:

```
https://hwchase17--auth-example-f-dev.modal.run?secret={SECRET}
```

Replace `{SECRET}` with the secret key you created to access the Modal service.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading