add webhooks modal example

docs/monitoring/faq/webhooks.mdx

@@ -77,3 +77,116 @@ When delivering events to your webhook endpoint we follow these guidelines
 - If your endpoint returns a 5xx status code in less than 5 seconds we retry up to 2 times with exponential backoff.
 - If your endpoint returns a 4xx status code, we declare the delivery failed and do not retry.
 - If your endpoint returns any body do not read it
+
+## Example with Modal
+
+### Setup
+
+For an example of how to set this up, we will use [Modal](https://modal.com/).
+
+First, create a Modal account. Then, locally install the Modal SDK:
+
+```shell
+pip install modal
+```
+
+### Secrets
+
+Next, you will need to set up some secrets.
+
+First, we will set up a secret to use to authenticate to our server.
+You can see instructions for secrets [here](https://modal.com/docs/guide/secrets).
+For this purpose, let's call our secret `ls-webhook` and have it set an environment variable with the name `LS_WEBHOOK`.
+
+We can also set up a LangSmith secret - luckily there is already an integration template for this!
+
+![LangSmith Modal Template](../static/modal_langsmith_secret.png)
+
+### Service
+
+After that, you can create a Python file that will serve as your endpoint.
+An example is below, with comments explaining what is going on:
+
+```python
+from fastapi import Depends, HTTPException, status, Request, Query
+from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
+
+from modal import Secret, Stub, web_endpoint, Image
+from langsmith import Client
+
+# Create a job called "auth-example" that install the `langsmith` package as a dependency
+# Installing `langsmith` is just an example, you may need to install other packages
+# depending on your logic.
+stub = Stub("auth-example", image=Image.debian_slim().pip_install("langsmith"))
+
+# These are the secrets we defined above
+@stub.function(
+    secrets=[
+        Secret.from_name("ls-webhook"),
+        Secret.from_name("my-langsmith-secret")
+    ]
+ )
+# We want this to be a `POST` endpoint since we will post data here
+@web_endpoint(method="POST")
+# We set up a `secret` query parameter
+async def f(request: Request, secret: str = Query(...), ):
+
+    # First, we validate the secret key we pass
+    import os
+
+    if secret != os.environ["LS_WEBHOOK"]:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Incorrect bearer token",
+            headers={"WWW-Authenticate": "Bearer"},
+        )
+
+    # Now we load the JSON payload for the runs that are passed to the webhook
+    body = await request.json()
+    # This is where we put the logic for what should happen inside this webhook
+    runs = body['runs']
+    ls_client = Client()
+    ids = [r['id'] for r in runs]
+    feedback = list(ls_client.list_feedback(run_ids=ids))
+    for r, f in zip(runs, feedback):
+        try:
+            ls_client.create_example(
+                inputs=r['inputs'],
+                outputs={'output': f.correction},
+                dataset_name="classifier-github-issues"
+            )
+        except Exception as e:
+            raise ValueError(f"{r} and {f}")
+    # Function body
+    return "success!"
+
+```
+
+We can now deploy this easily with `modal deploy ...` (see docs [here](https://modal.com/docs/guide/managing-deployments)).
+
+You should now get something like:
+
+```
+✓ Created objects.
+├── 🔨 Created mount /Users/harrisonchase/workplace/langsmith-docs/example-webhook.py
+├── 🔨 Created mount PythonPackage:langsmith
+└── 🔨 Created f => https://hwchase17--auth-example-f.modal.run
+✓ App deployed! 🎉
+
+View Deployment: https://modal.com/apps/hwchase17/auth-example
+```
+
+The important thing to remember is `https://hwchase17--auth-example-f.modal.run` - the function we created to run.
+NOTE: this is NOT the final deployment URL, make sure not to accidentally use that.
+
+### Hooking it up
+
+We can now take the function URL we create above and add it as a webhook.
+We have to remember to also pass in the secret key as a query parameter.
+Putting it all together, it should look something like:
+
+```
+https://hwchase17--auth-example-f-dev.modal.run?secret={SECRET}
+```
+
+Replace `{SECRET}` with the secret key you created to access the Modal service.

docs/monitoring/use_cases/classification.mdx

@@ -0,0 +1,194 @@
+---
+sidebar_label: Optimizing a Classifier
+sidebar_position: 2
+table_of_contents: true
+---
+
+# Optimizing a Classifier
+
+This tutorial walks through optimizing a classifier based on user a feedback.
+Classifiers are great to optimize because its generally pretty simple to collect the desired output, which makes it easy to create few shot examples based on user feedback.
+That is exactly what we will do in this example.
+
+## The objective
+
+In this example, we will build a bot that classify GitHub issues based on their title.
+It will take in a title and classify it into one of many different classes.
+Then, we will start to collect user feedback and use that to shape how this classifier performs.
+
+## Getting started
+
+To get started, we will first set it up so that we send all traces to a specific project.
+We can do this by setting an environment variable:
+
+```python
+import os
+os.environ["LANGCHAIN_PROJECT"] = "classifier"
+```
+
+We can then create our initial application. This will be a really simple function that just takes in a GitHub issue title and tries to label it.
+
+```python
+import openai
+from langsmith import traceable, Client
+import uuid
+
+client = openai.Client()
+
+available_topics = [
+    "bug",
+    "improvement",
+    "new_feature",
+    "documentation",
+    "integration",
+]
+
+@traceable(
+    run_type="chain",
+    name="Classifier",
+)
+def topic_classifier(
+    topic: str
+):
+    return client.chat.completions.create(
+        model="gpt-3.5-turbo",
+        messages=[{"role": "user", "content": f"Classify the type of issue as one of {','.join(available_topics)}\n\nIssue: {topic}"}],
+    ).choices[0].message.content
+```
+
+We can then start to interact with it.
+When interacting with it, we will generate the LangSmith run id ahead of time and pass that into this function.
+We do this so we can attach feedback later on.
+
+Here's how we can invoke the application:
+
+```python
+run_id = uuid.uuid4()
+topic_classifier(
+    "fix bug in LCEL",
+    langsmith_extra={"run_id": run_id}
+)
+```
+
+Here's how we can attach feedback after.
+We can collect feedback in two forms.
+
+First, we can collect "positive" feedback - this is for examples that the model got right.
+
+
+```python
+ls_client = Client()
+run_id = uuid.uuid4()
+
+topic_classifier(
+    "fix bug in LCEL",
+    langsmith_extra={"run_id": run_id}
+)
+
+ls_client.create_feedback(
+    run_id,
+    key="user-score",
+    score=1.0,
+)
+```
+
+Next, we can focus on collecting feedback that corresponds to a "correction" to the generation.
+In this example the model will classify it as a bug, whereas I really want this to be classified as documentation.
+
+```python
+ls_client = Client()
+
+run_id = uuid.uuid4()
+topic_classifier(
+    "fix bug in documentation",
+    langsmith_extra={"run_id": run_id}
+)
+
+ls_client.create_feedback(
+    run_id,
+    key="correction",
+    correction="documentation"
+)
+```
+
+## Set up automations
+
+We can now set up automations to move examples with feedback of some form into a dataset.
+We will set up two automations, one for positive feedback and the other for negative feedback.
+
+The first will take all runs with positive feedback and automatically add them to a dataset.
+The logic behind this is that any run with positive feedback we can use as a good example in future iterations.
+Let's create a dataset called `classifier-github-issues` to add this data to.
+
+![Optimization Positive](../static/optimization-classification-positive.png)
+
+The second will take all runs with a correction and use a webhook to add them to a dataset
+
+![Optimization Negative](../static/optimization-classification-negative.png)
+
+In order to do this, we need to set up a webhook. We will do this using [Modal](https://modal.com/).
+To see in-depth instructions on this, see instructions [here](../faq/webhooks).
+
+The Python file that defines our Modal service should look like:
+
+```python
+from fastapi import Depends, HTTPException, status, Request, Query
+from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
+
+from modal import Secret, Stub, web_endpoint, Image
+from langsmith import Client
+
+# Create a job called "auth-example" that install the `langsmith` package as a dependency
+# Installing `langsmith` is just an example, you may need to install other packages
+# depending on your logic.
+stub = Stub("auth-example", image=Image.debian_slim().pip_install("langsmith"))
+
+# These are the secrets we defined above
+@stub.function(
+    secrets=[
+        Secret.from_name("ls-webhook"),
+        Secret.from_name("my-langsmith-secret")
+    ]
+ )
+# We want this to be a `POST` endpoint since we will post data here
+@web_endpoint(method="POST")
+# We set up a `secret` query parameter
+async def f(request: Request, secret: str = Query(...), ):
+
+    # First, we validate the secret key we pass
+    import os
+
+    if secret != os.environ["LS_WEBHOOK"]:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Incorrect bearer token",
+            headers={"WWW-Authenticate": "Bearer"},
+        )
+
+    # Now we load the JSON payload for the runs that are passed to the webhook
+    body = await request.json()
+    # We get all runs
+    runs = body['runs']
+    ls_client = Client()
+    # We get the ids of the runs we are passed
+    ids = [r['id'] for r in runs]
+    # We fetch feedback for all those runs
+    feedback = list(ls_client.list_feedback(run_ids=ids))
+    # We then iterate over these runs and associated feedback
+    for r, f in zip(runs, feedback):
+        # We create an example in the dataset
+        # This example has the same inputs as our application
+        # But the outputs are now the "correction" that we left as feedback
+        ls_client.create_example(
+            inputs=r['inputs'],
+            outputs={'output': f.correction},
+            dataset_name="classifier-github-issues"
+        )
+    # Function body
+    return "success!"
+
+```
+
+## Updating the application
+
+We can now update our code to pull down the dataset we are sending runs to.