fine-tuning-examples/export-to-openai/fine-tuning-on-chat-runs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "adbda4ad-08f2-4764-b4d4-32344780cedf",
   "metadata": {},
   "source": [
    "# OpenAI Fine-Tuning\n",
    "[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langsmith-cookbook/blob/main/fine-tuning-examples/export-to-openai/fine-tuning-on-chat-runs.ipynb)\n",
    "\n",
    "Once you've captured run traces from your deployment (production or beta), it's likely you'll want to use that data to\n",
    "fine-tune a model. This walkthrough will show a quick way to do so.\n",
    "\n",
    "Steps:\n",
    "\n",
    "1. Query runs (optionally filtering by project, time, tags, etc.)\n",
    "   - [Optional] Create a 'training' dataset to keep track of the data used for this model.\n",
    "2. Convert runs to OpenAI messages or another format)\n",
    "3. Fine-tune and use new model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b9f241f3-68a1-402a-85bf-60f23769dec1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langsmith import Client\n",
    "\n",
    "client = Client()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8001c4b-70bf-48bd-9605-a120c6579838",
   "metadata": {},
   "source": [
    "## 1. Query runs\n",
    "\n",
    "LangSmith saves traces for each runnable component in your LLM application. You can then query these runs in a variety of ways to construct your a training dataset. We will show a few common patterns below.\n",
    "\n",
    "For examples of more 'advanced' filtering, check out the [filtering guide](https://docs.smith.langchain.com/tracing/faq/querying_traces) in the LangSmith docs.\n",
    "\n",
    "**List all LLM runs for a specific project.**\n",
    "\n",
    "The simplest query is just listing \"llm\" runs in your project (filtering out runs with errors). Below is an example where we list all LLM runs in the default project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2e14cf13-1070-4840-a66e-f8a4d81ceeb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "project_name = \"default\"\n",
    "run_type = \"llm\"\n",
    "end_time = datetime.datetime.now()\n",
    "\n",
    "runs = client.list_runs(\n",
    "    project_name=project_name,\n",
    "    run_type=run_type,\n",
    "    error=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4d88f96-d097-4a2e-8e92-2b6652c19a6e",
   "metadata": {},
   "source": [
    "#### Filter by feedback\n",
    "\n",
    "Depending on how you're fine-tuning, you'll likely want to filter out 'bad' examples (and want to filter in 'good' examples).\n",
    "\n",
    "You can directly list by feedback! Usually you assign feedback to the root of the run trace, so we will use 2 queries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "7678dd1e-d001-41a9-976e-923d8d60ad1b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain import chains, chat_models, prompts, schema, callbacks\n",
    "\n",
    "chain = (\n",
    "    prompts.ChatPromptTemplate.from_template(\"Tell a joke for:\\n{input}\")\n",
    "    | chat_models.ChatAnthropic(tags=[\"my-anthropic-run\"])\n",
    "    | schema.output_parser.StrOutputParser()\n",
    ")\n",
    "\n",
    "with callbacks.collect_runs() as cb:\n",
    "    chain.invoke({\"input\": \"foo\"})\n",
    "    # Assume feedback is logged\n",
    "    run = cb.traced_runs[0]\n",
    "    client.create_feedback(run.id, key=\"user_click\", score=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "eb2db550-2346-4a9a-8792-7402a2937ab9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "project_name = \"default\"\n",
    "end_time = datetime.datetime.now()\n",
    "\n",
    "runs = client.list_runs(\n",
    "    project_name=project_name,\n",
    "    execution_order=1,\n",
    "    filter='and(eq(feedback_key, \"user_click\"), eq(feedback_score, 1))',\n",
    "    # For continuous scores, you can filter for >, <, >=, <= with the followingg arguments: gt/lt/gte/lte(feedback_score, 0.9)\n",
    "    # filter='and(eq(feedback_key, \"user_click\"), gt(feedback_score, 0.9))',\n",
    "    error=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8dcfd9c-71e0-4746-a042-e5ecb0105f48",
   "metadata": {},
   "source": [
    "Once you have these run ids, you can find the LLM run if it is a direct child of the root or if you\n",
    "use a tag for a given trace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "43b661da-8e2d-4a31-962c-92669579f10e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['my-anthropic-run']"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm_runs = []\n",
    "for run in runs:\n",
    "    llm_run = next(\n",
    "        client.list_runs(\n",
    "            project_name=project_name, run_type=\"llm\", parent_run_id=run.id\n",
    "        )\n",
    "    )\n",
    "    llm_runs.append(llm_run)\n",
    "\n",
    "llm_runs[0].tags"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1776323-440b-4333-b19e-a175600f2b17",
   "metadata": {},
   "source": [
    "#### Filter by tags\n",
    "\n",
    "It's common to have multiple chain types in a single project, meaning that the LLM calls may span multiple tasks and domains. Tags are a useful way to organize runs by task, component, test variant, etc, so you can curate a coherent dataset.\n",
    "\n",
    "Below is a quick example. Please also reference the [Tracing FAQs](https://docs.smith.langchain.com/tracing/faq/customizing_trace_attributes#adding-metadata-and-tags-to-tracess) for more information on tagging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "07f36f4b-2e36-44f8-bca3-451f4ec62653",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input': 'podcasting these days', 'text': ' Here\\'s a joke about podcasting these days:\\n\\nEveryone seems to have a podcast these days. My neighbor started one where he just reads his grocery lists out loud. It\\'s called \"What\\'s in Store?\" and it\\'s surprisingly addicting. I also heard about a podcast where someone literally just snoozes and snores into the mic for an hour. They call it \"Podsleeping.\" And don\\'t even get me started on my cousin\\'s mumbling podcast. You can\\'t understand a word she\\'s saying, but she swears it\\'s riveting stuff. I guess when it comes to podcasts, you can record and release pretty much anything nowadays. The bar is so low, it\\'s practically underground at this point!'}\n",
      " Here's a joke about podcasting these days:\n",
      "\n",
      "It seems like everyone has a podcast now. My grandma just started one called \"The Quilting Hour\" where she talks about different stitch patterns while soft piano music plays in the background. Then my dog launched his own podcast where he just barks and pants into the microphone for 30 minutes. At this rate, my goldfish is going to start a podcast where he blows bubbles near the mic. Podcasting has really jumped the shark. Pretty soon we'll have more podcasts than actual listeners!\n"
     ]
    }
   ],
   "source": [
    "# For any \"Chain\" object, you can add tags directly on the Example with LLMChain\n",
    "import uuid\n",
    "\n",
    "unique_tag = f\"call:{uuid.uuid4()}\"\n",
    "\n",
    "chain = chains.LLMChain(\n",
    "    llm=chat_models.ChatAnthropic(\n",
    "        tags=[\"my-cool-llm-tag\"]\n",
    "    ),  # This tag will only be applied to the LLM\n",
    "    prompt=prompts.ChatPromptTemplate.from_template(\n",
    "        \"Tell a joke based on the following prompt:\\n\\nPrompt:{input}\"\n",
    "    ),\n",
    "    tags=[\"my-tag\"],\n",
    ")\n",
    "\n",
    "# You can also define at call time for the call/invoke/batch methods.\n",
    "# This tag will be propagated to all child calls\n",
    "print(chain({\"input\": \"podcasting these days\"}, tags=[unique_tag]))\n",
    "\n",
    "# If you're defining using Runnables (aka langchain expression language)\n",
    "runnable = (\n",
    "    prompts.ChatPromptTemplate.from_template(\n",
    "        \"Tell a joke based on the following prompt:\\n\\nPrompt:{input}\"\n",
    "    )\n",
    "    | chat_models.ChatAnthropic(\n",
    "        tags=[\"my-cool-llm-tag\"]\n",
    "    )  # This tag will only be applied to the LLM\n",
    "    | schema.StrOutputParser(tags=[\"some-parser-tag\"])\n",
    ")\n",
    "\n",
    "# Again, you can tag at call time as well. This tag will be propagated to all child calls\n",
    "print(runnable.invoke({\"input\": \"podcasting these days\"}, {\"tags\": [unique_tag]}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "81d2ceb2-0493-4d01-8e92-48d25141fd55",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project_name = \"default\"\n",
    "end_time = datetime.datetime.now()\n",
    "\n",
    "runs = client.list_runs(\n",
    "    execution_order=1,  # Only return the root trace\n",
    "    filter=f'has(tags, \"{unique_tag}\")',\n",
    ")\n",
    "len(list(runs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2750e28d-704a-4f03-8b90-aae034998b09",
   "metadata": {},
   "source": [
    "#### Filter by run name.\n",
    "\n",
    "By default, the run name is the class of the object being traced. You can filter by run name to narrow your search by, e.g., the LLM class.\n",
    "\n",
    "Below, we will list all runs sent to a \"ChatAnthropic\" llm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "f820926a-cae4-48ce-99ac-147f41f9a012",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "project_name = \"default\"\n",
    "run_type = \"llm\"\n",
    "end_time = datetime.datetime.now()\n",
    "\n",
    "runs = client.list_runs(\n",
    "    project_name=project_name,\n",
    "    run_type=run_type,\n",
    "    filter='eq(name, \"ChatAnthropic\")',\n",
    "    error=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed5c5ef2-55a9-4ca2-b5ae-07fc2c02a94b",
   "metadata": {},
   "source": [
    "#### Retrieve prompt inputs directly\n",
    "\n",
    "If you fetch the LLM or chat run directly, the input will be the formatted prompt, with the values injected. You may want to separate the injected values from the prompt templating to remove or reduce the quantity of instruction prompting needed to obtain the desired prediction.\n",
    "\n",
    "If your chain is composed as runnables (for instance, if you use LangChain Expression Language),\n",
    "each __prompt__ runnable will be given its own run trace. You can fetch the inputs to the prompt template directly so that when you fine-tune, you can elide the other template content and train directly on the input values and LLM outputs.\n",
    "\n",
    "Take the following chain, for instance, which is promoted to a [RunnableSequence](https://api.python.langchain.com/en/latest/runnables/langchain_core.runnables.base.RunnableSequence.html) via the piping operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "d84a7147-3c38-4f0f-b1a7-1161914fd248",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='The chat log consists of a simple greeting exchange, with one person saying \"hi there\" and the other responding with \"hello.\"', additional_kwargs={}, example=False)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example chain for the following query\n",
    "from langchain import prompts, chat_models\n",
    "\n",
    "chain = (\n",
    "    prompts.ChatPromptTemplate.from_template(\n",
    "        \"Summarize the following chat log: {input}\"\n",
    "    )\n",
    "    | chat_models.ChatOpenAI()\n",
    ")\n",
    "\n",
    "chain.invoke({\"input\": \"hi there, hello....\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "004ff8f5-8044-4681-a8dd-dd3c59617ca2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "project_name = \"default\"\n",
    "run_type = \"prompt\"\n",
    "end_time = datetime.datetime.now()\n",
    "\n",
    "runs = client.list_runs(\n",
    "    project_name=project_name,\n",
    "    run_type=run_type,\n",
    "    end_time=end_time,\n",
    "    error=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "f663151a-a3b2-4ea6-ae6f-67c145cd1194",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# You can then get a sibling LLM run by searching by parent_run_id and including other criteria\n",
    "for prompt_run in runs:\n",
    "    llm_run = next(\n",
    "        client.list_runs(\n",
    "            project_name=project_name,\n",
    "            run_type=\"llm\",\n",
    "            parent_run_id=prompt_run.parent_run_id,\n",
    "        )\n",
    "    )\n",
    "    inputs, outputs = prompt_run.inputs, llm_run.outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f023f35-79cc-4caf-8356-af908adb067f",
   "metadata": {},
   "source": [
    "## 1a: (Recommended) Add to a training dataset\n",
    "\n",
    "While not necessary for the fast-path of making your first fine-tuned model, datasets help build in a principled way by helping track the exact data used in a given model. They also are a natural place to add manual review or spot checking in the web app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "c76dabcd-fd84-4d07-99e6-e348e7777aec",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = client.create_dataset(\n",
    "    dataset_name=\"Fine-Tuning Dataset Example\",\n",
    "    description=f\"Chat logs taken from project {project_name} for fine-tuning\",\n",
    "    data_type=\"chat\",\n",
    ")\n",
    "for run in runs:\n",
    "    if \"messages\" not in run.inputs or not run.outputs:\n",
    "        # Filter out non chat runs\n",
    "        continue\n",
    "    try:\n",
    "        # Convenience method for creating a chat example\n",
    "        client.create_example_from_run(\n",
    "            dataset_id=dataset.id,\n",
    "            run=run,\n",
    "        )\n",
    "        # Or if you want to select certain keys/values in inputs\n",
    "        # inputs = convert_inputs(run.inputs)\n",
    "        # outputs = convert_outputs(run.outputs)\n",
    "        # client.create_example(\n",
    "        #     dataset_id=dataset.id,\n",
    "        #     inputs=inputs,\n",
    "        #     outputs=outputs,\n",
    "        #     run=run,\n",
    "        # )\n",
    "    except:\n",
    "        # Duplicate inputs raise an exception\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64f52848-ce39-4673-999f-af7d2fc1921a",
   "metadata": {},
   "source": [
    "## 2. Load examples as messages\n",
    "\n",
    "We will first load the messages as LangChain objects then take advantage of the OpenAI adapter helper to convert these\n",
    "to dictionaries in the form expected by OpenAI's fine-tuning endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "664c63b3-ef2c-42cd-82bb-06c08a3490f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langsmith import schemas\n",
    "from langchain import load\n",
    "\n",
    "\n",
    "def convert_messages(example: schemas.Example) -> dict:\n",
    "    messages = load.load(example.inputs)[\"messages\"]\n",
    "    message_chunk = load.load(example.outputs)[\"generations\"][0][\"message\"]\n",
    "    return {\"messages\": messages + [message_chunk]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0adb0188-a5de-4ab5-8eb5-a241f8d77cd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "messages = [\n",
    "    convert_messages(example)\n",
    "    for example in client.list_examples(dataset_name=\"Fine-Tuning Dataset Example\")\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d600fee-cc3d-4b50-a952-10ef635fa137",
   "metadata": {},
   "source": [
    "Now that we have the traces back as LangChain message objects, you can use the adapters\n",
    "to convert to other formats, such as OpenAI's fine-tuning format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7e2f4b91-ab27-41df-974f-367169579c64",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.adapters import openai as openai_adapter\n",
    "\n",
    "finetuning_messages = openai_adapter.convert_messages_for_finetuning(messages)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "407fbfa5-a48f-4181-9fed-224278738a25",
   "metadata": {},
   "source": [
    "## 3. Finetune\n",
    "\n",
    "Now you can use these message dictionaries for downstream tasks like fine-tuning. Note that the OpenAI API doesn't currently support the 'function_call' argument when fine-tuning. We will filter these out first here. It may be that this requirement is relaxed by the time you read this guide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "68039265-b8dc-4d61-89f6-4faf1b0143ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File file-ixTtVCKDGrZ7PiVZFszQr6kN ready after 30.55 seconds.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "import json\n",
    "import io\n",
    "\n",
    "import openai\n",
    "\n",
    "my_file = io.BytesIO()\n",
    "for group in finetuning_messages:\n",
    "    if any([\"function_call\" in message for message in group]):\n",
    "        continue\n",
    "    my_file.write((json.dumps({\"messages\": group}) + \"\\n\").encode(\"utf-8\"))\n",
    "\n",
    "my_file.seek(0)\n",
    "training_file = openai.File.create(file=my_file, purpose=\"fine-tune\")\n",
    "\n",
    "# Wait while the file is processed\n",
    "status = openai.File.retrieve(training_file.id).status\n",
    "start_time = time.time()\n",
    "while status != \"processed\":\n",
    "    print(f\"Status=[{status}]... {time.time() - start_time:.2f}s\", end=\"\\r\", flush=True)\n",
    "    time.sleep(5)\n",
    "    status = openai.File.retrieve(training_file.id).status\n",
    "print(f\"File {training_file.id} ready after {time.time() - start_time:.2f} seconds.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bce9daa-d4c5-4dd2-bf57-0cd2d2a3765b",
   "metadata": {},
   "source": [
    "Next, fine-tune the model. This could take 10+ minutes depending on the server's load and your dataset size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "39408d83-12b2-4634-9dff-36a3d13a53b6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status=[running]... 743.73s\r"
     ]
    }
   ],
   "source": [
    "job = openai.FineTuningJob.create(\n",
    "    training_file=training_file.id,\n",
    "    model=\"gpt-3.5-turbo\",\n",
    ")\n",
    "\n",
    "# It may take 10-20+ minutes to complete training.\n",
    "status = openai.FineTuningJob.retrieve(job.id).status\n",
    "start_time = time.time()\n",
    "while status != \"succeeded\":\n",
    "    print(f\"Status=[{status}]... {time.time() - start_time:.2f}s\", end=\"\\r\", flush=True)\n",
    "    time.sleep(5)\n",
    "    job = openai.FineTuningJob.retrieve(job.id)\n",
    "    status = job.status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6813f53-904a-4b47-b489-2f4054b52f45",
   "metadata": {},
   "source": [
    "Now you can use the model within langchain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "1d731298-68c0-46e8-af59-ca3d1a573507",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='I am designed to assist anyone who has questions or needs help related to LangChain. This could include developers, users, or anyone else who is interested in or using LangChain.', additional_kwargs={}, example=False)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain import chat_models, prompts\n",
    "\n",
    "model_name = job.fine_tuned_model\n",
    "# Example: ft:gpt-3.5-turbo-0613:personal::5mty86jblapsed\n",
    "model = chat_models.ChatOpenAI(model=model_name)\n",
    "chain.invoke({\"input\": \"Who are you designed to assist?\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fcea9a3-40d1-4cff-a55d-abbcaa12fcce",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Congratulations! You've fine-tuned a model on your traced LLM runs.\n",
    "\n",
    "This is an extremely simple recipe that demonstrates the end-to-end workflow.\n",
    "It is likely that you will want to use various methods to filter, fix, and observe the data you choose to fine-tune the model on. We welcome additional recipes of things that have worked for you!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}