3_summarizing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f1b0f5c9-8070-44b8-b1ea-c0a337d373ce",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Summarization\n",
    "\n",
    "In this part of the course, we will attempt to use a language model to make summaries of some papers.\n",
    "Making summaries of documents is also known as summarizing or summarization.\n",
    "There exists specialized software for making summaries.\n",
    "However, general large language models are also becoming quite good at this task.\n",
    "\n",
    "Again, we will use [LangChain](https://www.langchain.com/), an open-source library for making applications with LLMs.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d75d3df-17c0-4ff9-8659-36489f174b33",
   "metadata": {},
   "source": [
    "## Document location\n",
    "We have collected some papers licensed with a Creative Commons license.\n",
    "We will try to load all the documents in the folder defined below.\n",
    "If you prefer, you can change this to a different folder name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e62dd1c-6122-45c1-90b3-16badbf11313",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "document_folder = '/fp/projects01/ec443/documents/terrorism'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab8e02df-c8fc-4c67-8ba7-a4c96e2ee715",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "remove-cell"
    ]
   },
   "source": [
    "## Some configuration\n",
    "To conserve memory, we configure more efficient memory use on the GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07fb9a33-487c-45b7-b443-eeec81a8c3ec",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "#%env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62855d10-417b-4648-a14e-f5512c1f4f18",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## The Language Model\n",
    "We'll use models from [HuggingFace](https://huggingface.co/), a website that has tools and models for machine learning.\n",
    "We'll use the open-weights LLM \n",
    "[mistralai/Ministral-8B-Instruct-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410).\n",
    "We use the 8B model because it has a large context window of 128k tokens.\n",
    "This means that it can process quite large documents.\n",
    "\n",
    "```{admonition} Tokens versus Words\n",
    "Short words can be a single token, but longer words usually consist of multiple tokens.\n",
    "Therefore, the maximum document size with this model is less than 128k words.\n",
    "Exactly how words are converted to tokens depends on the *tokenizer*.\n",
    "LLMs usually come with tokenizers.\n",
    "We will use the default tokenizer that ship with the LLM we use.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac38b664-7075-4ed6-9a21-bb1465b7c095",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [],
   "source": [
    "%env HF_HOME=/fp/projects01/ec443/huggingface/cache/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fb595bd",
   "metadata": {},
   "source": [
    "To use the model, we create a *pipeline*.\n",
    "A pipeline can consist of several processing steps, but in this case, we only need one step.\n",
    "We can use the method `HuggingFacePipeline.from_model_id()`, which automatically downloads the specified model from HuggingFace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3059fb20-861e-4d7d-b5a4-cd141b8b352b",
   "metadata": {
    "editable": true,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [],
   "source": [
    "from langchain_huggingface.llms import HuggingFacePipeline\n",
    "\n",
    "llm = HuggingFacePipeline.from_model_id(\n",
    "    model_id='mistralai/Ministral-8B-Instruct-2410',\n",
    "    task='text-generation',\n",
    "    device=0,\n",
    "    pipeline_kwargs={\n",
    "        'max_new_tokens': 1000,\n",
    "        #'do_sample': True,\n",
    "        #'temperature': 0.3,\n",
    "        #'num_beams': 4,\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42032173",
   "metadata": {},
   "source": [
    "\n",
    "We can give some arguments to the pipeline:\n",
    "- `model_id`: the name of the  model on HuggingFace\n",
    "- `task`:  the task you want to use the model for\n",
    "- `device`: the GPU hardware device to use. If we don't specify a device, no GPU will be used.\n",
    "- `pipeline_kwargs`: additional parameters that are passed to the model.\n",
    "    - `max_new_tokens`: maximum length of the generated text\n",
    "    - `do_sample`: by default, the most likely next word is chosen.  This makes the output deterministic. We can introduce some randomness by sampling among the  most likely words instead.\n",
    "    - `temperature`: the temperature controls the statistical *distribution* of the next word and is usually between 0 and 1. A low temperature increases the probability of common words. A high temperature increases the probability of outputting a rare word. Model makers often recommend a temperature setting, which we can use as a starting point.\n",
    "    - `num_beams`: by default the model works with a single sequence of  tokens/words. With beam search, the program  builds multiple sequences at the same time, and then selects the best one in the end.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1e788a2",
   "metadata": {},
   "source": [
    "## Making a Prompt\n",
    "We can use a *prompt* to tell the language model how to answer.\n",
    "The prompt should contain a few short, helpful instructions.\n",
    "In addition, we provide placeholders for the input,  called *context*.\n",
    "LangChain replaces the placeholder with the input document when we execute a query.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d166216-7da7-498a-bb58-73f9ecb451ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "from langchain.chains.llm import LLMChain\n",
    "from langchain.prompts import PromptTemplate\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3f7e173-efe0-4241-af09-a205ea6ba274",
   "metadata": {},
   "outputs": [],
   "source": [
    "separator = '\\nYour Summary:\\n'\n",
    "prompt_template = '''Write a summary of the following:\n",
    "\n",
    "{context}\n",
    "''' + separator\n",
    "prompt = PromptTemplate(template=prompt_template,\n",
    "                        input_variables=['context'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03a9cdf9-4fa9-4e17-b20a-a1ef7f3445a3",
   "metadata": {},
   "source": [
    "## Create chain\n",
    "\n",
    "The document loader loads each PDF page as a separate 'document'.\n",
    "This is partly for technical reasons because that is the way PDFs are structured.\n",
    "Therefore, we use the chain called  `create_stuff_documents_chain` which joins multiple documents  into a single large document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c4e44b3-2059-4a8c-860a-932d3ed1a467",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = create_stuff_documents_chain(llm, prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2a917d7-bb6f-4e45-9f6d-2c6842598256",
   "metadata": {},
   "source": [
    "## Function to Separate  the Summary from the Input\n",
    "\n",
    "LangChain  returns both the input prompt and the generated response in one long text.\n",
    "To get only the summary, we must split the summary from the document that we sent as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a13ba30-5729-4551-9bf6-b88927b3251b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_result(result):\n",
    "    \"Split the reply from the prompt, should be done with output parser?\"\n",
    "    position = result.find(separator)\n",
    "    summary = result[position + len(separator) :]\n",
    "    return summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a3fa297-2be5-499c-8e4f-d9777925002f",
   "metadata": {},
   "source": [
    "## Loading the Documents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b58d5773-652f-4fa1-86f7-8058b62f61dd",
   "metadata": {},
   "source": [
    "We use LangChain’s `DirectoryLoader` to load all in files in `document_folder`.\n",
    "`document_folder` is defined at the start of this  Notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce54f747-8ddf-4781-aef9-a4a32dbffe3a",
   "metadata": {
    "editable": true,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import DirectoryLoader\n",
    "\n",
    "loader = DirectoryLoader(document_folder)\n",
    "documents = loader.load()\n",
    "print('number of documents:', len(documents))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "058c3c33-46d8-4bcc-83ff-9ba88e0fc500",
   "metadata": {},
   "source": [
    "## Creating the Summaries\n",
    "Now, we can iterate over these documents with a `for`-loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b4f1369-f45b-4ad5-a489-bd2c146c49ed",
   "metadata": {
    "editable": true,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [],
   "source": [
    "summaries = {}\n",
    "\n",
    "for document in documents:\n",
    "    filename = document.metadata['source']\n",
    "    print(filename)\n",
    "    summary = chain.invoke({\"context\": [document]})\n",
    "    summary = split_result(summary)\n",
    "    summaries[filename] = summary\n",
    "    print('Summary of file', filename)\n",
    "    print(summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "126ffc86-13f6-4c90-a699-bf23353ea578",
   "metadata": {},
   "source": [
    "## Saving the Summaries to Text Files\n",
    "Finally, we save the summaries for later use.\n",
    "We save all the summaries in the file `summaries.txt`.\n",
    "If you like, you can store each summary in a separate file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ef69d3f-07cd-4d88-ba92-5d729d4f0e81",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('summaries.txt', 'w') as outfile:\n",
    "    for filename in summaries:\n",
    "        print('Summary of ', filename, file = outfile)\n",
    "        print(summaries[filename], file=outfile)\n",
    "        print(file=outfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d135a84-c29d-4619-b2d7-cef07dac929f",
   "metadata": {},
   "source": [
    "## Bonus Material"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a0e2f6d-23ff-4feb-bd03-daa7b256e213",
   "metadata": {
    "editable": true,
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "::::{admonition} Make an Overall Summary\n",
    ":class: tip, dropdown\n",
    "\n",
    "We can also try to generate an overall summary of all the documents.\n",
    "This doesn't make much sense with documents on different topics.\n",
    "If all the documents are related or on the same topic, it could make sense to make an overall summary of all the summaries.\n",
    "\n",
    "First, we need to import some more functions:\n",
    "\n",
    "```python\n",
    "from langchain.schema.document import Document\n",
    "from langchain.prompts import ChatPromptTemplate\n",
    "```\n",
    "\n",
    "We make a new prompt, with more specific instructions than for the regular summaries.\n",
    "\n",
    "```python\n",
    "total_prompt = ChatPromptTemplate.from_messages(\n",
    "    [(\"system\", \"Below is a list of summaries of some papers. Make a total summary all the information in all the papers:\\n\\n{context}\\n\\nTotal Summary:\")]\n",
    ")\n",
    "```\n",
    "\n",
    "Then, we can make a new chain based on the LLM and the prompt:\n",
    "\n",
    "```python\n",
    "total_chain = create_stuff_documents_chain(llm, total_prompt)\n",
    "```\n",
    "\n",
    "This chain needs a list of  `Document` objects as input.\n",
    "\n",
    "\n",
    "```python\n",
    "list_of_summaries = [Document(summary) for summary in summaries.values()]\n",
    "```\n",
    "\n",
    "Now, we can invoke the chain with this list as input, and print the result:\n",
    "\n",
    "```python\n",
    "total_summary = total_chain.invoke({\"context\": list_of_summaries})\n",
    "\n",
    "print('Summary of all the summaries:')\n",
    "print(total_summary)\n",
    "```\n",
    "\n",
    "Finally, we save the overall summary to a text file:\n",
    "\n",
    "```python\n",
    "with open('total_summary.txt', 'w') as outfile:\n",
    "    print(total_summary, file=outfile)\n",
    "```\n",
    "::::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "882ee051-292e-4e37-9af9-7fa5299866b1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# Here is the code above in a single cell for testing\n",
    "from langchain.schema.document import Document\n",
    "from langchain.prompts import ChatPromptTemplate\n",
    "\n",
    "total_prompt = ChatPromptTemplate.from_messages(\n",
    "    [(\"system\", \"Below is a list of summaries of some papers. Make a total summary all the information in all the papers:\\n\\n{context}\\n\\nTotal Summary:\")]\n",
    ")\n",
    "total_chain = create_stuff_documents_chain(llm, total_prompt)\n",
    "list_of_summaries = [Document(summary) for summary in summaries.values()]\n",
    "total_summary = total_chain.invoke({\"context\": list_of_summaries})\n",
    "\n",
    "print('Summary of all the summaries:')\n",
    "print(total_summary)\n",
    "\n",
    "with open('total_summary.txt', 'w') as outfile:\n",
    "    print(total_summary, file=outfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6f128d6-2894-4034-8e96-54d872702ca1",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}