lavrov_gpt2_trained.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "lavrov_gpt2_trained.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "t3he1QdbwZlN",
        "colab_type": "code",
        "outputId": "75c9be7e-e2b8-4da3-c7b3-8c1d4ef145df",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        }
      },
      "source": [
        "#make sure you are running on a GPU, preferrably not the free GPU with collab as your session may timeout\n",
        "#nvidia p100 recommended\n",
        "!nvidia-smi                                                                                  "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sun May  3 18:05:26 2020       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 440.64.00    Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   37C    P0    26W / 250W |      0MiB / 16280MiB |      0%      Default |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                       GPU Memory |\n",
            "|  GPU       PID   Type   Process name                             Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_ruJJCUTn1E0",
        "colab_type": "code",
        "outputId": "78bd0f0c-fbfb-4fda-ab04-35da682a2e31",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "source": [
        "#this cell loads the older version of tensorflow due to gpt-2 simple import constraints\n",
        "%tensorflow_version 1.x\n",
        "!pip install -q gpt-2-simple\n",
        "import gpt_2_simple as gpt2\n",
        "from datetime import datetime\n",
        "from google.colab import files"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 1.x selected.\n",
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mBQPDYt4vD6r",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "#get current working directory\n",
        "import json\n",
        "import string\n",
        "articles_string = ''\n",
        "#produces a further cleaning of scraped data result should include date and article text only\n",
        "with open(\"/content/drive/My Drive/lavrov.json\") as f:\n",
        "    jsonString = f.read()\n",
        "    jsonData = json.loads(jsonString)\n",
        "    for i in range(0, len(jsonData)):\n",
        "      cleaned_articletext= jsonData[i]['ArticleText'].replace('Toggle navigation      / / Asset Publisher' ,'')\n",
        "      cleaned_articletext = cleaned_articletext.replace('Advanced settings         Switcher      6 Photos close','')\n",
        "      articles_string = articles_string + ' ' + cleaned_articletext\n",
        "lavrovwritefile=open(\"lavrov_cleaned.txt\",'w')\n",
        "lavrovwritefile.write(articles_string)\n",
        "lavrovwritefile.close()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SrhjbrgLqpST",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#code adpated from google collab tutorial on importing and running gpt-2 (https://colab.research.google.com/drive/1VLG8e7YSEwypxU-noRNhsv5dW4NfTGce)\n",
        "!pip install gpt2_client\n",
        "from gpt2_client import GPT2Client\n",
        "#download gpt2 model, or use one cached in content->gpt-s->models\n",
        "gpt2 = GPT2Client('345M') # This could also be '117', '345M', '774M', or '1558M'.\n",
        "gpt2.load_model(force_download=False) # Use cached versions if available."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Z2vWsMXq8Uu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from gpt2_client import GPT2Client\n",
        "#training on anything larger than 345 will take a very long time. ~5-10 hours depending on model\n",
        "gpt2 = GPT2Client('345M') # This could also be `345M`, `774M`, or `1558M`\n",
        "\n",
        "\n",
        "my_corpus = '/content/drive/My Drive/lavrov_cleaned.txt' # path to corpus\n",
        "custom_text = gpt2.finetune(my_corpus, return_text=True) # Load your custom dataset"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ACl2CU4jsFl_",
        "colab_type": "code",
        "outputId": "b06774aa-73a0-43fc-c835-3661224763f8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "#----------------------------------------------------------------------------------------------------------------------------------\n",
        "#run this cell to import the trained model from google drive. It should be located within the drive and not on the temporary VM shell\n",
        "import gpt_2_simple as gpt2\n",
        "gpt2.copy_checkpoint_from_gdrive(run_name='run1')\n",
        "sess = gpt2.start_tf_sess()\n",
        "gpt2.load_gpt2(sess, run_name='run1')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading checkpoint checkpoint/run1/model-1000\n",
            "INFO:tensorflow:Restoring parameters from checkpoint/run1/model-1000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HWVbJDUC63SC",
        "colab_type": "code",
        "outputId": "465cfb17-abc0-45d0-a189-4308c4c98b86",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        }
      },
      "source": [
        "#optional arguments:\n",
        "#    Interactively run the model (from https://github.com/openai/gpt-2/blob/master/src/interactive_conditional_samples.py)\n",
        "#    :model_name=124M : String, which model to use\n",
        "#    :seed=None : Integer seed for random number generators, fix seed to reproduce\n",
        "#     results\n",
        "#    :nsamples=1 : Number of samples to return total\n",
        "#    :batch_size=1 : Number of batches (only affects speed/memory).  Must divide nsamples.\n",
        "#    :length=None : Number of tokens in generated text, if None (default), is\n",
        "#     determined by model hyperparameters\n",
        "#    :temperature=1 : Float value controlling randomness in boltzmann\n",
        "#     distribution. Lower temperature results in less random completions. As the\n",
        "#     temperature approaches zero, the model will become deterministic and\n",
        "#     repetitive. Higher temperature results in more random completions.\n",
        "#    :top_k=0 : Integer value controlling diversity. 1 means only 1 word is\n",
        "#     considered for each step (token), resulting in deterministic completions,\n",
        "#     while 40 means 40 words are considered at each step. 0 (default) is a\n",
        "#     special setting meaning no restrictions. 40 generally is a good value.\n",
        "#     :models_dir : path to parent folder containing model subfolders\n",
        "#     (i.e. contains the <model_name> folder)\n",
        "#\n",
        "#\n",
        "#\n",
        "#\n",
        "#\n",
        "gpt2.generate(sess,\n",
        "              checkpoint_dir='/content/checkpoint',\n",
        "              length=250,\n",
        "              temperature = .8,\n",
        "              nsamples=5,\n",
        "              batch_size=5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "In an unprecedented attempt to mislead the public, the Russian government is attempting to portray the activities of Gazprom International as having anti-Russian content. Is there a word for it in just about any matters, including international affairs? This unfortunately violates the principles of international coordination and coordination of legitimate concerns. International issues are the subject of a constitutional convention, a decision taken by the UN Security Council, designed to ensure a balance of interests at the negotiating table. Thus, in addition to representing the interests of Russian companies, the convention is also being used to further strengthen economic and social relations between our two countries. This is a serious cause for concern. The Russian Federation's proposal to hold a constitutional conference to draft the document on holding a conference on holding an international conference on the issue of holding of a constitutional conference involving all the foreign participants with the participation of the participants of permanent seats on the UN Security Council. The Constitutional Convention must take place with the participation of representatives of civil society, activists and representatives of constitutional experts. If the Convention witnesses the acts of serious authoritarianism, it will be considered illegal and, possibly, even dangerous. I am referring to the November meeting of the constitutional convention, which was held under the command of the Georgian government. The Georgian\n",
            "====================\n",
            "American billionaires and corporatists are desperately trying to influence the current election to the United States Congress, including funding it with money borrowed from the Federal Reserve. The recent money shortage in the US has provided a perfect opportunity for ruling-class politicians and holders of the money, which is legally but not legally available to them, to finally understand the desperate situation of the global financial system. This is mainly a government sponsored crisis. It is obvious that they cannot continue about such dangerous policies. They are simply unable to make it through this Congress without being dragged into it. The United States Congress remains a major part of the system of global finance. We have been witnessing as well as understanding the efforts of the big US banks and corporations to create and maintain a hide-out for their clients. Congress has become the most exclusive forum for discussions and the \"reference\" session for their attempts to implement their \"global solutions to the problems of the 21st century\". This is a serious situation when the current Congress representatives, be it the House of Representatives or the Senate, are also trying to implement their own unreformed policies. This is a far from easy task, especially since the United States Congress is dominated by members who are active members of the banking industry, who are\n",
            "====================\n",
            "Media playback is unsupported on your device Media caption Claireing in the Baltic Sea: a tragic war that has not ended yet?\n",
            "\n",
            "Over 80 people have died and over 3,000 have been injured in the Baltic Sea, which has been a hotbed of tensions for many years.\n",
            "\n",
            "As a result of the outbreak of the war in this region, some countries of the European Union have reintroduced visa requirements, and this has resulted in a further increase of tensions as well. All this has created a severe emergency situation.\n",
            "\n",
            "There are some pressing issues on the European agenda; however, for example, our relations with the United States, Europe's main economic partner, are being seriously affected by the actions of the United States and the European Union. In this respect the issue of security on the Eastern borders of the US, is the topic that we are fighting.\n",
            "\n",
            "As you know, Russian President Vladimir Putin has repeatedly made his position on the Middle East settlement process. Now it is time to go into the details. A conflict between Israel and Palestine is another one of these key issues. It is illegal to interfere in the Middle East settlement process, and the current European Union has done it. Regarding the region, I would like to\n",
            "====================\n",
            "We have decided to stick to our agreements and to come up with a new proposal.  The whole package applies to all the discrepancies between the Russian side and the involvement of Western countries in the negotiation of the Transnistian problem.  The EU and the US have proposed that there should be no further negotiations on the Transnistian issues, and they asked for the preparation of a statement with these stages.  We have not heard anything about this.  One thing we are convinced of - a refusal of the European Union and the US to agree on a new agreement reflects their own interests and is very dangerous, if these countries came to hold talks with us.  The European Union is actually trying to remove the Ukrainian leadership, organise a coup and use the situation for their own purposes, and it is possible that the Minsk Agreements were not agreed upon by activists in Kiev, but were barely implemented, the so-called \"eastern regions\" did not get any help from them and they were regularly hit (they got the first one!), so now they have decided to form their own government, and if it falls on its own, the other governments will not have any alternative.  Will the Ukrainian president sign this legal agreement? \n",
            "====================\n",
            "There are a lot of things that can happen in the next six months. It depends on the first day of the new election. What is our tie-up between Moscow and Oslo in the new situation?  The first day of the new Russian-Norwegian election campaign is in full swing. We will be ready for the same. There is a very important teamwork and serious cooperation between Moscow and Oslo. Of course, on the first day of the election we will be talking about our ties in the fight against terrorism. I am confident that we will touch upon the issues related to our joint efforts with the Norwegian authorities to fight extremism and the promotion of dialogue between Russia and Norway. This is one of the priorities of our bilateral relations. We will certainly provide a detailed answer to the questions that are being asked now about Russia's inhuman actions against Russian citizens and the activities of Russian border guards crossing into Norway. The Norwegian authorities are able to fulfil their obligations under international law and the UN Charter, providing for their immediate transfer into the Russian system. This is a very important issue, as it is connected with the key concerns of security in the Nordic countries, whose borders are under international control. It is not accidental that the launching of the ceasefire agreement between the\n",
            "====================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xYV1BQep-88S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}