Publication_Integrity.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "id": "25TB5xWC-BBd",
      "metadata": {
        "id": "25TB5xWC-BBd"
      },
      "source": [
        "# Team              : CTRL C CTRL V\n",
        "# Track Company     : Track-1 Cactus Communications\n",
        "# Track Name        : Paperpal - Future of Academic Writing\n",
        "# Chec Type         : Content Centric Checks\n",
        "# Check Implemented : Publication Integrity"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "id": "ac8db6b5",
      "metadata": {},
      "source": [
        "### Salient Features: \n",
        "\n",
        "1. Typo errors are flagged and checked, by converting a pdf and preprocessing data using series of  preprocessors, autocorrects and spell checkers.\n",
        "2. Plagiarism Checker uses web scraping from Wikipedia, assumed to be the authentic source, converted to a corpus, passed through a series of preprocessors and uses state-of-the art doc2Vec model. The model is trained and evaluated for, against our pdf/docx, uses co-sine similarity and produces results in decreasing sorted order of similarity."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b4812e73",
      "metadata": {},
      "source": [
        "# Typo"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "LGjHnAhZG5XW",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LGjHnAhZG5XW",
        "outputId": "2b7b2dc7-0dab-4be4-cba4-4531ea6daa4d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting PyPDF2\n",
            "  Downloading pypdf2-3.0.1-py3-none-any.whl (232 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m232.6/232.6 KB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: typing_extensions>=3.10.0.0 in /usr/local/lib/python3.8/dist-packages (from PyPDF2) (4.5.0)\n",
            "Installing collected packages: PyPDF2\n",
            "Successfully installed PyPDF2-3.0.1\n"
          ]
        }
      ],
      "source": [
        "!pip install PyPDF2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "R5v7gHYSRMsb",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R5v7gHYSRMsb",
        "outputId": "d7b09148-cc34-4097-f905-458ffa69bf3a"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading collection 'all'\n",
            "[nltk_data]    | \n",
            "[nltk_data]    | Downloading package abc to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/abc.zip.\n",
            "[nltk_data]    | Downloading package alpino to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/alpino.zip.\n",
            "[nltk_data]    | Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
            "[nltk_data]    | Downloading package averaged_perceptron_tagger_ru to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping\n",
            "[nltk_data]    |       taggers/averaged_perceptron_tagger_ru.zip.\n",
            "[nltk_data]    | Downloading package basque_grammars to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping grammars/basque_grammars.zip.\n",
            "[nltk_data]    | Downloading package bcp47 to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package biocreative_ppi to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/biocreative_ppi.zip.\n",
            "[nltk_data]    | Downloading package bllip_wsj_no_aux to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping models/bllip_wsj_no_aux.zip.\n",
            "[nltk_data]    | Downloading package book_grammars to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping grammars/book_grammars.zip.\n",
            "[nltk_data]    | Downloading package brown to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/brown.zip.\n",
            "[nltk_data]    | Downloading package brown_tei to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/brown_tei.zip.\n",
            "[nltk_data]    | Downloading package cess_cat to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/cess_cat.zip.\n",
            "[nltk_data]    | Downloading package cess_esp to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/cess_esp.zip.\n",
            "[nltk_data]    | Downloading package chat80 to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/chat80.zip.\n",
            "[nltk_data]    | Downloading package city_database to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/city_database.zip.\n",
            "[nltk_data]    | Downloading package cmudict to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/cmudict.zip.\n",
            "[nltk_data]    | Downloading package comparative_sentences to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/comparative_sentences.zip.\n",
            "[nltk_data]    | Downloading package comtrans to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package conll2000 to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/conll2000.zip.\n",
            "[nltk_data]    | Downloading package conll2002 to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/conll2002.zip.\n",
            "[nltk_data]    | Downloading package conll2007 to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package crubadan to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/crubadan.zip.\n",
            "[nltk_data]    | Downloading package dependency_treebank to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/dependency_treebank.zip.\n",
            "[nltk_data]    | Downloading package dolch to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/dolch.zip.\n",
            "[nltk_data]    | Downloading package europarl_raw to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/europarl_raw.zip.\n",
            "[nltk_data]    | Downloading package extended_omw to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    | Downloading package floresta to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/floresta.zip.\n",
            "[nltk_data]    | Downloading package framenet_v15 to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/framenet_v15.zip.\n",
            "[nltk_data]    | Downloading package framenet_v17 to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/framenet_v17.zip.\n",
            "[nltk_data]    | Downloading package gazetteers to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/gazetteers.zip.\n",
            "[nltk_data]    | Downloading package genesis to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/genesis.zip.\n",
            "[nltk_data]    | Downloading package gutenberg to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/gutenberg.zip.\n",
            "[nltk_data]    | Downloading package ieer to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/ieer.zip.\n",
            "[nltk_data]    | Downloading package inaugural to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/inaugural.zip.\n",
            "[nltk_data]    | Downloading package indian to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/indian.zip.\n",
            "[nltk_data]    | Downloading package jeita to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package kimmo to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/kimmo.zip.\n",
            "[nltk_data]    | Downloading package knbc to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package large_grammars to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping grammars/large_grammars.zip.\n",
            "[nltk_data]    | Downloading package lin_thesaurus to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/lin_thesaurus.zip.\n",
            "[nltk_data]    | Downloading package mac_morpho to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/mac_morpho.zip.\n",
            "[nltk_data]    | Downloading package machado to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package masc_tagged to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package maxent_ne_chunker to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping chunkers/maxent_ne_chunker.zip.\n",
            "[nltk_data]    | Downloading package maxent_treebank_pos_tagger to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping taggers/maxent_treebank_pos_tagger.zip.\n",
            "[nltk_data]    | Downloading package moses_sample to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping models/moses_sample.zip.\n",
            "[nltk_data]    | Downloading package movie_reviews to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/movie_reviews.zip.\n",
            "[nltk_data]    | Downloading package mte_teip5 to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/mte_teip5.zip.\n",
            "[nltk_data]    | Downloading package mwa_ppdb to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping misc/mwa_ppdb.zip.\n",
            "[nltk_data]    | Downloading package names to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/names.zip.\n",
            "[nltk_data]    | Downloading package nombank.1.0 to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package nonbreaking_prefixes to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/nonbreaking_prefixes.zip.\n",
            "[nltk_data]    | Downloading package nps_chat to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/nps_chat.zip.\n",
            "[nltk_data]    | Downloading package omw to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package omw-1.4 to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package opinion_lexicon to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/opinion_lexicon.zip.\n",
            "[nltk_data]    | Downloading package panlex_swadesh to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    | Downloading package paradigms to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/paradigms.zip.\n",
            "[nltk_data]    | Downloading package pe08 to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/pe08.zip.\n",
            "[nltk_data]    | Downloading package perluniprops to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping misc/perluniprops.zip.\n",
            "[nltk_data]    | Downloading package pil to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/pil.zip.\n",
            "[nltk_data]    | Downloading package pl196x to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/pl196x.zip.\n",
            "[nltk_data]    | Downloading package porter_test to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping stemmers/porter_test.zip.\n",
            "[nltk_data]    | Downloading package ppattach to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/ppattach.zip.\n",
            "[nltk_data]    | Downloading package problem_reports to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/problem_reports.zip.\n",
            "[nltk_data]    | Downloading package product_reviews_1 to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/product_reviews_1.zip.\n",
            "[nltk_data]    | Downloading package product_reviews_2 to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/product_reviews_2.zip.\n",
            "[nltk_data]    | Downloading package propbank to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package pros_cons to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/pros_cons.zip.\n",
            "[nltk_data]    | Downloading package ptb to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/ptb.zip.\n",
            "[nltk_data]    | Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data]    | Downloading package qc to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/qc.zip.\n",
            "[nltk_data]    | Downloading package reuters to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package rslp to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping stemmers/rslp.zip.\n",
            "[nltk_data]    | Downloading package rte to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/rte.zip.\n",
            "[nltk_data]    | Downloading package sample_grammars to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping grammars/sample_grammars.zip.\n",
            "[nltk_data]    | Downloading package semcor to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package senseval to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/senseval.zip.\n",
            "[nltk_data]    | Downloading package sentence_polarity to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/sentence_polarity.zip.\n",
            "[nltk_data]    | Downloading package sentiwordnet to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/sentiwordnet.zip.\n",
            "[nltk_data]    | Downloading package shakespeare to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/shakespeare.zip.\n",
            "[nltk_data]    | Downloading package sinica_treebank to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/sinica_treebank.zip.\n",
            "[nltk_data]    | Downloading package smultron to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/smultron.zip.\n",
            "[nltk_data]    | Downloading package snowball_data to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    | Downloading package spanish_grammars to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping grammars/spanish_grammars.zip.\n",
            "[nltk_data]    | Downloading package state_union to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/state_union.zip.\n",
            "[nltk_data]    | Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/stopwords.zip.\n",
            "[nltk_data]    | Downloading package subjectivity to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/subjectivity.zip.\n",
            "[nltk_data]    | Downloading package swadesh to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/swadesh.zip.\n",
            "[nltk_data]    | Downloading package switchboard to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/switchboard.zip.\n",
            "[nltk_data]    | Downloading package tagsets to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping help/tagsets.zip.\n",
            "[nltk_data]    | Downloading package timit to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/timit.zip.\n",
            "[nltk_data]    | Downloading package toolbox to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/toolbox.zip.\n",
            "[nltk_data]    | Downloading package treebank to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/treebank.zip.\n",
            "[nltk_data]    | Downloading package twitter_samples to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/twitter_samples.zip.\n",
            "[nltk_data]    | Downloading package udhr to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/udhr.zip.\n",
            "[nltk_data]    | Downloading package udhr2 to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/udhr2.zip.\n",
            "[nltk_data]    | Downloading package unicode_samples to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/unicode_samples.zip.\n",
            "[nltk_data]    | Downloading package universal_tagset to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping taggers/universal_tagset.zip.\n",
            "[nltk_data]    | Downloading package universal_treebanks_v20 to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    | Downloading package vader_lexicon to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    | Downloading package verbnet to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/verbnet.zip.\n",
            "[nltk_data]    | Downloading package verbnet3 to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/verbnet3.zip.\n",
            "[nltk_data]    | Downloading package webtext to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/webtext.zip.\n",
            "[nltk_data]    | Downloading package wmt15_eval to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping models/wmt15_eval.zip.\n",
            "[nltk_data]    | Downloading package word2vec_sample to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping models/word2vec_sample.zip.\n",
            "[nltk_data]    | Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package wordnet2021 to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package wordnet2022 to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/wordnet2022.zip.\n",
            "[nltk_data]    | Downloading package wordnet31 to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package wordnet_ic to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/wordnet_ic.zip.\n",
            "[nltk_data]    | Downloading package words to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/words.zip.\n",
            "[nltk_data]    | Downloading package ycoe to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/ycoe.zip.\n",
            "[nltk_data]    | \n",
            "[nltk_data]  Done downloading collection all\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import nltk\n",
        "nltk.download('all')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "iU-YQObfU8C8",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iU-YQObfU8C8",
        "outputId": "d0a1b4c6-b287-49a1-9902-867d0a07b939"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting autocorrect\n",
            "  Downloading autocorrect-2.6.1.tar.gz (622 kB)\n",
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/622.8 KB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m286.7/622.8 KB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m622.8/622.8 KB\u001b[0m \u001b[31m11.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Building wheels for collected packages: autocorrect\n",
            "  Building wheel for autocorrect (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for autocorrect: filename=autocorrect-2.6.1-py3-none-any.whl size=622381 sha256=4d01c356fe5724e416e50c2972f9a3a3ecfe0ac7b7dff19773157c5a430c2ec3\n",
            "  Stored in directory: /root/.cache/pip/wheels/72/b8/3b/a90246d13090e85394a8a44b78c8abf577c0766f29d6543c75\n",
            "Successfully built autocorrect\n",
            "Installing collected packages: autocorrect\n",
            "Successfully installed autocorrect-2.6.1\n"
          ]
        }
      ],
      "source": [
        "!pip install autocorrect"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "iMUySjknGw8Q",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iMUySjknGw8Q",
        "outputId": "84c0a6f1-3858-4d43-dec4-eb296c8d9a6f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['ieee', 'malicious', 'detection', 'using', 'multilayer', 'filtering', 'model', 'rajesh', 'kumar', 'xiaosong', 'zhang', 'hussain', 'ahmad', 'tariq', 'riaz', 'ullah', 'khan', 'school', 'computer', 'science', 'engineering', 'university', 'electro', 'science', 'technology', 'china', 'email', 'rajakumarlohanogmailcom', 'rerukhanoutlookcom', 'abstract', 'malicious', 'urls', 'harmful', 'every', 'aspect', 'computer', 'users', 'detecting', 'malicious', 'important', 'currently', 'detection', 'malicious', 'webpages', 'techniques', 'includes', 'blacklist', 'whitelist', 'methodology', 'machine', 'learning', 'classification', 'algorithms', 'used', 'however', 'blacklist', 'whitelist', 'technology', 'useless', 'particular', 'list', 'paper', 'propose', 'multilayer', 'model', 'detecting', 'malicious', 'filter', 'directly', 'determine', 'training', 'threshold', 'layer', 'filter', 'reaches', 'threshold', 'otherwise', 'filter', 'also', 'used', 'example', 'verify', 'model', 'improve', 'accuracy', 'detection', 'keywords', 'malicious', 'blacklist', 'whitelist', 'tech', 'nology', 'machine', 'learning', 'multilayer', 'filtering', 'model', 'introduction', 'recent', 'years', 'internet', 'playing', 'bigger', 'bigger', 'role', 'peoples', 'work', 'life', 'currently', 'observed', 'every', 'website', 'userfriendly', 'profitable', 'malicious', 'websites', 'began', 'appear', 'malicious', 'websites', 'endangering', 'aspects', 'user', 'lead', 'user', 'towards', 'economic', 'losses', 'even', 'create', 'confusion', 'management', 'country', 'detecting', 'stopping', 'malicious', 'websites', 'become', 'important', 'control', 'measure', 'avoid', 'risk', 'information', 'security', 'generally', 'entrance', 'website', 'malicious', 'level', 'entrance', 'identification', 'information', 'loss', 'malicious', 'identification', 'information', 'security', 'spam', 'malicious', 'webpages', 'urls', 'redirect', 'mislead', 'legitimate', 'users', 'malware', 'scams', 'adult', 'content', 'perhaps', 'correlated', 'internet', 'identify', 'malicious', 'urls', 'based', 'classifies', 'draw', 'features', 'webpages', 'content', 'lexical', 'visual', 'lexical', 'features', 'redirect', 'paths', 'host', 'based', 'features', 'combinations', 'classifies', 'usually', 'conjunction', 'knowledge', 'bases', 'usually', 'browser', 'service', 'providers', 'classifier', 'urlbased', 'features', 'extracting', 'features', 'mostly', 'using', 'supervised', 'learning', 'paradigm', 'naive', 'bases', 'support', 'actor', 'machine', 'different', 'kernels', 'logistic', 'regression', 'popular', 'machine', 'learning', 'classifies', 'filtering', 'spam', 'phishing', 'meanwhile', 'based', 'learning', 'deal', 'active', 'attackers', 'also', 'evaluated', 'spam', 'filtering', 'paper', 'solve', 'problem', 'using', 'multilayer', 'filtering', 'model', 'classification', 'able', 'handle', 'relatively', 'good', 'data', 'multilayer', 'filter', 'model', 'malicious', 'detection', 'typical', 'classification', 'application', 'scenario', 'malicious', 'part', 'machine', 'learning', 'classification', 'algorithms', 'useful', 'wide', 'range', 'applications', 'also', 'applied', 'malicious', 'detection', 'scenarios', 'order', 'give', 'brief', 'overview', 'advantages', 'classifier', 'article', 'designed', 'malicious', 'multi', 'layer', 'detection', 'model', 'model', 'mainly', 'composed', 'layer', 'classifier', 'classifies', 'also', 'called', 'filters', 'model', 'model', 'consists', 'layer', 'filter', 'composition', 'stratified', 'filter', 'black', 'white', 'list', 'filter', 'models', 'firstlevel', 'filter', 'blackandwhite', 'list', 'filter', 'validated', 'recognizing', 'normal', 'urls', 'malicious', 'urls', 'normal', 'addresses', 'stored', 'white', 'list', 'file', 'blacklist', 'file', 'detect', 'traverse', 'list', 'black', 'white', 'determine', 'whether', 'black', 'list', 'white', 'list', 'naive', 'bayesian', 'filter', 'second', 'layer', 'filter', 'model', 'naive', 'bayesian', 'filter', 'trains', 'model', 'authorized', 'licensed', 'limited', 'nirma', 'university', 'institute', 'technology', 'downloaded', 'september', 'ieee', 'xplore', 'restrictions', 'apply', 'dividing', 'main', 'steps', 'training', 'samples', 'twodimensional', 'arrays', 'store', 'probability', 'value', 'malicious', 'website', 'normal', 'website', 'formula', 'alpha', 'names', 'threshold', 'training', 'nbayesd', 'represent', 'respectively', 'calculated', 'naive', 'bayesian', 'formula', 'model', 'malicious', 'normal', 'probability', 'value', 'size', 'nbayes', 'represent', 'classification', 'judgment', 'credibility', 'names', 'describes', 'belongs', 'categories', 'probability', 'much', 'greater', 'probability', 'belonging', 'another', 'class', 'nbayes', 'arrives', 'certain', 'threshold', 'size', 'consider', 'naive', 'bayesian', 'good', 'data', 'threshold', 'small', 'reached', 'ideal', 'classification', 'accuracy', 'threshold', 'large', 'naive', 'shellfish', 'data', 'yeast', 'good', 'filters', 'small', 'article', 'discusses', 'another', 'group', 'training', 'data', 'train', 'threshold', 'specific', 'method', 'discussed', 'briefly', 'section', 'assume', 'appropriate', 'training', 'threshold', 'isnbayesd', 'upper', 'filter', 'detect', 'traine', 'naive', 'bayesian', 'model', 'calculate', 'names', 'names', 'nbayesdd', 'think', 'good', 'bayesian', 'model', 'good', 'data', 'otherwise', 'cannot', 'considered', 'good', 'data', 'naive', 'bayesian', 'model', 'cannot', 'determine', 'nature', 'record', 'classification', 'results', 'move', 'next', 'filte', 'cart', 'decision', 'tree', 'filter', 'models', 'thirdlevel', 'filter', 'cart', 'decision', 'tree', 'filter', 'model', 'split', 'steps', 'model', 'training', 'cart', 'tree', 'constructed', 'training', 'samples', 'urls', 'leaf', 'nodes', 'decision', 'nodes', 'store', 'cart', 'tree', 'file', 'system', 'cart', 'threshold', 'training', 'maxn', 'nnbayesd', 'represents', 'number', 'malicious', 'leaf', 'nodes', 'represents', 'number', 'normal', 'leaf', 'nodes', 'size', 'cart', 'characterize', 'type', 'occupation', 'leaf', 'node', 'decides', 'similar', 'naive', 'bayesian', 'filter', 'threshold', 'neither', 'small', 'large', 'train', 'threshold', 'training', 'data', 'group', 'specific', 'method', 'discussed', 'section', 'ifnbayes', 'nbayesdd', 'think', 'cart', 'decision', 'tree', 'model', 'good', 'data', 'otherwise', 'record', 'classification', 'results', 'filtered', 'next', 'filter', 'final', 'filter', 'model', 'filter', 'training', 'model', 'mainly', 'classified', 'models', 'derived', 'classification', 'function', 'upper', 'layer', 'filter', 'record', 'classification', 'results', 'combined', 'naive', 'bayesian', 'filter', 'cart', 'decision', 'tree', 'filtering', 'collectively', 'determine', 'classification', 'instance', 'validation', 'data', 'sources', 'experimental', 'environment', 'malicious', 'dataset', 'experiment', 'downloaded', 'malicious', 'website', 'httpwwwmwslorgcn', 'normal', 'dataset', 'collected', 'first', 'category', 'directory', 'httpwwwdir', 'samples', 'taken', 'dataset', 'malicious', 'urls', 'normal', 'urls', 'article', 'uses', 'features', 'extraction', 'data', 'modeling', 'python', 'language', 'implementation', 'programming', 'windows', 'operating', 'system', 'core', 'used', 'personal', 'computer', 'table', 'feature', 'vector', 'extraction', 'rules', 'features', 'domain', 'names', 'contained', 'consecutive', 'numbers', 'domain', 'name', 'contains', 'special', 'characters', 'five', 'domain', 'name', 'number', 'domain', 'name', 'domain', 'name', 'total', 'length', 'length', 'longest', 'domain', 'name', 'segment', 'meaningful', 'coefficients', 'primary', 'domain', 'names', 'feature', 'selection', 'garera', 'attain', 'make', 'comparisons', 'comprehensive', 'study', 'regarding', 'feature', 'selection', 'article', 'mainly', 'perspective', 'domain', 'name', 'cost', 'malicious', 'website', 'creator', 'knows', 'domain', 'name', 'site', 'great', 'risk', 'banned', 'purchase', 'authorized', 'licensed', 'limited', 'nirma', 'university', 'institute', 'technology', 'downloaded', 'september', 'ieee', 'xplore', 'restrictions', 'apply', 'buying', 'domain', 'name', 'often', 'cheaper', 'even', 'free', 'domains', 'cost', 'savings', 'name', 'domain', 'often', 'following', 'characteristics', 'mainstream', 'domain', 'name', 'domain', 'name', 'special', 'characters', 'domain', 'name', 'length', 'long', 'main', 'domain', 'consists', 'meaningless', 'letters', 'numbers', 'many', 'confuse', 'domain', 'name', 'structure', 'based', 'information', 'paper', 'chooses', 'seven', 'features', 'shown', 'table', 'calculation', 'method', 'table', 'follows', 'text', 'chooses', 'commonly', 'used', 'english', 'words', 'chinese', 'word', 'phonetic', 'together', 'constitute', 'meaningful', 'word', 'tree', 'string', 'main', 'domain', 'name', 'meaningful', 'list', 'tree', 'words', 'match', 'match', 'considered', 'match', 'characters', 'meaning', 'rules', 'characters', 'total', 'length', 'characters', 'ratio', 'meaning', 'coefficient', 'example', 'bookdsxihuancom', 'domain', 'name', 'contain', 'consecutive', 'number', 'domain', 'name', 'contain', 'following', 'special', 'words', 'example', 'contains', 'five', 'domain', 'names', 'domain', 'name', 'contains', 'therefore', 'total', 'length', 'domain', 'name', 'example', 'case', 'longest', 'domain', 'name', 'bookdsxihuan', 'length', 'main', 'domain', 'name', 'example', 'bookdsxihuan', 'meaningful', 'word', 'book', 'xihuan', 'match', 'meaningful', 'length', 'primary', 'domain', 'length', 'meaningful', 'coefficient', 'feature', 'vector', 'domain', 'name', 'expressed', 'table', 'data', 'dataset', 'name', 'dataset', 'description', 'training', 'model', 'dataset', 'malicious', 'normal', 'training', 'threshold', 'dataset', 'malicious', 'normal', 'testing', 'dataset', 'malicious', 'normal', 'table', 'test', 'results', 'various', 'models', 'model', 'name', 'accuracy', 'rate', 'rexall', 'rate', 'precise', 'rate', 'multilayer', 'filtering', 'model', 'simple', 'naive', 'bases', 'single', 'decision', 'tree', 'single', 'training', 'model', 'threshold', 'paper', 'divide', 'urls', 'collected', 'section', 'three', 'experiment', 'part', 'shown', 'table', 'tested', 'directly', 'judged', 'malicious', 'normal', 'experimental', 'tuning', 'process', 'consider', 'black', 'white', 'list', 'filter', 'blacklist', 'filter', 'considered', 'authoritative', 'model', 'experimental', 'process', 'follows', 'perform', 'data', 'three', 'datasets', 'using', 'feature', 'extraction', 'rules', 'section', 'train', 'separate', 'machine', 'learning', 'trainmodel', 'sample', 'classifies', 'including', 'naive', 'bayesian', 'model', 'cart', 'decision', 'tree', 'model', 'model', 'build', 'separate', 'machine', 'learning', 'model', 'multilayer', 'filter', 'model', 'beginning', 'give', 'large', 'threshold', 'pair', 'nbayes', 'cart', 'substitute', 'train', 'threshold', 'sample', 'multi', 'layer', 'filter', 'model', 'calculate', 'check', 'measurement', 'accuracy', 'gradually', 'reduce', 'thresholds', 'article', 'uses', 'formula', 'program', 'names', 'names', 'cart', 'cart', 'record', 'thresholds', 'accuracy', 'finally', 'pick', 'combination', 'highest', 'accuracy', 'function', 'names', 'card', 'substituting', 'threshold', 'combination', 'names', 'card', 'multilayer', 'filter', 'model', 'test', 'multilayer', 'filter', 'model', 'three', 'separate', 'test', 'sample', 'sets', 'classifies', 'record', 'accuracy', 'rate', 'recall', 'rate', 'accuracy', 'experiment', 'results', 'optimal', 'threshold', 'pair', 'trained', 'section', 'names', 'cart', 'results', 'shown', 'table', 'multilayer', 'filter', 'model', 'naive', 'bayesian', 'filter', 'determined', 'decision', 'tree', 'filter', 'determined', 'another', 'jointly', 'detected', 'three', 'filters', 'result', 'also', 'line', 'table', 'performance', 'separate', 'classification', 'models', 'seen', 'three', 'separate', 'models', 'decision', 'tree', 'model', 'comparatively', 'best', 'performing', 'model', 'naive', 'bases', 'also', 'observed', 'table', 'multilayer', 'filter', 'model', 'performs', 'better', 'three', 'classifier', 'models', 'multiplayer', 'filter', 'model', 'classifier', 'deal', 'good', 'data', 'every', 'layer', 'model', 'plays', 'beneficial', 'role', 'classification', 'urls', 'ultimately', 'improves', 'detection', 'malicious', 'terms', 'accuracy', 'authorized', 'licensed', 'limited', 'nirma', 'university', 'institute', 'technology', 'downloaded', 'september', 'ieee', 'xplore', 'restrictions', 'apply', 'conclusion', 'paper', 'black', 'white', 'list', 'technology', 'machine', 'learning', 'algorithms', 'used', 'formed', 'multi', 'layer', 'filtering', 'model', 'detection', 'malicious', 'urls', 'model', 'trained', 'machine', 'learning', 'algorithm', 'naive', 'bayesian', 'classification', 'decision', 'tree', 'classifier', 'threshold', 'threshold', 'used', 'refer', 'guide', 'classifies', 'filtering', 'combined', 'naive', 'bayesian', 'classifier', 'decision', 'tree', 'classifier', 'classifies', 'multiplayer', 'model', 'improve', 'malicious', 'detection', 'system', 'terms', 'accuracy', 'observed', 'real', 'examples', 'multiplayer', 'filtering', 'models', 'effective', 'detection', 'malicious', 'urls', 'references', 'hugh', 'chipman', 'edward', 'george', 'robert', 'mcculloch', 'bayesian', 'cart', 'model', 'search', 'journal', 'american', 'statistical', 'association', 'september', 'sujata', 'garera', 'niel', 'proves', 'monica', 'chew', 'aviel', 'rubin', 'framework', 'detection', 'measurement', 'phishing', 'attacks', 'meetings', 'workshop', 'recurring', 'malicious', 'code', 'worm', 'page', 'abhishek', 'gattani', 'anhai', 'doan', 'digvijay', 'lamba', 'nikesh', 'garera', 'mitul', 'tiwari', 'xiaoyong', 'chai', 'sanjib', 'subramaniam', 'anand', 'rajaraman', 'venky', 'harinarayan', 'entit', 'extraction', 'linking', 'classifica', 'tion', 'tagging', 'social', 'media', 'proceedings', 'vadb', 'endowment', 'august', 'david', 'lewis', 'naive', 'bases', 'forty', 'independence', 'assumption', 'information', 'retrieval', 'pages', 'justin', 'lawrence', 'saul', 'stefan', 'savage', 'geoffrey', 'voelker', 'learning', 'detect', 'malicious', 'urls', 'transactions', 'intelligent', 'systems', 'technology', 'april', 'fadi', 'thabtah', 'mater', 'aburrous', 'mahossain', 'keshav', 'dahal', 'telligent', 'phishing', 'detection', 'system', 'banking', 'using', 'fuzzy', 'data', 'mining', 'expert', 'systems', 'applications', 'ankush', 'meshram', 'christian', 'haas', 'anomaly', 'detection', 'industrial', 'networks', 'using', 'machine', 'learning', 'roadm', 'machine', 'learning', 'cyber', 'physical', 'systems', 'pages', 'springer', 'berlin', 'heidelberg', 'berlin', 'heidelberg', 'xuequn', 'wang', 'thompsontanya', 'jane', 'mcgill', 'security', 'begins', 'home', 'determinants', 'home', 'computer', 'mobile', 'device', 'security', 'behavior', 'computers', 'security', 'rstenberg', 'phillip', 'colla', 'cart', 'classification', 'regression', 'trees', 'algorithms', 'data', 'mining', 'teal', 'information', 'security', 'techniques', 'including', 'detection', 'interdiction', 'andor', 'mitigation', 'memory', 'injection', 'attacks', 'google', 'patents', 'kurt', 'thomas', 'chris', 'grier', 'justin', 'vern', 'paxson', 'dawn', 'song', 'design', 'evaluation', 'real', 'time', 'spam', 'filtering', 'service', 'ieee', 'symposium', 'security', 'privacy', 'sean', 'whalen', 'nathaniel', 'bogus', 'salvador', 'stolfo', 'model', 'aggregation', 'distributed', 'content', 'anomaly', 'detection', 'proceedings', 'workshop', 'artificial', 'intelligent', 'security', 'workshop', 'aisec', 'york', 'press', 'ying', 'yang', 'geoffrey', 'webb', 'discretization', 'naivebayes', 'learning', 'managing', 'discretization', 'bias', 'variance', 'machine', 'learning', 'authorized', 'licensed', 'limited', 'nirma', 'university', 'institute', 'technology', 'downloaded', 'september', 'ieee', 'xplore', 'restrictions', 'apply']\n"
          ]
        }
      ],
      "source": [
        "import PyPDF2\n",
        "import io\n",
        "import random\n",
        "\n",
        "pdfReader = PyPDF2.PdfReader(\"/content/Malicious_URL_detection_using_multi-layer_filtering_model.pdf\")\n",
        "\n",
        "num_pages = len(pdfReader.pages)\n",
        "count = 0\n",
        "text = \"\"\n",
        "  #The while loop will read each page.\n",
        "while count < num_pages:\n",
        "  pageObj = pdfReader.pages[count]\n",
        "  count +=1\n",
        "  text += pageObj.extract_text()\n",
        "\n",
        "import re\n",
        "\n",
        "words = re.findall(r\"[\\s\\w]\", text)\n",
        "words = \"\".join(words)\n",
        "words = words.lower()\n",
        "for i in range(len(words)):\n",
        "  if words[i].isdigit():\n",
        "    words = words[:i]+' '+words[i+1:]\n",
        "\n",
        "words = words.split()\n",
        "for i in words:\n",
        "  if len(i)<3:\n",
        "    words.remove(i)\n",
        "\n",
        "stop = nltk.corpus.stopwords.words('english')\n",
        "for i in stop:\n",
        "  while i in words:\n",
        "    words.remove(i)\n",
        "\n",
        "words = ' '.join(words)\n",
        "\n",
        "x = words.split()\n",
        "tokens = []\n",
        "for i in x:\n",
        "  if len(i)>3:\n",
        "    tokens.append(i)\n",
        "\n",
        "tags = nltk.pos_tag(tokens)\n",
        "\n",
        "from autocorrect import Speller\n",
        "spell = Speller()\n",
        "for i in range(len(tokens)):\n",
        "  if tags[i][1] not in ['NN', 'JJ', 'NNP']:\n",
        "    tokens[i] = spell(tokens[i])\n",
        "\n",
        "print(tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9ng8EfqbvSDY",
      "metadata": {
        "id": "9ng8EfqbvSDY"
      },
      "source": [
        "# Plag Check"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "13v6Uv7G2x1d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "13v6Uv7G2x1d",
        "outputId": "7dbc6db8-f18f-4390-d4c8-6d721f562658"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting wikipedia\n",
            "  Downloading wikipedia-1.4.0.tar.gz (27 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.8/dist-packages (from wikipedia) (4.6.3)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from wikipedia) (2.25.1)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (1.26.14)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (4.0.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2022.12.7)\n",
            "Building wheels for collected packages: wikipedia\n",
            "  Building wheel for wikipedia (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for wikipedia: filename=wikipedia-1.4.0-py3-none-any.whl size=11695 sha256=af56f9fe6bfe6308993cc107a5ca83f8c5e5d7ba10fa77c2e222f8ab1f4122b7\n",
            "  Stored in directory: /root/.cache/pip/wheels/07/93/05/72c05349177dca2e0ba31a33ba4f7907606f7ddef303517c6a\n",
            "Successfully built wikipedia\n",
            "Installing collected packages: wikipedia\n",
            "Successfully installed wikipedia-1.4.0\n"
          ]
        }
      ],
      "source": [
        "!pip install wikipedia"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "RctQSTI5BX96",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RctQSTI5BX96",
        "outputId": "d59757e7-0bce-40d9-d8af-04ddb6c479ca"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/wikipedia/wikipedia.py:389: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system (\"lxml\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n",
            "\n",
            "The code that caused this warning is on line 389 of the file /usr/local/lib/python3.8/dist-packages/wikipedia/wikipedia.py. To get rid of this warning, pass the additional argument 'features=\"lxml\"' to the BeautifulSoup constructor.\n",
            "\n",
            "  lis = BeautifulSoup(html).find_all('li')\n",
            "WARNING:gensim.models.base_any2vec:consider setting layer size to a multiple of 4 for greater performance\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[<WikipediaPage 'Institute of Electrical and Electronics Engineers'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Rajesh'>, <WikipediaPage 'Kumar'>, <WikipediaPage 'Gao Xiaosong'>, <WikipediaPage 'Hussein'>, <WikipediaPage 'Ahmad'>, <WikipediaPage 'Tariq'>, <WikipediaPage 'Riaz'>, <WikipediaPage 'Ulla'>, <WikipediaPage 'School'>, <WikipediaPage 'Engineering'>, <WikipediaPage 'University'>, <WikipediaPage 'Electricity'>, <WikipediaPage 'Technology'>, <WikipediaPage 'Mail'>, <WikipediaPage 'URL'>, <WikipediaPage 'Harmful'>, <WikipediaPage 'Expect'>, <WikipediaPage 'Music Detected'>, <WikipediaPage 'Importance'>, <WikipediaPage 'Web page'>, <WikipediaPage 'Include directive'>, <WikipediaPage 'Blacklisting'>, <WikipediaPage 'Whitelist'>, <WikipediaPage 'Methodology'>, <WikipediaPage 'Machine'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Algorithm'>, <WikipediaPage 'Blacklisting'>, <WikipediaPage 'Whitelist'>, <WikipediaPage 'Technology'>, <WikipediaPage 'Copula (linguistics)'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Music Detected'>, <WikipediaPage 'Finger'>, <WikipediaPage 'GiveDirectly'>, <WikipediaPage 'Determinative'>, <WikipediaPage 'Training'>, <WikipediaPage 'Lawyer'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Accuracy and precision'>, <WikipediaPage 'Blacklisting'>, <WikipediaPage 'Whitelist'>, <WikipediaPage 'Machine'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Holocene'>, <WikipediaPage 'Year'>, <WikipediaPage 'Interest'>, <WikipediaPage 'Play (activity)'>, <WikipediaPage '1'>, <WikipediaPage 'People'>, <WikipediaPage 'World'>, <WikipediaPage 'Time'>, <WikipediaPage 'Observation'>, <WikipediaPage 'Website'>, <WikipediaPage 'User Friendly'>, <WikipediaPage 'Profit (economics)'>, <WikipediaPage 'Website'>, <WikipediaPage 'Bogan, Iran'>, <WikipediaPage 'Website'>, <WikipediaPage 'Endangerment'>, <WikipediaPage 'Towards Zero'>, <WikipediaPage 'Economics'>, <WikipediaPage 'Management'>, <WikipediaPage 'Country'>, <WikipediaPage 'Music Detected'>, <WikipediaPage 'Stoping'>, <WikipediaPage 'Website'>, <WikipediaPage 'Importance'>, <WikipediaPage 'Information'>, <WikipediaPage 'Security'>, <WikipediaPage 'General officer'>, <WikipediaPage 'Website'>, <WikipediaPage 'Information'>, <WikipediaPage 'Information'>, <WikipediaPage 'Security'>, <WikipediaPage 'Web page'>, <WikipediaPage 'URL'>, <WikipediaPage 'Misleading graph'>, <WikipediaPage 'Malware'>, <WikipediaPage 'Adult'>, <WikipediaPage 'Correlation'>, <WikipediaPage 'Interest'>, <WikipediaPage 'URL'>, <WikipediaPage 'Vase'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Web page'>, <WikipediaPage 'Lexicon'>, <WikipediaPage 'Visual system'>, <WikipediaPage 'Lexicon'>, <WikipediaPage 'Vase'>, <WikipediaPage 'Combination'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Knowledge'>, <WikipediaPage 'Providers'>, <WikipediaPage 'Clarifier'>, <WikipediaPage 'Extract'>, <WikipediaPage 'Supervised learning'>, <WikipediaPage 'Paradigm'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Machine'>, <WikipediaPage 'Logistics'>, <WikipediaPage 'Machine'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Pish'>, <WikipediaPage 'Vase'>, <WikipediaPage 'Death'>, <WikipediaPage 'Attackers'>, <WikipediaPage 'Evaluation'>, <WikipediaPage 'Problem solving'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Handal'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Scenario'>, <WikipediaPage 'Machine'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Algorithm'>, <WikipediaPage 'We'>, <WikipediaPage 'Applied arts'>, <WikipediaPage 'Scenario'>, <WikipediaPage 'Clarifier'>, <WikipediaPage 'Design'>, <WikipediaPage 'Lawyer'>, <WikipediaPage 'Lawyer'>, <WikipediaPage 'Clarifier'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Train'>, <WikipediaPage 'Lawyer'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Stratified sampling'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Copula (linguistics)'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Model'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Black and white'>, <WikipediaPage 'Copula (linguistics)'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Recall (memory)'>, <WikipediaPage 'Norway'>, <WikipediaPage 'URL'>, <WikipediaPage 'URL'>, <WikipediaPage 'Norway'>, <WikipediaPage 'Address'>, <WikipediaPage 'Stored'>, <WikipediaPage 'Copula (linguistics)'>, <WikipediaPage 'Film'>, <WikipediaPage 'Blacklisting'>, <WikipediaPage 'Film'>, <WikipediaPage 'Copula (linguistics)'>, <WikipediaPage 'Determinative'>, <WikipediaPage 'Interrogative word'>, <WikipediaPage 'Copula (linguistics)'>, <WikipediaPage 'Copula (linguistics)'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Second'>, <WikipediaPage 'Lawyer'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Train'>, <WikipediaPage 'Authorization'>, <WikipediaPage 'License'>, <WikipediaPage 'University'>, <WikipediaPage 'Institute'>, <WikipediaPage 'Technology'>, <WikipediaPage 'Download'>, <WikipediaPage 'September'>, <WikipediaPage 'Institute of Electrical and Electronics Engineers'>, <WikipediaPage 'Training'>, <WikipediaPage 'Plane (mathematics)'>, <WikipediaPage 'Array'>, <WikipediaPage 'Probability'>, <WikipediaPage 'Volume'>, <WikipediaPage 'Website'>, <WikipediaPage 'Norway'>, <WikipediaPage 'Website'>, <WikipediaPage 'Formula'>, <WikipediaPage 'Alpha'>, <WikipediaPage 'Name'>, <WikipediaPage 'Training'>, <WikipediaPage 'Present'>, <WikipediaPage 'Spanish language'>, <WikipediaPage 'Calculated'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Formula'>, <WikipediaPage 'Norway'>, <WikipediaPage 'Probability'>, <WikipediaPage 'Volume'>, <WikipediaPage 'Time'>, <WikipediaPage 'Present'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Judgement'>, <WikipediaPage 'Credibility'>, <WikipediaPage 'Name'>, <WikipediaPage 'Describing function'>, <WikipediaPage 'Probability'>, <WikipediaPage 'Probability'>, <WikipediaPage 'Certainty'>, <WikipediaPage 'Time'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Reached'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Accuracy and precision'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Shellfish'>, <WikipediaPage 'East'>, <WikipediaPage 'Discus throw'>, <WikipediaPage 'Training'>, <WikipediaPage 'Debate'>, <WikipediaPage 'Résumé'>, <WikipediaPage 'Training'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Train'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Calculation'>, <WikipediaPage 'Name'>, <WikipediaPage 'Name'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'List of films considered the best'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Determinative'>, <WikipediaPage 'Nature'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Car'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Model'>, <WikipediaPage 'Tertiary education'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Car'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Sport'>, <WikipediaPage 'Training'>, <WikipediaPage 'Car'>, <WikipediaPage 'Training'>, <WikipediaPage 'URL'>, <WikipediaPage 'Car'>, <WikipediaPage 'Film'>, <WikipediaPage 'System'>, <WikipediaPage 'Car'>, <WikipediaPage 'Training'>, <WikipediaPage 'Number'>, <WikipediaPage 'Number'>, <WikipediaPage 'Norway'>, <WikipediaPage 'Time'>, <WikipediaPage 'Car'>, <WikipediaPage 'List of topics characterized as pseudoscience'>, <WikipediaPage 'Time'>, <WikipediaPage 'Code'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Training'>, <WikipediaPage 'Debate'>, <WikipediaPage 'Car'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Filtration'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Training'>, <WikipediaPage 'Model'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Lawyer'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Car'>, <WikipediaPage 'Collective'>, <WikipediaPage 'Determinative'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Experiment'>, <WikipediaPage 'Data set'>, <WikipediaPage 'Experiment'>, <WikipediaPage 'Download'>, <WikipediaPage 'Website'>, <WikipediaPage 'Norway'>, <WikipediaPage 'Data set'>, <WikipediaPage 'Collect'>, <WikipediaPage 'Take'>, <WikipediaPage 'Data set'>, <WikipediaPage 'URL'>, <WikipediaPage 'Norway'>, <WikipediaPage 'URL'>, <WikipediaPage 'Model'>, <WikipediaPage 'Language'>, <WikipediaPage 'Implementation'>, <WikipediaPage 'Window'>, <WikipediaPage 'Operating system'>, <WikipediaPage 'System'>, <WikipediaPage 'Code'>, <WikipediaPage 'Role'>, <WikipediaPage 'Name'>, <WikipediaPage 'Consecutive'>, <WikipediaPage 'Number'>, <WikipediaPage 'Name'>, <WikipediaPage 'Name'>, <WikipediaPage 'Number'>, <WikipediaPage 'Name'>, <WikipediaPage 'Name'>, <WikipediaPage 'Length'>, <WikipediaPage 'Length'>, <WikipediaPage 'Name'>, <WikipediaPage 'Semantics'>, <WikipediaPage 'UEFA coefficient'>, <WikipediaPage 'Name'>, <WikipediaPage 'Election'>, <WikipediaPage 'Comparison'>, <WikipediaPage 'Regarding Henry'>, <WikipediaPage 'Election'>, <WikipediaPage 'Name'>, <WikipediaPage 'Court'>, <WikipediaPage 'Website'>, <WikipediaPage 'Knowledge'>, <WikipediaPage 'Name'>, <WikipediaPage 'Purchasing'>, <WikipediaPage 'Authorization'>, <WikipediaPage 'License'>, <WikipediaPage 'University'>, <WikipediaPage 'Institute'>, <WikipediaPage 'Technology'>, <WikipediaPage 'Download'>, <WikipediaPage 'September'>, <WikipediaPage 'Institute of Electrical and Electronics Engineers'>, <WikipediaPage 'Buy Buy Baby'>, <WikipediaPage 'Name'>, <WikipediaPage 'Often'>, <WikipediaPage 'Court'>, <WikipediaPage 'Saving'>, <WikipediaPage 'Name'>, <WikipediaPage 'Often'>, <WikipediaPage 'Following'>, <WikipediaPage 'Name'>, <WikipediaPage 'Name'>, <WikipediaPage 'Name'>, <WikipediaPage 'Length'>, <WikipediaPage 'Train'>, <WikipediaPage 'Number'>, <WikipediaPage 'May'>, <WikipediaPage 'Confusion'>, <WikipediaPage 'Name'>, <WikipediaPage 'Structure'>, <WikipediaPage 'Vase'>, <WikipediaPage 'Information'>, <WikipediaPage 'Calculation'>, <WikipediaPage 'Follows'>, <WikipediaPage 'Word'>, <WikipediaPage 'World'>, <WikipediaPage 'Phonetics'>, <WikipediaPage 'Constituting America'>, <WikipediaPage 'Semantics'>, <WikipediaPage 'World'>, <WikipediaPage 'Name'>, <WikipediaPage 'Semantics'>, <WikipediaPage 'Copula (linguistics)'>, <WikipediaPage 'Word'>, <WikipediaPage 'March'>, <WikipediaPage 'March'>, <WikipediaPage 'List of films considered the best'>, <WikipediaPage 'March'>, <WikipediaPage 'Role'>, <WikipediaPage 'Length'>, <WikipediaPage 'Radio'>, <WikipediaPage 'Coefficient'>, <WikipediaPage 'Name'>, <WikipediaPage 'Consecutive'>, <WikipediaPage 'Number'>, <WikipediaPage 'Name'>, <WikipediaPage 'Following'>, <WikipediaPage 'Word'>, <WikipediaPage 'Name'>, <WikipediaPage 'Name'>, <WikipediaPage 'Therefore sign'>, <WikipediaPage 'Length'>, <WikipediaPage 'Name'>, <WikipediaPage 'Name'>, <WikipediaPage 'Length'>, <WikipediaPage 'Name'>, <WikipediaPage 'Semantics'>, <WikipediaPage 'World'>, <WikipediaPage 'Book'>, <WikipediaPage 'Sichuan'>, <WikipediaPage 'March'>, <WikipediaPage 'Semantics'>, <WikipediaPage 'Length'>, <WikipediaPage 'Length'>, <WikipediaPage 'Semantics'>, <WikipediaPage 'Coefficient'>, <WikipediaPage 'Name'>, <WikipediaPage 'Data set'>, <WikipediaPage 'Name'>, <WikipediaPage 'Data set'>, <WikipediaPage 'Description'>, <WikipediaPage 'Training'>, <WikipediaPage 'Data set'>, <WikipediaPage 'Norway'>, <WikipediaPage 'Training'>, <WikipediaPage 'Data set'>, <WikipediaPage 'Norway'>, <WikipediaPage 'Exam'>, <WikipediaPage 'Data set'>, <WikipediaPage 'Norway'>, <WikipediaPage 'Model'>, <WikipediaPage 'Name'>, <WikipediaPage 'Accuracy and precision'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Training'>, <WikipediaPage 'URL'>, <WikipediaPage 'Collect'>, <WikipediaPage 'The'>, <WikipediaPage 'Experiment'>, <WikipediaPage 'Tested'>, <WikipediaPage 'GiveDirectly'>, <WikipediaPage 'Judgement'>, <WikipediaPage 'Norway'>, <WikipediaPage 'Experiment'>, <WikipediaPage 'Process'>, <WikipediaPage 'Copula (linguistics)'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Blacklisting'>, <WikipediaPage 'Finger'>, <WikipediaPage 'List of films considered the best'>, <WikipediaPage 'Authority'>, <WikipediaPage 'Experiment'>, <WikipediaPage 'Process'>, <WikipediaPage 'Follows'>, <WikipediaPage 'Performance'>, <WikipediaPage 'The'>, <WikipediaPage 'Data set'>, <WikipediaPage 'Role'>, <WikipediaPage 'Machine'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Car'>, <WikipediaPage 'Machine'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Car'>, <WikipediaPage 'Lawyer'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Calculation'>, <WikipediaPage 'Measurement'>, <WikipediaPage 'Accuracy and precision'>, <WikipediaPage 'Formula'>, <WikipediaPage 'Name'>, <WikipediaPage 'Name'>, <WikipediaPage 'Car'>, <WikipediaPage 'Car'>, <WikipediaPage 'Accuracy and precision'>, <WikipediaPage 'Combination'>, <WikipediaPage 'Accuracy and precision'>, <WikipediaPage 'Name'>, <WikipediaPage 'Car'>, <WikipediaPage 'The Substitute'>, <WikipediaPage 'Combination'>, <WikipediaPage 'Name'>, <WikipediaPage 'Car'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Finger'>, <WikipediaPage 'The'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Accuracy and precision'>, <WikipediaPage 'Accuracy and precision'>, <WikipediaPage 'Experiment'>, <WikipediaPage 'Optics'>, <WikipediaPage 'Training'>, <WikipediaPage 'Name'>, <WikipediaPage 'Car'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Determinacy'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Determinacy'>, <WikipediaPage 'Music Detected'>, <WikipediaPage 'The'>, <WikipediaPage 'Performance'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Model'>, <WikipediaPage 'The'>, <WikipediaPage 'Model'>, <WikipediaPage 'Comparative'>, <WikipediaPage 'Bes'>, <WikipediaPage 'Performing arts'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Observation'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Performativity'>, <WikipediaPage 'Netter'>, <WikipediaPage 'The'>, <WikipediaPage 'Clarifier'>, <WikipediaPage 'Model'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Finger'>, <WikipediaPage 'Clarifier'>, <WikipediaPage 'Death'>, <WikipediaPage 'Lawyer'>, <WikipediaPage '1'>, <WikipediaPage 'Classification'>, <WikipediaPage 'URL'>, <WikipediaPage 'The Netanyahus'>, <WikipediaPage 'Improvement'>, <WikipediaPage 'Accuracy and precision'>, <WikipediaPage 'Authorization'>, <WikipediaPage 'License'>, <WikipediaPage 'University'>, <WikipediaPage 'Institute'>, <WikipediaPage 'Technology'>, <WikipediaPage 'Download'>, <WikipediaPage 'September'>, <WikipediaPage 'Institute of Electrical and Electronics Engineers'>, <WikipediaPage 'Copula (linguistics)'>, <WikipediaPage 'Technology'>, <WikipediaPage 'Machine'>, <WikipediaPage 'Algorithm'>, <WikipediaPage 'Explosively formed penetrator'>, <WikipediaPage 'Lawyer'>, <WikipediaPage 'URL'>, <WikipediaPage 'Training'>, <WikipediaPage 'Machine'>, <WikipediaPage 'Algorithm'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Clarifier'>, <WikipediaPage 'Guide'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Nave'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Clarifier'>, <WikipediaPage 'Clarifier'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'System'>, <WikipediaPage 'Accuracy and precision'>, <WikipediaPage 'Observation'>, <WikipediaPage 'Year'>, <WikipediaPage 'Multiplayer video game'>, <WikipediaPage 'Model'>, <WikipediaPage 'Effectiveness'>, <WikipediaPage 'URL'>, <WikipediaPage 'Reference'>, <WikipediaPage 'Edward'>, <WikipediaPage 'Robert'>, <WikipediaPage 'Bayesian inference'>, <WikipediaPage 'Car'>, <WikipediaPage 'Research'>, <WikipediaPage 'Statistics'>, <WikipediaPage 'September'>, <WikipediaPage 'Nigel'>, <WikipediaPage 'Provence'>, <WikipediaPage 'Aviel'>, <WikipediaPage 'Measurement'>, <WikipediaPage 'Pish'>, <WikipediaPage 'Meeting'>, <WikipediaPage 'Workshop'>, <WikipediaPage 'Abhishek Bachchan'>, <WikipediaPage 'Muammar Gaddafi'>, <WikipediaPage 'Anhai'>, <WikipediaPage 'Digvijay'>, <WikipediaPage 'Lambda'>, <WikipediaPage 'Nikos'>, <WikipediaPage 'Tiwari'>, <WikipediaPage 'Sanjib'>, <WikipediaPage 'Subramaniam'>, <WikipediaPage 'Venky Atluri'>, <WikipediaPage 'Venky Harinarayan'>, <WikipediaPage 'Entity'>, <WikipediaPage 'Lion'>, <WikipediaPage 'Social'>, <WikipediaPage 'Conference proceedings'>, <WikipediaPage 'August'>, <WikipediaPage 'David'>, <WikipediaPage 'Nave'>, <WikipediaPage '40 (number)'>, <WikipediaPage 'Information'>, <WikipediaPage 'Voelker'>, <WikipediaPage 'URL'>, <WikipediaPage 'Intelligence'>, <WikipediaPage 'System'>, <WikipediaPage 'Technology'>, <WikipediaPage 'April'>, <WikipediaPage 'Feda'>, <WikipediaPage 'Thatching'>, <WikipediaPage 'Maler'>, <WikipediaPage 'Aburrow'>, <WikipediaPage 'Keshav'>, <WikipediaPage 'Pushpa Kamal Dahal'>, <WikipediaPage 'Telligent Systems'>, <WikipediaPage 'Pish'>, <WikipediaPage 'System'>, <WikipediaPage 'Expect'>, <WikipediaPage 'System'>, <WikipediaPage 'Ankush'>, <WikipediaPage 'Christians'>, <WikipediaPage 'Machine'>, <WikipediaPage 'Road'>, <WikipediaPage 'Machine'>, <WikipediaPage 'Fiber'>, <WikipediaPage 'System'>, <WikipediaPage 'Berlin'>, <WikipediaPage 'Heidelberg'>, <WikipediaPage 'Berlin'>, <WikipediaPage 'Heidelberg'>, <WikipediaPage 'Want'>, <WikipediaPage 'Gill'>, <WikipediaPage 'Security'>, <WikipediaPage 'Bogan, Iran'>, <WikipediaPage 'Time'>, <WikipediaPage 'Determinant'>, <WikipediaPage 'Time'>, <WikipediaPage 'Security'>, <WikipediaPage 'Behavior'>, <WikipediaPage 'Computer'>, <WikipediaPage 'Security'>, <WikipediaPage 'Mexico at the 1994 Winter Olympics'>, <WikipediaPage 'Philip'>, <WikipediaPage 'Coll'>, <WikipediaPage 'Car'>, <WikipediaPage 'Classification'>, <WikipediaPage 'Algorithm'>, <WikipediaPage 'Information'>, <WikipediaPage 'Security'>, <WikipediaPage 'Interdiction'>, <WikipediaPage 'Mitigation'>, <WikipediaPage 'Memory'>, <WikipediaPage 'Infection'>, <WikipediaPage 'Parent'>, <WikipediaPage 'Kurt'>, <WikipediaPage 'Chris'>, <WikipediaPage 'Pam Grier'>, <WikipediaPage 'Verb'>, <WikipediaPage 'John Paxson'>, <WikipediaPage 'Son'>, <WikipediaPage 'Design'>, <WikipediaPage 'Evolution'>, <WikipediaPage 'Year'>, <WikipediaPage 'The'>, <WikipediaPage 'Institute of Electrical and Electronics Engineers'>, <WikipediaPage 'Symposium'>, <WikipediaPage 'Security'>, <WikipediaPage 'Sea'>, <WikipediaPage 'Whelan'>, <WikipediaPage 'Nathaniel'>, <WikipediaPage 'Astolfo'>, <WikipediaPage 'Conference proceedings'>, <WikipediaPage 'Workshop'>, <WikipediaPage 'Artificiality'>, <WikipediaPage 'Intelligence'>, <WikipediaPage 'Security'>, <WikipediaPage 'Workshop'>, <WikipediaPage 'LATAM Colombia'>, <WikipediaPage 'York'>, <WikipediaPage 'Cross'>, <WikipediaPage 'Discretization'>, <WikipediaPage 'Naive Bayes classifier'>, <WikipediaPage 'Manning'>, <WikipediaPage 'Discretization'>, <WikipediaPage 'Machine'>, <WikipediaPage 'Authorization'>, <WikipediaPage 'License'>, <WikipediaPage 'University'>, <WikipediaPage 'Institute'>, <WikipediaPage 'Technology'>, <WikipediaPage 'Download'>, <WikipediaPage 'September'>, <WikipediaPage 'Institute of Electrical and Electronics Engineers'>]\n",
            "The descending order of similarity index with different websites is:\n",
            "https://en.wikipedia.org/wiki/Kumar 0.3245643409797063\n",
            "https://en.wikipedia.org/wiki/Hussein 0.3243350597141768\n",
            "https://en.wikipedia.org/wiki/Robert 0.32402628162470454\n",
            "https://en.wikipedia.org/wiki/Combination 0.3173342175885807\n",
            "https://en.wikipedia.org/wiki/Edward 0.31696183677603734\n",
            "https://en.wikipedia.org/wiki/Naive_Bayes_classifier 0.3064859916665657\n",
            "https://en.wikipedia.org/wiki/Philip 0.30625970814856945\n"
          ]
        }
      ],
      "source": [
        "import gensim\n",
        "from gensim.models.doc2vec import Doc2Vec, TaggedDocument\n",
        "from nltk.tokenize import word_tokenize\n",
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "import wikipedia\n",
        "\n",
        "wiki_pages = []\n",
        "for i in tokens:\n",
        "  try:\n",
        "    wiki_pages.append(wikipedia.page(i))\n",
        "  except:\n",
        "    continue\n",
        "print(wiki_pages)\n",
        "\n",
        "model = gensim.models.doc2vec.Doc2Vec(vector_size=30, min_count=2, epochs=80)\n",
        "tagged_data = [TaggedDocument(words=word_tokenize(_d.lower()), tags=[str(i)]) for i, _d in enumerate(text)]\n",
        "model.build_vocab(tagged_data)\n",
        "model.train(tagged_data, total_examples=model.corpus_count, epochs=80)\n",
        "\n",
        "data = word_tokenize(' '.join(tokens))\n",
        "token_vector = model.infer_vector(data)\n",
        "\n",
        "wiki_dict = {}\n",
        "for i in wiki_pages:\n",
        "  data = word_tokenize(i.content)\n",
        "  wiki_dict[i.url] = model.infer_vector(data)\n",
        "\n",
        "def cosine_sim(a, b):\n",
        "  sum = 0\n",
        "  sum_a = 0\n",
        "  sum_b = 0\n",
        "  for i in range(len(a)):\n",
        "    sum_a += a[i]**2\n",
        "    sum_b += b[i]**2\n",
        "    sum += (a[i]*b[i])\n",
        "\n",
        "  return sum/((sum_a**(1/2))*(sum_b**(1/2)))\n",
        "\n",
        "cos_vals = {}\n",
        "for i in wiki_pages:\n",
        "  cos_vals[i.url] = cosine_sim(token_vector, wiki_dict[i.url])\n",
        "\n",
        "print('The descending order of similarity index with different websites is:')\n",
        "ranked = sorted(cos_vals, key=lambda x:cos_vals[x], reverse=True)\n",
        "for i in ranked:\n",
        "  if cos_vals[i]>0.3:\n",
        "    print(i, cos_vals[i])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "-YaW9NWIjI1_",
      "metadata": {
        "id": "-YaW9NWIjI1_"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}