ipynb/ammar_alyousfi-youtube_trending_videos_analysis.ipynb

{
    "cells": [
        {
            "metadata": {
                "_uuid": "b3df1737a6682114c26a8dd64248229b6eb115a0"
            },
            "cell_type": "markdown",
            "source": "# YouTube Trending Videos Analysis (More Than 40,000 Videos)\n\n<p>&nbsp;</p>\n<img src=\"https://s3.eu-west-2.amazonaws.com/ammar-blog-post-images/2018/Sep/ytanalysis.png\" width=600>\n<p>&nbsp;</p>\n\n### Introduction\nYouTube is the most popular and most used video platfrom in the world today. YouTube has [a list of **trending videos**](https://www.youtube.com/feed/trending) that is updated constantly. Here we will use **Python** with some packages like **Pandas** and **Matplotlib** to analyze a dataset that was collected over 205 days. For each of those days, the dataset contains data about the trending videos of that day. It contains data about **more than `40,000` trending videos**. We will analyze this data to get insights into YouTube trending videos, to see what is common between these videos. Those insights might also be used by people who want to increase popularity of their videos on YouTube.\n\nThe dataset that we will use is obtained from Kaggle [here](https://www.kaggle.com/datasnaek/youtube-new). It contains data about trending videos for many countries. Here we will analyze USA trending videos. \n\n### Goals of the analysis\nWe want to answer questions like:\n* How many views do our trending videos have? Do most of them have a large number of views? Is having a large number of views required for a video to become trending?\n* The same questions above, but applied to likes and comment count instead of views.\n* Which video remained the most on the trendin-videos list?\n* How many trending videos contain a fully-capitalized word in their titles?\n* What are the lengths of trending video titles? Is this length related to the video becoming trendy?\n* How are views, likes, dislikes, comment count, title length, and other attributes correlate with (relate to) each other? How are they connected?\n* What are the most common words in trending video titles?\n* Which YouTube channels have the largest number of trending videos?\n* Which video category (e.g. Entertainment, Gaming, Comedy, etc.) has the largest number of trending videos?\n* When were trending videos published? On which days of the week? at which times of the day?\n\nLet's get started.\n\n### Importing some packages\nFirst, we import some Python packages that will help us analyzing the data, especially `pandas` for data analysis and `matplotlib` for visualization"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "164ee507aa6ce7b0b16a9e6b9a960e2c25106972"
            },
            "cell_type": "code",
            "source": "import pandas as pd\nimport numpy as np\nimport matplotlib as mpl\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\n\nimport warnings\nfrom collections import Counter\nimport datetime\nimport wordcloud\nimport json",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "d5a3139b1c1e3d4e362718e04e16951d2999b43a"
            },
            "cell_type": "code",
            "source": "# Hiding warnings for cleaner display\nwarnings.filterwarnings('ignore')\n\n# Configuring some options\n%matplotlib inline\n%config InlineBackend.figure_format = 'retina'\n# If you want interactive plots, uncomment the next line\n# %matplotlib notebook",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "64cb706b46f7d421fb43098b4135f096369f8530"
            },
            "cell_type": "markdown",
            "source": "### Reading the dataset\nThen we read the dataset file which is in csv format"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "840f86adb72a916416f0d6d3ebbdcdd6f625a27a"
            },
            "cell_type": "code",
            "source": "df = pd.read_csv(\"../input/USvideos.csv\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "c9575926a97428567545a4f0b5da802476cd1057"
            },
            "cell_type": "markdown",
            "source": "We set some configuration options just for improving visualization graphs; nothing crucial"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "9c638de51c17ea9ca8c977f99d1f512bb6ab5f30"
            },
            "cell_type": "code",
            "source": "PLOT_COLORS = [\"#268bd2\", \"#0052CC\", \"#FF5722\", \"#b58900\", \"#003f5c\"]\npd.options.display.float_format = '{:.2f}'.format\nsns.set(style=\"ticks\")\nplt.rc('figure', figsize=(8, 5), dpi=100)\nplt.rc('axes', labelpad=20, facecolor=\"#ffffff\", linewidth=0.4, grid=True, labelsize=14)\nplt.rc('patch', linewidth=0)\nplt.rc('xtick.major', width=0.2)\nplt.rc('ytick.major', width=0.2)\nplt.rc('grid', color='#9E9E9E', linewidth=0.4)\nplt.rc('font', family='Arial', weight='400', size=10)\nplt.rc('text', color='#282828')\nplt.rc('savefig', pad_inches=0.3, dpi=300)",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "33d86db99ba99d7da01813cf5106b07d4df3521c"
            },
            "cell_type": "markdown",
            "source": "### Getting a feel of the dataset\nLet's get a feel of our dataset by displaying its first few rows"
        },
        {
            "metadata": {
                "scrolled": false,
                "trusted": true,
                "_uuid": "393a26eb237ec760470b82bad3af1a257fd0297a"
            },
            "cell_type": "code",
            "source": "df.head()",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "b29eaf07122a7943accbdecc9543a8879925839a"
            },
            "cell_type": "markdown",
            "source": "Now, let's see some information about our dataset"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "39c6f0d0ba87d0ded6c3104fedd01779c48067b6"
            },
            "cell_type": "code",
            "source": "df.info()",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "3aaf9a7f56087e5c4dae09f0552fe4ead5eabba1"
            },
            "cell_type": "markdown",
            "source": "We can see that there are `40,949` entries in the dataset. We can see also that all columns in the dataset are complete (i.e. they have `40,949` non-null entries) except `description` column which has some `null` values; it only has `40,379` non-null values.\n\n#### Data cleaning\nThe `description` column has some null values. These are some of the rows whose description values are null. We can see that null values are denoted by `NaN`"
        },
        {
            "metadata": {
                "scrolled": false,
                "trusted": true,
                "_uuid": "9b822830d5c12ebf2e78e023db06acbaee950a8c"
            },
            "cell_type": "code",
            "source": "df[df[\"description\"].apply(lambda x: pd.isna(x))].head(3)",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "84a324278c182466e7b8632af0c38c3f0ebe0f36"
            },
            "cell_type": "markdown",
            "source": "So to do some sort of data cleaning, and to get rid of those null values, we put an empty string in place of each null value in the `description` column"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "52a92198121fd8840df9ff5a2807f8c5571ce585"
            },
            "cell_type": "code",
            "source": "df[\"description\"] = df[\"description\"].fillna(value=\"\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "d5d3ed516f27694fd2197480baa2c88c019aa48a"
            },
            "cell_type": "markdown",
            "source": "#### Dataset collection years\nLet's see in which years the dataset was collected"
        },
        {
            "metadata": {
                "scrolled": false,
                "trusted": true,
                "_uuid": "6f7f61ebfd42f4aef6eb71f2bedfbfb9e4e3a8fd"
            },
            "cell_type": "code",
            "source": "cdf = df[\"trending_date\"].apply(lambda x: '20' + x[:2]).value_counts() \\\n            .to_frame().reset_index() \\\n            .rename(columns={\"index\": \"year\", \"trending_date\": \"No_of_videos\"})\n\nfig, ax = plt.subplots()\n_ = sns.barplot(x=\"year\", y=\"No_of_videos\", data=cdf, \n                palette=sns.color_palette(['#ff764a', '#ffa600'], n_colors=7), ax=ax)\n_ = ax.set(xlabel=\"Year\", ylabel=\"No. of videos\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "70a711a8c677e9c88ace63ca6451def966fbb887"
            },
            "cell_type": "code",
            "source": "df[\"trending_date\"].apply(lambda x: '20' + x[:2]).value_counts(normalize=True)",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "7160965b2d3cb022bffad38bebe1c2e6980f3aec"
            },
            "cell_type": "markdown",
            "source": "We can see that the dataset was collected in 2017 and 2018 with `77%` of it in 2018 and `23%` in 2017."
        },
        {
            "metadata": {
                "_uuid": "8b3cb47f7c79d7ff3947d3dd9ff3c1ec3f9ad0d2"
            },
            "cell_type": "markdown",
            "source": "### Describtion of numerical columns\nNow, let's see some statistical information about the numerical columns of our dataset"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "f4340486c8bcfcc4edfe3c40a8d39561a52e7c76"
            },
            "cell_type": "code",
            "source": "df.describe()",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "c69c8cf4c856e0591bd3db08bf99cb90a309091e"
            },
            "cell_type": "markdown",
            "source": "We note from the table above that \n- The average number of views of a trending video is `2,360,784`. The median value for the number of views is `681,861`, which means that half the trending videos have views that are less than that number, and the other half have views larger than that number\n- The average number of likes of a trending video is `74,266`, while the average number of dislikes is `3,711`. The \n- Average comment count is `8,446` while the median is `1,856`\n\nHow useful are the observations above? Do they really represent the data? Let's examine more. \n\n### Views histogram\nlet's plot a [histogram](https://www.mathsisfun.com/data/histograms.html) for the `views` column to take a look at its distribution: to see how many videos have between `10` million and `20` million views, how many videos have between `20` million and `30` million views, and so on."
        },
        {
            "metadata": {
                "scrolled": false,
                "trusted": true,
                "_uuid": "ceab9e9f2e7588aa56b55ed32c4022aa2abb2bbd"
            },
            "cell_type": "code",
            "source": "fig, ax = plt.subplots()\n_ = sns.distplot(df[\"views\"], kde=False, color=PLOT_COLORS[4], \n                 hist_kws={'alpha': 1}, bins=np.linspace(0, 2.3e8, 47), ax=ax)\n_ = ax.set(xlabel=\"Views\", ylabel=\"No. of videos\", xticks=np.arange(0, 2.4e8, 1e7))\n_ = ax.set_xlim(right=2.5e8)\n_ = plt.xticks(rotation=90)",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "a4b283a71f8f8db5be20fc6ef0648ccc48c9606a"
            },
            "cell_type": "markdown",
            "source": "We note that the vast majority of trending videos have `5` million views or less. We get the `5` million number by calculating\n\n\\begin{align}\n\\frac{0.1 \\times 10^8}{2} = 5 \\times 10^6\n\\end{align}\n\nNow let us plot the histogram just for videos with `25` million views or less to get a closer look at the distribution of the data"
        },
        {
            "metadata": {
                "scrolled": false,
                "trusted": true,
                "_uuid": "152a2ab45009ce1160f07f693e35a89f03924130"
            },
            "cell_type": "code",
            "source": "fig, ax = plt.subplots()\n_ = sns.distplot(df[df[\"views\"] < 25e6][\"views\"], kde=False, \n                 color=PLOT_COLORS[4], hist_kws={'alpha': 1}, ax=ax)\n_ = ax.set(xlabel=\"Views\", ylabel=\"No. of videos\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "a0117677a976b59f48dacaf5be9c2f03f793c29b"
            },
            "cell_type": "markdown",
            "source": "Now we see that the majority of trending videos have `1` million views or less. Let's see the exact percentage of videos less than `1` million views"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "4d7af2262b29c3d8db397d77a268a13485c8c092"
            },
            "cell_type": "code",
            "source": "df[df['views'] < 1e6]['views'].count() / df['views'].count() * 100",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "c6194fb2ea258bd33cccbf18bedabe635d62081d"
            },
            "cell_type": "markdown",
            "source": "So, it is around `60%`. Similarly, we can see that the percentage of videos with less than `1.5` million views is around `71%`, and that the percentage of videos with less than `5` million views is around `91%`.\n\n### Likes histogram\n\nAfter `views`, we plot the histogram for `likes` column"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "17f7ef3276dc86cce576e237da938e38b94c4db7"
            },
            "cell_type": "code",
            "source": "plt.rc('figure.subplot', wspace=0.9)\nfig, ax = plt.subplots()\n_ = sns.distplot(df[\"likes\"], kde=False, \n                 color=PLOT_COLORS[4], hist_kws={'alpha': 1}, \n                 bins=np.linspace(0, 6e6, 61), ax=ax)\n_ = ax.set(xlabel=\"Likes\", ylabel=\"No. of videos\")\n_ = plt.xticks(rotation=90)",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "daf1fbe9ba4a3f0b64e2f4e4ab77882223a894f7"
            },
            "cell_type": "markdown",
            "source": "We note that the vast majority of trending videos have between `0` and `100,000` likes. Let us plot the histogram just for videos with `1000,000` likes or less to get a closer look at the distribution of the data"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "08811b8233fbfa84a90c8d450811388503505277",
                "scrolled": true
            },
            "cell_type": "code",
            "source": "fig, ax = plt.subplots()\n_ = sns.distplot(df[df[\"likes\"] <= 1e5][\"likes\"], kde=False, \n                 color=PLOT_COLORS[4], hist_kws={'alpha': 1}, ax=ax)\n_ = ax.set(xlabel=\"Likes\", ylabel=\"No. of videos\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "92d6cbeced9dc3b1a9a1bc87f6e47606f292de6e"
            },
            "cell_type": "markdown",
            "source": "Now we can see that the majority of trending videos have `40000` likes or less with a peak for videos with `2000` likes or less. \n\nLet's see the exact percentage of videos with less than `40000` likes "
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "393b2d13cb0a6c05c875eae23aa707c45fae4e8b"
            },
            "cell_type": "code",
            "source": "df[df['likes'] < 4e4]['likes'].count() / df['likes'].count() * 100",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "a5d80ca8f3f1b05c76ea4ae212f43998380d497f"
            },
            "cell_type": "markdown",
            "source": "Similarly, we can see that the percentage of videos with less than `100,000` likes is around `84%`"
        },
        {
            "metadata": {
                "_uuid": "c36407ae456f7a5a4e386e766aecc6777dec7ffd"
            },
            "cell_type": "markdown",
            "source": "### Comment count histogram"
        },
        {
            "metadata": {
                "scrolled": false,
                "trusted": true,
                "_uuid": "9385664b43dd96ad0723366de4d515160a4179f0"
            },
            "cell_type": "code",
            "source": "fig, ax = plt.subplots()\n_ = sns.distplot(df[\"comment_count\"], kde=False, rug=False, \n                 color=PLOT_COLORS[4], hist_kws={'alpha': 1}, ax=ax)\n_ = ax.set(xlabel=\"Comment Count\", ylabel=\"No. of videos\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "49d44584df518acb7d31494ff6df8bdce6530e38"
            },
            "cell_type": "markdown",
            "source": "Let's get a closer look by eliminating entries with comment count larger than `200000` comment"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "a55671b8b9ee69fa56b5807d33c1ad692756fc60"
            },
            "cell_type": "code",
            "source": "fig, ax = plt.subplots()\n_ = sns.distplot(df[df[\"comment_count\"] < 200000][\"comment_count\"], kde=False, rug=False, \n                 color=PLOT_COLORS[4], hist_kws={'alpha': 1}, \n                 bins=np.linspace(0, 2e5, 49), ax=ax)\n_ = ax.set(xlabel=\"Comment Count\", ylabel=\"No. of videos\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "e36823d0257666a719d8db85daf9661750575f1b"
            },
            "cell_type": "markdown",
            "source": "We see that most trending videos have around\n\n\\begin{align}\n\\frac{25000}{6} \\approx 4166 \\text{ comments}\n\\end{align}\n\nsince each division in the graph has six histogram bins. \n\nAs with views and likes, let's see the exact percentage of videos with less than `4000` comments"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "712d82c09f7811d7eec47f0c14d1ddd8edd86d87"
            },
            "cell_type": "code",
            "source": "df[df['comment_count'] < 4000]['comment_count'].count() / df['comment_count'].count() * 100",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "86ccf02686d7f6425efd2e166575a1f7bdec80b0"
            },
            "cell_type": "markdown",
            "source": "In a similar way, we can see that the percentage of videos with less than `25,000` comments  is around `93%`."
        },
        {
            "metadata": {
                "_uuid": "054055d806c8ffe047bb43efec96fc88a48a0ddc"
            },
            "cell_type": "markdown",
            "source": "### Description on non-numerical columns\nAfter we described numerical columns previously, we now describe non-numerical columns"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "b9692b317b23dfedd3eb05be4845d7057db2223e"
            },
            "cell_type": "code",
            "source": "df.describe(include = ['O'])",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "4fa39258f2e465ea3a04c59f7a7ae82d69e3615a"
            },
            "cell_type": "markdown",
            "source": "From the table above, we can see that there are `205` unique dates, which means that our dataset contains collected data about trending videos over **`205`** days.\n\nFrom `video_id` description, we can see that there are `40949` videos (which is expected because our dataset contains `40949` entries), but we can see also that there are only `6351` unique videos which means that some videos appeared on the trending videos list **on more than one day**.\nThe table also tells us that the top frequent title is `WE MADE OUR MOM CRY...HER DREAM CAME TRUE!` and that it appeared `30` times on the trending videos list.\n\nBut there is something strange in the description table above: Because there are `6351` unique video IDs, we expect to have `6351` unique video titles also, because we assume that each ID is linked to a corresponding title. One possible interpretation is that a trending video had some title when it appeared on the trending list, then it appeared again on another day but with a modified title. Similar explaination applies for `description` column as well.\nFor `publish_time` column, the unique values are less than `6351`, but there is nothing strange here, because two different videos may be published at the same time.\n\nTo verify our interpretation for `title` column, let's take a look at an example where a trending video appeared more than once on the trending list but with different titles"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "1f932edc5f5a36f512e5d3909d68c1376e53812d"
            },
            "cell_type": "code",
            "source": "grouped = df.groupby(\"video_id\")\ngroups = []\nwanted_groups = []\nfor key, item in grouped:\n    groups.append(grouped.get_group(key))\n\nfor g in groups:\n    if len(g['title'].unique()) != 1:\n        wanted_groups.append(g)\n\nwanted_groups[0]",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "47eb9a90343e82b1b64cb732924105f2d6812138"
            },
            "cell_type": "markdown",
            "source": "We can see that this video appeared on the list with two different titles."
        },
        {
            "metadata": {
                "_uuid": "3cec6b9e805122c58d5ab6952237e48d7fd2362e"
            },
            "cell_type": "markdown",
            "source": "### How many trending video titles contain capitalized word?\nNow we want to see how many trending video titles contain at least a capitalized word (e.g. HOW). To do that, we will add a new variable (column) to the dataset whose value is `True` if the video title has at least a capitalized word in it, and `False` otherwise"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "ea8587cc613548db5fed837d99d20fc3105b1315"
            },
            "cell_type": "code",
            "source": "def contains_capitalized_word(s):\n    for w in s.split():\n        if w.isupper():\n            return True\n    return False\n\n\ndf[\"contains_capitalized\"] = df[\"title\"].apply(contains_capitalized_word)\n\nvalue_counts = df[\"contains_capitalized\"].value_counts().to_dict()\nfig, ax = plt.subplots()\n_ = ax.pie([value_counts[False], value_counts[True]], labels=['No', 'Yes'], \n           colors=['#003f5c', '#ffa600'], textprops={'color': '#040204'}, startangle=45)\n_ = ax.axis('equal')\n_ = ax.set_title('Title Contains Capitalized Word?')",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "987716074dbdf6b5ceaaf652729eadf35cb36658"
            },
            "cell_type": "code",
            "source": "df[\"contains_capitalized\"].value_counts(normalize=True)",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "7722639a530841890c30e97fe4abcd3e9c159063"
            },
            "cell_type": "markdown",
            "source": "We can see that 44% of trending video titles contain at least a capitalized word. We will later use this added new column `contains_capitalized` in analyzing correlation between variables."
        },
        {
            "metadata": {
                "_uuid": "eeb1e23b401c800846f8a731172604be65c5c109"
            },
            "cell_type": "markdown",
            "source": "### Video title lengths\nLet's add another column to our dataset to represent the length of each video title, then plot the histogram of title length to get an idea about the lengths of trnding video titles"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "c8382545b86b285817b344df70c44b9d711731ee"
            },
            "cell_type": "code",
            "source": "df[\"title_length\"] = df[\"title\"].apply(lambda x: len(x))\n\nfig, ax = plt.subplots()\n_ = sns.distplot(df[\"title_length\"], kde=False, rug=False, \n                 color=PLOT_COLORS[4], hist_kws={'alpha': 1}, ax=ax)\n_ = ax.set(xlabel=\"Title Length\", ylabel=\"No. of videos\", xticks=range(0, 110, 10))",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "a957e8c0612d375287d87e69665d6670be197023"
            },
            "cell_type": "markdown",
            "source": "We can see that title-length distribution resembles a normal distribution, where most videos have title lengths between 30 and 60 character approximately.\n\nNow let's draw a [scatter plot](https://www.mathsisfun.com/data/scatter-xy-plots.html) between title length and number of views to see the relationship between these two variables"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "9e5956c90612d6c9c6a183c45d88acd5c2a39290"
            },
            "cell_type": "code",
            "source": "fig, ax = plt.subplots()\n_ = ax.scatter(x=df['views'], y=df['title_length'], color=PLOT_COLORS[2], edgecolors=\"#000000\", linewidths=0.5)\n_ = ax.set(xlabel=\"Views\", ylabel=\"Title Length\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "f4f3b82549616f1ce5b81af3ff0b66d7c2341639"
            },
            "cell_type": "markdown",
            "source": "By looking at the scatter plot, we can say that there is no relationship between the title length and the number of views. However, we notice an interesting thing: videos that have `100,000,000` views and more have title length between `33` and `55` characters approximately."
        },
        {
            "metadata": {
                "_uuid": "63dadc4fbc17a03acae861d12b3b61b9e97e07aa"
            },
            "cell_type": "markdown",
            "source": "### Correlation between dataset variables\nNow let's see how the dataset variables are [correlated](https://www.mathsisfun.com/data/correlation.html) with each other: for example, we would like to see how views and likes are correlated, meaning do views and likes increase and decrease together (positive correlation)? Does one of them increase when the other decrease and vice versa (negative correlation)? Or are they not correlated?\n\nCorrelation is represented as a value between `-1` and `+1` where `+1` denotes the highest positive correlation, `-1` denotes the highest negative correlation, and `0` denotes that there is no correlation.\n\nLet's see the correlation table between our dataset variables (numerical and boolean variables only)"
        },
        {
            "metadata": {
                "scrolled": true,
                "trusted": true,
                "_uuid": "09ac9f1186980247b021f8b90b37683c5b1cd56c"
            },
            "cell_type": "code",
            "source": "df.corr()",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "e880caf432f72f3475b1fb069f088c5a55ea5d01"
            },
            "cell_type": "markdown",
            "source": "We see for example that views and likes are highly positively correlated with a correlation value of `0.85`; we see also a high positive correlation (`0.80`) between likes and comment count, and between dislikes and comment count (`0.70`). \n\nThere is some positive correlation between views and dislikes, between views and comment count, between likes and dislikes.\n\nNow let's visualize the correlation table above using a [heatmap](https://www.wikiwand.com/en/Heat_map)"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "eb06e2efa0080cba9db6356937bca88d1911aeb5"
            },
            "cell_type": "code",
            "source": "h_labels = [x.replace('_', ' ').title() for x in \n            list(df.select_dtypes(include=['number', 'bool']).columns.values)]\n\nfig, ax = plt.subplots(figsize=(10,6))\n_ = sns.heatmap(df.corr(), annot=True, xticklabels=h_labels, yticklabels=h_labels, cmap=sns.cubehelix_palette(as_cmap=True), ax=ax)",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "b3fb00b3fd98d986d9f059b0c0a5aa993119d562"
            },
            "cell_type": "markdown",
            "source": "The correlation map and correlation table above say that views and likes are highly positively correlated. Let's verify that by plotting a scatter plot between views and likes to visualize the relationship between these variables"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "e123ed225974e27a9a2dff392cb8f122312d39e4"
            },
            "cell_type": "code",
            "source": "fig, ax = plt.subplots()\n_ = plt.scatter(x=df['views'], y=df['likes'], color=PLOT_COLORS[2], edgecolors=\"#000000\", linewidths=0.5)\n_ = ax.set(xlabel=\"Views\", ylabel=\"Likes\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "7b9774146c8efbe126e066156a347c4b3dd7cdf7"
            },
            "cell_type": "markdown",
            "source": "We see that views and likes are truly positively correlated: as one increases, the other increases too—mostly.\n\nAnother verification of the correlation matrix and map is the scatter plot we drew above between views and title length as it shows that there is no correlation between them."
        },
        {
            "metadata": {
                "_uuid": "58da20636680c66ae241b291221644806488e2d9"
            },
            "cell_type": "markdown",
            "source": "### Most common words in video titles\nLet's see if there are some words that are used significantly in trending video titles. We will display the `25` most common words in all trending video titles"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "e2fb61a91881fafa068ec6dc98d683307bd2c8c0"
            },
            "cell_type": "code",
            "source": "title_words = list(df[\"title\"].apply(lambda x: x.split()))\ntitle_words = [x for y in title_words for x in y]\nCounter(title_words).most_common(25)",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "2cac7a11ab304dfb69f8578f4be0eaad6d42d4b0"
            },
            "cell_type": "markdown",
            "source": "Ignoring words like \"the\" and \"of\", we can see that \"-\" and \"|\" symbols occured a lot in the `40949` trending video titles: `11452` times and `10663` times respectively. We notice also that words \"Video\", \"Trailer\", \"How\", and \"2018\" are common in trending video titles; each occured in 1613-1901 video titles.\n\nLet's draw a word cloud for the titles of our trending videos, which is a way to visualize most common words in the titles; the more common the word is, the bigger its font size is"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "e984e08f030b99f7168a50274cfbe103e9c457aa"
            },
            "cell_type": "code",
            "source": "# wc = wordcloud.WordCloud(width=1200, height=600, collocations=False, stopwords=None, background_color=\"white\", colormap=\"tab20b\").generate_from_frequencies(dict(Counter(title_words).most_common(150)))\nwc = wordcloud.WordCloud(width=1200, height=500, \n                         collocations=False, background_color=\"white\", \n                         colormap=\"tab20b\").generate(\" \".join(title_words))\nplt.figure(figsize=(15,10))\nplt.imshow(wc, interpolation='bilinear')\n_ = plt.axis(\"off\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "5ef64213dabb4ec0f816df63fad7e6a037dad066"
            },
            "cell_type": "markdown",
            "source": "### Which channels have the largest number of trending videos?"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "bb7887e44cb0389c33f0aab4fd53bf637e402a08"
            },
            "cell_type": "code",
            "source": "cdf = df.groupby(\"channel_title\").size().reset_index(name=\"video_count\") \\\n    .sort_values(\"video_count\", ascending=False).head(20)\n\nfig, ax = plt.subplots(figsize=(8,8))\n_ = sns.barplot(x=\"video_count\", y=\"channel_title\", data=cdf,\n                palette=sns.cubehelix_palette(n_colors=20, reverse=True), ax=ax)\n_ = ax.set(xlabel=\"No. of videos\", ylabel=\"Channel\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "e98a68e219b3feef8914c0d606f678d8344d6682"
            },
            "cell_type": "markdown",
            "source": "### Which video category has the largest number of trending videos?\nFirst, we will add a column that contains category names based on the values in `category_id` column. We will use a category JSON file provided with the dataset which contains information about each category."
        },
        {
            "metadata": {
                "scrolled": false,
                "trusted": true,
                "_uuid": "ac42ec58be60adee5d80fe4cf9573c918e938bca"
            },
            "cell_type": "code",
            "source": "with open(\"../input/US_category_id.json\") as f:\n    categories = json.load(f)[\"items\"]\ncat_dict = {}\nfor cat in categories:\n    cat_dict[int(cat[\"id\"])] = cat[\"snippet\"][\"title\"]\ndf['category_name'] = df['category_id'].map(cat_dict)",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "a06fa8776f82f4b027d1720a6b6a59ed0e6f7058"
            },
            "cell_type": "markdown",
            "source": "Now we can see which category had the largest number of trending videos"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "50e8f998e289137bfc6f1913043f0ff189eb270c"
            },
            "cell_type": "code",
            "source": "cdf = df[\"category_name\"].value_counts().to_frame().reset_index()\ncdf.rename(columns={\"index\": \"category_name\", \"category_name\": \"No_of_videos\"}, inplace=True)\nfig, ax = plt.subplots()\n_ = sns.barplot(x=\"category_name\", y=\"No_of_videos\", data=cdf, \n                palette=sns.cubehelix_palette(n_colors=16, reverse=True), ax=ax)\n_ = ax.set_xticklabels(ax.get_xticklabels(), rotation=90)\n_ = ax.set(xlabel=\"Category\", ylabel=\"No. of videos\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "6826a2afeeeee1a29f85b73c648ab44ed6594e7d"
            },
            "cell_type": "markdown",
            "source": "We see that the Entertainment category contains the largest number of trending videos among other categories: around `10,000` videos, followed by Music category with around `6,200` videos, followed by Howto & Style category with around `4,100` videos, and so on."
        },
        {
            "metadata": {
                "_uuid": "b69d59a218e8e7f062c50f6c7adbb5f8e209fe79"
            },
            "cell_type": "markdown",
            "source": "### Trending videos and their publishing time\nAn example value of the `publish_time` column in our dataset is `2017-11-13T17:13:01.000Z`. And according to information on this page: https://www.w3.org/TR/NOTE-datetime, this means that the date of publishing the video is `2017-11-13` and the time is `17:13:01` in Coordinated Universal Time (UTC) time zone.\n\nLet's add two columns to represent the date and hour of publishing each video, then delete the original `publish_time` column because we will not need it anymore"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "a61bb1ed2af2da97f53b58bfa8b331e63ca7a894"
            },
            "cell_type": "code",
            "source": "df[\"publishing_day\"] = df[\"publish_time\"].apply(\n    lambda x: datetime.datetime.strptime(x[:10], \"%Y-%m-%d\").date().strftime('%a'))\ndf[\"publishing_hour\"] = df[\"publish_time\"].apply(lambda x: x[11:13])\ndf.drop(labels='publish_time', axis=1, inplace=True)",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "1f4476707a7401eeebc06871688dbbfc1c4c38ae"
            },
            "cell_type": "markdown",
            "source": "Now we can see which days of the week had the largest numbers of trending videos"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "b46762767e1fc981cf6164cacdaeff1cea74adb0"
            },
            "cell_type": "code",
            "source": "cdf = df[\"publishing_day\"].value_counts()\\\n        .to_frame().reset_index().rename(columns={\"index\": \"publishing_day\", \"publishing_day\": \"No_of_videos\"})\nfig, ax = plt.subplots()\n_ = sns.barplot(x=\"publishing_day\", y=\"No_of_videos\", data=cdf, \n                palette=sns.color_palette(['#003f5c', '#374c80', '#7a5195', \n                                           '#bc5090', '#ef5675', '#ff764a', '#ffa600'], n_colors=7), ax=ax)\n_ = ax.set(xlabel=\"Publishing Day\", ylabel=\"No. of videos\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "d1bf38b80a7f779e8332f5f21f0055a63c798371"
            },
            "cell_type": "markdown",
            "source": "We can see that the number of trending videos published on Sunday and Saturday are noticeably less than the number of trending videos published on other days of the week.\n\nNow let's use `publishing_hour` column to see which publishing hours had the largest number of trending videos"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "ac6afebd4979dc8e1181c4982a321377d7580b64"
            },
            "cell_type": "code",
            "source": "cdf = df[\"publishing_hour\"].value_counts().to_frame().reset_index()\\\n        .rename(columns={\"index\": \"publishing_hour\", \"publishing_hour\": \"No_of_videos\"})\nfig, ax = plt.subplots()\n_ = sns.barplot(x=\"publishing_hour\", y=\"No_of_videos\", data=cdf, \n                palette=sns.cubehelix_palette(n_colors=24), ax=ax)\n_ = ax.set(xlabel=\"Publishing Hour\", ylabel=\"No. of videos\")",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "9eaef0f230a0f689b909bd2c0565ca8d9cffd62a"
            },
            "cell_type": "markdown",
            "source": "We can see that the period between 2PM and 7PM, peaking between 4PM and 5PM, had the largest number of trending videos. We notice also that the period between 12AM and 1PM has the smallest number of trending videos. But why is that? Is it because people publish a lot more videos between 2PM and 7PM? Is it because how YouTube algorithm chooses trending videos?"
        },
        {
            "metadata": {
                "_uuid": "831123e7443632df64ff210016722ccc93fc97b6"
            },
            "cell_type": "markdown",
            "source": "### How many trending videos have an error?\nTo see how many trending videos got removed or had some error, we can use `video_error_or_removed` column in the dataset"
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "36775a50027e8302643c12c121ff53bfa2efa825"
            },
            "cell_type": "code",
            "source": "value_counts = df[\"video_error_or_removed\"].value_counts().to_dict()\nfig, ax = plt.subplots()\n_ = ax.pie([value_counts[False], value_counts[True]], labels=['No', 'Yes'], \n        colors=['#003f5c', '#ffa600'], textprops={'color': '#040204'})\n_ = ax.axis('equal')\n_ = ax.set_title('Video Error or Removed?')",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "ff78c9588f04df2a582f2cbc20fa33be8c379970"
            },
            "cell_type": "code",
            "source": "df[\"video_error_or_removed\"].value_counts()",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "659dec5432200bc77c09346df776cd0eb7c7ae74"
            },
            "cell_type": "markdown",
            "source": "We can see that out of videos that appeared on trending list (`40949` videos), there is a tiny portion (`23` videos) with errors."
        },
        {
            "metadata": {
                "_uuid": "479ba6f858f16bbd3deff1f1397eb0d233d2bd0d"
            },
            "cell_type": "markdown",
            "source": "### How many trending videos have their commets disabled?\nTo know that, we use `comments_disabled` column"
        },
        {
            "metadata": {
                "scrolled": false,
                "trusted": true,
                "_uuid": "d424a320b73c2da3be5a4a50559cd1aa367c05de"
            },
            "cell_type": "code",
            "source": "value_counts = df[\"comments_disabled\"].value_counts().to_dict()\nfig, ax = plt.subplots()\n_ = ax.pie(x=[value_counts[False], value_counts[True]], labels=['No', 'Yes'], \n           colors=['#003f5c', '#ffa600'], textprops={'color': '#040204'})\n_ = ax.axis('equal')\n_ = ax.set_title('Comments Disabled?')",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "78cca1f60b800bbbed96d29cc4f546a569b11476"
            },
            "cell_type": "code",
            "source": "df[\"comments_disabled\"].value_counts(normalize=True)",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "a7e78a0c2ab4e904dab111a56c5f0e3d4869583f"
            },
            "cell_type": "markdown",
            "source": "We see that only `2%` of trending videos prevented users from commenting."
        },
        {
            "metadata": {
                "_uuid": "82d522d9e73c4c5f6bedab6abc16d7d86a329cbc"
            },
            "cell_type": "markdown",
            "source": "### How many trending videos have their ratings disabled?\nTo know that, we use `ratings_disabled` column"
        },
        {
            "metadata": {
                "scrolled": false,
                "trusted": true,
                "_uuid": "0fb1b8b343df896394c9d04ac5ce67a92e6094f9"
            },
            "cell_type": "code",
            "source": "value_counts = df[\"ratings_disabled\"].value_counts().to_dict()\nfig, ax = plt.subplots()\n_ = ax.pie([value_counts[False], value_counts[True]], labels=['No', 'Yes'], \n            colors=['#003f5c', '#ffa600'], textprops={'color': '#040204'})\n_ = ax.axis('equal')\n_ = ax.set_title('Ratings Disabled?')",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "trusted": true,
                "_uuid": "a1525acb8ba4602dd839ae796d677fba4a355a1d"
            },
            "cell_type": "code",
            "source": "df[\"ratings_disabled\"].value_counts()",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "18e807062d31353d177bb328981a854c46956ec8"
            },
            "cell_type": "markdown",
            "source": "We see that only `169` trending videos out of `40949` prevented users from commenting.\n\n### How many videos have both comments and ratings disabled?"
        },
        {
            "metadata": {
                "scrolled": false,
                "trusted": true,
                "_uuid": "c9b29a12a06f08151e3ff2073a424a04f71fad77"
            },
            "cell_type": "code",
            "source": "len(df[(df[\"comments_disabled\"] == True) & (df[\"ratings_disabled\"] == True)].index)",
            "execution_count": null,
            "outputs": []
        },
        {
            "metadata": {
                "_uuid": "17bf9d15b7681c92c229f1e6f308fb484297b650"
            },
            "cell_type": "markdown",
            "source": "So there are just `106` trending videos that have both comments and ratings disabled"
        },
        {
            "metadata": {
                "_uuid": "ebd16f14a6910188b58d715e5c0d95966ad53f21"
            },
            "cell_type": "markdown",
            "source": "### Conclusions\nHere are the some of the results we extracted from the analysis:\n* We analyzed a dataset that contains information about YouTube trending videos for 205 days. The dataset was collected in 2017 and 2018. It contains **`40949`** video entry.\n* `71%` of trending videos have less than `1.5` million views, and **`91%`** have less than **`5`** million views.\n* `68%` of trending videos have less than `40,000` likes, and **`84%`** have less than **`100,000`** likes.\n* `67%` of trending videos have less than `4,000` comments, and **`93%`** have less than **`25,000`** comments.\n* Some videos may appear on the trending videos list on more than one day. Our dataset contains `40494` entries but not for `40494` unique videos but for `6351`unique videos.\n* Trending videos that have **`100,000,000`** views and more  have title length between `33` and `55` characters approximately.\n* The delimiters `-` and `|` were common in trending video titles.\n* The words 'Official', 'Video', 'Trailer', 'How', and '2018' were common also in trending video titles.\n* There is a strong positive correlation between the number of views and the number of likes of trending videos: As one of them increases, the other increases, and vice versa.\n* There is a strong positive correlation also between the number of likes and the number of comments, and a slightly weaker one between the number of dislikes and the number of comments.\n* The category that has the largest number of trending videos is **'Entertainment'** with `9,964` videos, followed by 'Music' category with `6,472` videos, followed by 'Howto & Style' category with `4146` videos.\n* On the opposite side, the category that has the smallest number of trending videos is 'Shows' with `57` videos, followed by 'Nonprofits & Activisim' with `57` videos, followed by 'Autos & Vehicles' with `384` videos.\n\n"
        },
        {
            "metadata": {
                "_uuid": "972e2e40fac1a37e480ab390c8daf3de80e8df7f"
            },
            "cell_type": "markdown",
            "source": "### The end\nIf you like this analysis, please consider to **upvote** this analysis at the top of this page.\n\nFollow me on [Twitter](https://twitter.com/ammar_cel) to know when I publish something new, or visit [my website and blog](http://ammar-alyousfi.com)."
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "name": "python",
            "version": "3.6.6",
            "mimetype": "text/x-python",
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "pygments_lexer": "ipython3",
            "nbconvert_exporter": "python",
            "file_extension": ".py"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 1
}