07_analisi_esplorativa_e_preprocessamento_dei_dati.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I dati utilizzati in questo notebook sono stati presi dalla competizione di Analytics Vidhya [Practice Problem: Big Mart Sales III](https://datahack.analyticsvidhya.com/contest/practice-problem-big-mart-sales-iii/#data_dictionary)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analisi esplorativa e preprocessamento dei dati"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [Big Mart Sales](#big_mart_sales)<br>\n",
    "    1.1 [Descrizione](#descrizione)<br>\n",
    "    1.2 [Leggere i dati e separare la variabile risposta](#leggere_dati)<br>\n",
    "2. [Analisi esplorativa: studiare le variabili esplicative](#studiare_esplicative)<br>\n",
    "    2.1 [Contare i valori mancanti](#contare_valori_mancanti)<br>\n",
    "    2.2 [Dividere le variabili quantitative dalle variabili qualitative](#dividere_quantitative_qualitative)<br>\n",
    "    2.3 [Variabili quantitative](#quantitative)<br>\n",
    "    2.4 [Variabili qualitative](#qualitative)<br>\n",
    "3. [Preprocessare i dati](#preprocessare_dati)<br>\n",
    "    3.1 [Riempire i valori mancanti](#riempire_valori_mancanti)<br>\n",
    "    3.2 [Aggregare i livelli simili delle variabili qualitative](#aggregare_livelli_qualitative)<br>\n",
    "    3.3 [Eliminare le colonne che non si intendono utilizzare](#eliminare_colonne)<br>\n",
    "4. [Ottenere gli indici degli insiemi di *training*, *validation* e *test*](#train_val_test)<br>\n",
    "5. [Analisi esplorativa: studiare la relazione tra variabili esplicative e variabile risposta](#studiare_esplicative_risposta)<br>\n",
    "    5.1 [Relazione tra quantitative e risposta](#quantitative_risposta)<br>\n",
    "    5.2 [Relazione tra qualitative e risposta](#qualitative_risposta)<br>\n",
    "    5.3 [Relazione tra esplicative e risposta](#esplicative_risposta)<br>\n",
    "6. [Trasformare le variabili qualitative in dummy](#dummy)<br>\n",
    "7. [Standardizzare i dati](#standardizzare)<br>\n",
    "8. [Pipeline di preprocessamento](#pipeline)<br>\n",
    "    8.1 [Definire una `Pipeline`](#definire_pipeline)<br>\n",
    "    8.2 [Preprocessare i dati attraverso la pipeline](#preprocessare_pipeline)<br>\n",
    "9. [Salvare](#salvare)<br>\n",
    "    9.1 [Salvare i dati preprocessati](#salvare_dati)<br>\n",
    "    9.2 [Salvare la pipeline](#salvare_pipeline)<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import inspect\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Big Mart Sales <a id=big_mart_sales> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Descrizione <a id=descrizione> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem Statement\n",
    "\n",
    "The data scientists at BigMart have collected 2013 sales data for 1559 products across 10 stores in different cities. Also, certain attributes of each product and store have been defined. The aim is to build a predictive model and find out the sales of each product at a particular store.\n",
    "\n",
    "Using this model, BigMart will try to understand the properties of products and stores which play a key role in increasing sales.\n",
    "\n",
    " \n",
    "\n",
    "Please note that the data may have missing values as some stores might not report all the data due to technical glitches. Hence, it will be required to treat them accordingly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "\n",
    "We have train (8523) and test (5681) data set, train data set has both input and output variable(s). You need to predict the sales for test data set.\n",
    "\n",
    "|**Variable**                 | Description                                              |\n",
    "|-----------------------------|----------------------------------------------------------|\n",
    "|**Item_Identifier**          | Unique product ID                                        |\n",
    "|**Item_Weight**              | Weight of product                                        |\n",
    "|**Item_Fat_Content**         | Whether the product is low fat or not                    |\n",
    "|**Item_Visibility**          | The % of total display area of all products in a store allocated<br/>to the particular product|\n",
    "|**Item_Type**                |The category to which the product belongs                 |\n",
    "|**Item_MRP**                 |Maximum Retail Price (list price) of the product          |\n",
    "|**Outlet_Identifier**        |Unique store ID                                           |\n",
    "|**Outlet_Establishment_Year**|The year in which store was established                   |\n",
    "|**Outlet_Size**              |The size of the store in terms of ground area covered     |\n",
    "|**Outlet_Location_Type**     |The type of city in which the store is located            |\n",
    "|**Outlet_Type**              |Whether the outlet is just a grocery store or some sort of<br/>supermarket|\n",
    "|**Item_Outlet_Sales**        |Sales of the product in the particulat store. This is the outcome variable<br/>to be predicted|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Leggere i dati e separare la variabile risposta <a id=leggere_dati> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Leggere i dati"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH = \"datasets/big_mart_sales\" # cambiare in base a dove si è salvato il dataset\n",
    "\n",
    "dati = pd.read_csv(PATH + \"/Train_UWu5bXk.csv\")\n",
    "print(\"Dimensione del dataset: {} x {}\".format(*dati.shape))\n",
    "dati.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dividere le variabili esplicative dalla variabile risposta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "risposta = \"Item_Outlet_Sales\"\n",
    "esplicative = sorted(col for col in dati.columns if col != risposta)\n",
    "\n",
    "X, y = dati[esplicative].copy(), dati[risposta].copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Analisi esplorativa: studiare le variabili esplicative <a id=studiare_esplicative> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Contare i valori mancanti <a id=contare_valori_mancanti> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dati.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Dividere le variabili quantitative dalle variabili qualitative <a id=dividere_quantitative_qualitative> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Controllare i `dtypes` delle colonne"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Salvare i nomi delle colonne in due liste distinte"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quantitative = X.select_dtypes(include=[\"int64\", \"float64\"]).columns.tolist()\n",
    "qualitative = X.select_dtypes(include=[\"object\"]).columns.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Variabili quantitative <a id=quantitative> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X[quantitative].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Descrivere le variabili"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.describe() # nota: vengono automaticamente considerate solo le colonne numeriche"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 10))\n",
    "\n",
    "sns.pairplot(X[quantitative])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4 Variabili qualitative <a id=qualitative> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X[qualitative].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Contare il numero di livelli"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X[qualitative].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Contare il numero di osservazioni per ogni livello"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for col in qualitative:\n",
    "    display(X[col].value_counts().head(16)) # mi limito ai primi 16 valori di ogni livello"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Esercizio\n",
    "\n",
    "Esplorare ulteriormente per via grafica le variabili esplicative (istogrammi, boxplot, ...).\n",
    "\n",
    "> Suggerimento: considerare le librerie [Matplotlib](https://matplotlib.org/), [Seaborn](https://seaborn.pydata.org/) o, per grafici interattivi, [Bokeh](https://bokeh.pydata.org/en/latest/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Esercizio\n",
    "\n",
    "Elencare quanto scoperto grazie all'analisi esplorativa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Preprocessare i dati <a id=preprocessare_dati> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Riempire i valori mancanti <a id=riempire_valori_mancanti> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Studiare la relazione tra *Item_Identifier* e *Item_Weight*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weight_grby_id = X[[\"Item_Identifier\", \"Item_Weight\"]].groupby(\"Item_Identifier\").\\\n",
    "    agg([\"count\", \"min\", \"max\"])[\"Item_Weight\"]\n",
    "weight_grby_id.sort_values(\"count\", inplace=True, ascending=False)\n",
    "\n",
    "print(\"Item_Identifier senza nemmeno un Item_Weight associato: {}\".format((weight_grby_id[\"count\"] == 0).sum()))\n",
    "weight_grby_id = weight_grby_id.loc[weight_grby_id[\"count\"] > 0]\n",
    "print(\"Percentuale di Item_Identifier con almeno un Item_Weight per cui min è uguale a max: {}%\"\n",
    "      .format(100 * (weight_grby_id[\"min\"] == weight_grby_id[\"min\"]).mean()))\n",
    "display(weight_grby_id.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Riempire i valori mancanti di *Item_Weight*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from msbd.preprocessamento import RiempireNAItemWeight\n",
    "\n",
    "print(inspect.getsource(RiempireNAItemWeight))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Valori mancanti di Item_Weight prima della sostituzione: {}\".format(X[\"Item_Weight\"].isnull().sum()))\n",
    "\n",
    "riempire_na_item_weight = RiempireNAItemWeight()\n",
    "\n",
    "X = riempire_na_item_weight.fit_transform(X)\n",
    "\n",
    "print(\"Valori mancanti di Item_Weight dopo la sostituzione: {}\".format(X[\"Item_Weight\"].isnull().sum()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Studiare la relazione tra *Outlet_Location_Type* e *Outlet_Size*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "size_grby_location = X.groupby(\"Outlet_Location_Type\")[\"Outlet_Size\"].value_counts().unstack().fillna(0)\n",
    "\n",
    "size_grby_location"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Studiare la relazione tra *Outlet_Type* e *Outlet_Size*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "size_grby_type = X.groupby(\"Outlet_Type\")[\"Outlet_Size\"].value_counts().unstack().fillna(0)\n",
    "\n",
    "size_grby_type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Riempire i valori mancanti di *Outlet_Size*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from msbd.preprocessamento import RiempireNAOutletSize\n",
    "\n",
    "print(inspect.getsource(RiempireNAOutletSize))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Esercizio\n",
    "\n",
    "Completare il metodo `transform()`della classe `RiempireNAOutletSize` basandosi sui risultati dell'analisi esplorativa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Valori mancanti di Outlet_Size prima della sostituzione: {}\".format(X[\"Outlet_Size\"].isnull().sum()))\n",
    "\n",
    "riempire_na_outlet_size = RiempireNAOutletSize()\n",
    "\n",
    "X = riempire_na_outlet_size.fit_transform(X)\n",
    "\n",
    "print(\"Valori mancanti di Outlet_Size dopo della sostituzione: {}\".format(X[\"Outlet_Size\"].isnull().sum()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Riempire gli ultimi valori mancanti rimasti delle variabili quantitative utilizzando il valor medio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from msbd.preprocessamento import RiempireNAMedia\n",
    "\n",
    "print(inspect.getsource(RiempireNAMedia))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Valori mancanti prima della sostituzione: \\n{}\".format(X.isnull().sum()))\n",
    "\n",
    "riempire_na_media = RiempireNAMedia()\n",
    "\n",
    "X = riempire_na_media.fit_transform(X)\n",
    "\n",
    "print(\"\\nValori mancanti dopo la sostituzione: \\n{}\".format(X.isnull().sum()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Aggregare i livelli simili delle variabili qualitative <a id=aggregare_livelli_qualitative> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregare i livelli simili di *Item_Fat_Content*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from msbd.preprocessamento import Sostituire\n",
    "\n",
    "print(inspect.getsource(Sostituire))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_replace = {\"Item_Fat_Content\": {\"Low Fat\": \"Low_Fat\", \"LF\": \"Low_Fat\", \"low fat\": \"Low_Fat\", \"reg\": \"Regular\"}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Livelli prima della sostituzione: {}\".format(X[\"Item_Fat_Content\"].unique()))\n",
    "\n",
    "sostituire_item_fat_content = Sostituire(to_replace)\n",
    "\n",
    "X = sostituire_item_fat_content.fit_transform(X)\n",
    "\n",
    "print(\"Livelli dopo la sostituzione: {}\".format(X[\"Item_Fat_Content\"].unique()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Eliminare le colonne che non si intendono utilizzare <a id=eliminare_colonne> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Eliminare *Item_Identifier*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from msbd.preprocessamento import Eliminare\n",
    "\n",
    "print(inspect.getsource(Eliminare))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "elim = Eliminare(\"Item_Identifier\")\n",
    "\n",
    "X = elim.fit_transform(X)\n",
    "\n",
    "esplicative.remove(\"Item_Identifier\")\n",
    "qualitative.remove(\"Item_Identifier\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Ottenere gli indici degli insiemi di *training*, *validation* e *test* <a id=train_val_test> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "idx_train, idx_test = train_test_split(y.index.values, test_size=1000, random_state=42)\n",
    "idx_train, idx_val = train_test_split(idx_train, test_size=1000, random_state=42)\n",
    "\n",
    "print(\"Dimensione del training set: {}\".format(len(idx_train)))\n",
    "print(\"Dimensione del validation set: {}\".format(len(idx_val)))\n",
    "print(\"Dimensione del test set: {}\".format(len(idx_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Analisi esplorativa: studiare la relazione tra variabili esplicative e variabile risposta <a id=studiare_esplicative_risposta> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-danger fade in\">\n",
    "<strong>IMPORTANTE</strong>: Le analisi relative (anche) alla variabile risposta vanno effettuate utilizzando solo l'insieme di <em>training</em>. Utilizzare in questa fase anche gli insiemi di <em>validation</em> e/o di <em>test</em> può inficiare in modo più o meno grave le conclusioni che si traggono su di essi.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 Relazione tra quantitative e risposta <a id=quantitative_risposta> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from msbd.grafici import grafico_dispersione_quantitative_risposta\n",
    "\n",
    "print(inspect.getsource(grafico_dispersione_quantitative_risposta))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 10))\n",
    "\n",
    "grafico_dispersione_quantitative_risposta(X.loc[idx_train], y.loc[idx_train], quantitative, 2)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 Relazione tra qualitative e risposta <a id=qualitative_risposta> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from msbd.grafici import grafico_barre_qualitative_risposta\n",
    "\n",
    "print(inspect.getsource(grafico_barre_qualitative_risposta))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 10))\n",
    "\n",
    "grafico_barre_qualitative_risposta(X.loc[idx_train], y.loc[idx_train], qualitative, 2)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3 Relazione tra esplicative e risposta <a id=esplicative_risposta> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "\n",
    "sns.scatterplot(X.loc[idx_train, \"Item_MRP\"], y=y.loc[idx_train],\n",
    "                hue=X.loc[idx_train, \"Outlet_Type\"],\n",
    "                hue_order=[\"Supermarket Type3\", \"Supermarket Type2\", \"Supermarket Type1\", \"Grocery Store\"],\n",
    "                style=X.loc[idx_train, \"Outlet_Location_Type\"],\n",
    "                style_order=[\"Tier 3\", \"Tier 2\", \"Tier 1\"],\n",
    "                size=X.loc[idx_train, \"Outlet_Size\"],\n",
    "                size_order=[\"High\", \"Medium\", \"Small\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Trasformare le variabili qualitative in dummy <a id=dummy> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from msbd.preprocessamento import OttenereDummy\n",
    "\n",
    "print(inspect.getsource(OttenereDummy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Esercizio\n",
    "\n",
    "Per come è definito, il metodo `fit()` della classe `OttenereDummy` crea le variabili dummy solo per salvarne i nomi. Ottenere lo stesso risultato senza utilizzare la funzione `get_dummies()` e senza creare le dummy in `fit()` (le dummy verranno create in `transform()`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"X prima della creazione delle variabili dummy:\")\n",
    "display(X.head(2))\n",
    "\n",
    "od = OttenereDummy(drop_first=True)\n",
    "\n",
    "X = od.fit_transform(X)\n",
    "\n",
    "print(\"\\nX dopo la creazione delle variabili dummy:\")\n",
    "X.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Esercizio\n",
    "\n",
    "Perché abbiamo scelto `drop_first=True`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Standardizzare i dati <a id=standardizzare> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from msbd.preprocessamento import Standardizzare\n",
    "\n",
    "print(inspect.getsource(Standardizzare))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"X prima della standardizzazione:\")\n",
    "display(X.head(2))\n",
    "\n",
    "std = Standardizzare()\n",
    "\n",
    "X = std.fit_transform(X)\n",
    "\n",
    "print(\"X dopo la standardizzazione:\")\n",
    "X.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Esercizio\n",
    "\n",
    "1. Completare il metodo `inverse_transform()` della classe `Standardizzare`;\n",
    "2. Verificare il corretto funzionamento di `inverse_transform()` utilizzando `pytest`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8. Pipeline di preprocessamento <a id=pipeline> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.1 Definire una `Pipeline` <a id=definire_pipeline> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preproc = Pipeline([\n",
    "    (\"riempire_na_item_weight\", RiempireNAItemWeight()),\n",
    "    (\"riempire_na_outlet_size\", RiempireNAOutletSize()),\n",
    "    (\"riempire_na_media\", RiempireNAMedia()),\n",
    "    (\"eliminare_item_identifier\", Eliminare(columns=\"Item_Identifier\")),\n",
    "    (\"ottenere_dummy\", OttenereDummy(drop_first=True)),\n",
    "    (\"standardizzare\", Standardizzare()),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.2 Preprocessare i dati attraverso la pipeline <a id=preprocessare_pipeline> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "risposta = \"Item_Outlet_Sales\"\n",
    "esplicative = sorted(col for col in dati.columns if col != risposta)\n",
    "\n",
    "X, y = dati[esplicative].copy(), dati[risposta].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Dati grezzi:\")\n",
    "display(X.head(2))\n",
    "\n",
    "X_preproc = preproc.fit_transform(X)\n",
    "\n",
    "print(\"Dati preprocessati:\")\n",
    "X_preproc.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Esercizio\n",
    "\n",
    "Verificare che la pipeline costruita applichi tutte le trasformazioni viste in questo notebook analizzando `X_preproc`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9. Salvare <a id=salvare> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "OUTPUT_DIR = \"output/07\"\n",
    "\n",
    "os.makedirs(OUTPUT_DIR, exist_ok=True) # se non esiste, crea la cartella finale e tutte le intermedie"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.1 Salvare i dati preprocessati <a id=salvare_dati> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_preproc.loc[idx_train].to_pickle(OUTPUT_DIR + \"/X_train.pkl\")\n",
    "X_preproc.loc[idx_val].to_pickle(OUTPUT_DIR + \"/X_val.pkl\")\n",
    "X_preproc.loc[idx_test].to_pickle(OUTPUT_DIR + \"/X_test.pkl\")\n",
    "y.loc[idx_train].to_pickle(OUTPUT_DIR + \"/y_train.pkl\")\n",
    "y.loc[idx_val].to_pickle(OUTPUT_DIR + \"/y_val.pkl\")\n",
    "y.loc[idx_test].to_pickle(OUTPUT_DIR + \"/y_test.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.2 Salvare la pipeline <a id=salvare_pipeline> </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.externals import joblib\n",
    "\n",
    "joblib.dump(preproc, OUTPUT_DIR + \"/preproc.joblib\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}