performance_metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Performance Metrics\n",
    "\n",
    "They help evaluate the performace of ML algorithms, both classification as well as regression algorithms\n",
    "\n",
    "There are different perfomance metrics for classification and regression problems\n",
    "\n",
    "## 1. Performance metrics for classification\n",
    "\n",
    "\n",
    "Some of the performance metrics evaluation for classification are:- \n",
    "\n",
    "- Accuracy\n",
    "- Confusion Matrix\n",
    "- Precision\n",
    "- Recall\n",
    "- F-Score\n",
    "- AUC (Area Under the Curve)-ROC\n",
    "\n",
    "### 1. Accuracy\n",
    "\n",
    "It can be determined as the number of correct predictions to the total number of predictions\n",
    "\n",
    "![accuracy](accuracy.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### When to use accuracy\n",
    "\n",
    "When the target variable classes in the data are appproximately balanced\n",
    "\n",
    "\n",
    "#### When not use accuracy\n",
    "\n",
    "When the target variable majorly belongs to one class\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Confusion Matrix\n",
    "\n",
    "It's a tabular representation of prediction outcomes of any any binary classifier\n",
    "\n",
    "The matrix compares the actual target values with those predicted by the ML model\n",
    "\n",
    "The table has two dimensions, `Actual` and `Predicted`\n",
    "\n",
    "Both dimensions have `True Positives (TP)`, `False Positives (FP)`, `False Negatives (FN)` and `True Negatives (TN)`\n",
    "\n",
    "For binary classification the matrix will be a 2x2 table, for multi-class the matrix shape will be equal to the number of classes\n",
    "\n",
    "![example confusion matrix](confusion_matrix.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhcAAAHHCAYAAAAMD3r6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAABD2UlEQVR4nO3deXiNd/7/8ddJyElKFkrEmtia2FXRRqaW2pcK7bSoaULpTFtKUV90E1ETrdpaRRfEUtrqENOVoCiitQVVlFpSJNQuWkHO/fujP2d6JCEn7uNkeT7muq8r+Zz7vj/vO5dM3n1/lttiGIYhAAAAk3i4OwAAAFC4kFwAAABTkVwAAABTkVwAAABTkVwAAABTkVwAAABTkVwAAABTkVwAAABTkVwAAABTkVwALrR//361a9dO/v7+slgsSkhIMPX+hw8flsViUXx8vKn3Lchatmypli1bujsMoEgjuUCh98svv+hf//qXqlWrJm9vb/n5+SkiIkJTp07VH3/84dK+o6OjtWvXLo0bN07z589X48aNXdrfndSnTx9ZLBb5+fll+3Pcv3+/LBaLLBaL3nrrLafvf/z4ccXExCg5OdmEaAHcScXcHQDgSl9++aUee+wxWa1WRUVFqW7durpy5YrWr1+v4cOHa/fu3Xr//fdd0vcff/yhpKQkvfzyyxo4cKBL+ggODtYff/yh4sWLu+T+t1KsWDH9/vvv+vzzz/X44487fPbRRx/J29tbly9fztO9jx8/rjFjxigkJEQNGzbM9XUrVqzIU38AzENygULr0KFD6tmzp4KDg7V69WqVL1/e/tmAAQN04MABffnlly7r/7fffpMkBQQEuKwPi8Uib29vl93/VqxWqyIiIrRo0aIsycXChQvVuXNn/ec//7kjsfz++++666675OXldUf6A5AzhkVQaL355ptKT0/XrFmzHBKL62rUqKHBgwfbv7927ZrGjh2r6tWry2q1KiQkRC+99JIyMjIcrgsJCVGXLl20fv16NW3aVN7e3qpWrZrmzZtnPycmJkbBwcGSpOHDh8tisSgkJETSn8MJ17/+q5iYGFksFoe2xMRE/e1vf1NAQIBKliyp0NBQvfTSS/bPc5pzsXr1aj344IMqUaKEAgICFBkZqT179mTb34EDB9SnTx8FBATI399fffv21e+//57zD/YGTzzxhL7++mudO3fO3rZ582bt379fTzzxRJbzz5w5oxdffFH16tVTyZIl5efnp44dO2rHjh32c9asWaMmTZpIkvr27WsfXrn+nC1btlTdunW1detWNW/eXHfddZf953LjnIvo6Gh5e3tnef727durVKlSOn78eK6fFUDukFyg0Pr8889VrVo1NWvWLFfn9+/fX6+99poaNWqkyZMnq0WLFoqLi1PPnj2znHvgwAH9/e9/V9u2bTVx4kSVKlVKffr00e7duyVJjzzyiCZPnixJ6tWrl+bPn68pU6Y4Ff/u3bvVpUsXZWRkKDY2VhMnTlTXrl21YcOGm163cuVKtW/fXidPnlRMTIyGDh2qjRs3KiIiQocPH85y/uOPP66LFy8qLi5Ojz/+uOLj4zVmzJhcx/nII4/IYrFoyZIl9raFCxcqLCxMjRo1ynL+wYMHlZCQoC5dumjSpEkaPny4du3apRYtWtj/0NeqVUuxsbGSpH/+85+aP3++5s+fr+bNm9vvc/r0aXXs2FENGzbUlClT1KpVq2zjmzp1qsqWLavo6GhlZmZKkt577z2tWLFC77zzjipUqJDrZwWQSwZQCJ0/f96QZERGRubq/OTkZEOS0b9/f4f2F1980ZBkrF692t4WHBxsSDLWrVtnbzt58qRhtVqNYcOG2dsOHTpkSDImTJjgcM/o6GgjODg4SwyjR482/vorOXnyZEOS8dtvv+UY9/U+5syZY29r2LChERgYaJw+fdretmPHDsPDw8OIiorK0t9TTz3lcM/u3bsbd999d459/vU5SpQoYRiGYfz97383WrdubRiGYWRmZhpBQUHGmDFjsv0ZXL582cjMzMzyHFar1YiNjbW3bd68OcuzXdeiRQtDkjFz5sxsP2vRooVD2/Llyw1Jxuuvv24cPHjQKFmypNGtW7dbPiOAvKFygULpwoULkiRfX99cnf/VV19JkoYOHerQPmzYMEnKMjejdu3aevDBB+3fly1bVqGhoTp48GCeY77R9bkay5Ytk81my9U1qampSk5OVp8+fVS6dGl7e/369dW2bVv7c/7VM8884/D9gw8+qNOnT9t/hrnxxBNPaM2aNUpLS9Pq1auVlpaW7ZCI9Oc8DQ+PP/+vJzMzU6dPn7YP+Wzbti3XfVqtVvXt2zdX57Zr107/+te/FBsbq0ceeUTe3t567733ct0XAOeQXKBQ8vPzkyRdvHgxV+cfOXJEHh4eqlGjhkN7UFCQAgICdOTIEYf2KlWqZLlHqVKldPbs2TxGnFWPHj0UERGh/v37q1y5curZs6c+/fTTmyYa1+MMDQ3N8lmtWrV06tQpXbp0yaH9xmcpVaqUJDn1LJ06dZKvr68++eQTffTRR2rSpEmWn+V1NptNkydPVs2aNWW1WlWmTBmVLVtWO3fu1Pnz53PdZ8WKFZ2avPnWW2+pdOnSSk5O1ttvv63AwMBcXwvAOSQXKJT8/PxUoUIF/fjjj05dd+OEypx4enpm224YRp77uD4f4DofHx+tW7dOK1eu1JNPPqmdO3eqR48eatu2bZZzb8ftPMt1VqtVjzzyiObOnaulS5fmWLWQpH//+98aOnSomjdvrgULFmj58uVKTExUnTp1cl2hkf78+Thj+/btOnnypCRp165dTl0LwDkkFyi0unTpol9++UVJSUm3PDc4OFg2m0379+93aD9x4oTOnTtnX/lhhlKlSjmsrLjuxuqIJHl4eKh169aaNGmSfvrpJ40bN06rV6/Wt99+m+29r8e5b9++LJ/t3btXZcqUUYkSJW7vAXLwxBNPaPv27bp48WK2k2Cv++yzz9SqVSvNmjVLPXv2VLt27dSmTZssP5PcJnq5cenSJfXt21e1a9fWP//5T7355pvavHmzafcH4IjkAoXW//3f/6lEiRLq37+/Tpw4keXzX375RVOnTpX0Z1lfUpYVHZMmTZIkde7c2bS4qlevrvPnz2vnzp32ttTUVC1dutThvDNnzmS59vpmUjcuj72ufPnyatiwoebOnevwx/rHH3/UihUr7M/pCq1atdLYsWM1bdo0BQUF5Xiep6dnlqrI4sWLdezYMYe260lQdomYs0aMGKGUlBTNnTtXkyZNUkhIiKKjo3P8OQK4PWyihUKrevXqWrhwoXr06KFatWo57NC5ceNGLV68WH369JEkNWjQQNHR0Xr//fd17tw5tWjRQj/88IPmzp2rbt265bjMMS969uypESNGqHv37ho0aJB+//13zZgxQ/fcc4/DhMbY2FitW7dOnTt3VnBwsE6ePKnp06erUqVK+tvf/pbj/SdMmKCOHTsqPDxc/fr10x9//KF33nlH/v7+iomJMe05buTh4aFXXnnllud16dJFsbGx6tu3r5o1a6Zdu3bpo48+UrVq1RzOq169ugICAjRz5kz5+vqqRIkSuv/++1W1alWn4lq9erWmT5+u0aNH25fGzpkzRy1bttSrr76qN99806n7AcgFN69WAVzu559/Np5++mkjJCTE8PLyMnx9fY2IiAjjnXfeMS5fvmw/7+rVq8aYMWOMqlWrGsWLFzcqV65sjBo1yuEcw/hzKWrnzp2z9HPjEsiclqIahmGsWLHCqFu3ruHl5WWEhoYaCxYsyLIUddWqVUZkZKRRoUIFw8vLy6hQoYLRq1cv4+eff87Sx43LNVeuXGlEREQYPj4+hp+fn/Hwww8bP/30k8M51/u7canrnDlzDEnGoUOHcvyZGobjUtSc5LQUddiwYUb58uUNHx8fIyIiwkhKSsp2CemyZcuM2rVrG8WKFXN4zhYtWhh16tTJts+/3ufChQtGcHCw0ahRI+Pq1asO5w0ZMsTw8PAwkpKSbvoMAJxnMQwnZm0BAADcAnMuAACAqUguAACAqUguAACAqUguAAAohK6/+fivR1hY2E2vWbx4scLCwuTt7a169epl+8qA3CC5AACgkKpTp45SU1Ptx/r163M8d+PGjerVq5f69eun7du3q1u3burWrZvTOx1LEqtFAAAohGJiYpSQkKDk5ORcnd+jRw9dunRJX3zxhb3tgQceUMOGDTVz5kyn+qZyAQBAAZGRkaELFy44HDfbaXb//v2qUKGCqlWrpt69eyslJSXHc5OSktSmTRuHtvbt2+fqFQo3KpQ7dF49Zd5rr4HCxKfCg7c+CShirl05duuTbpNZf5fips3TmDFjHNpGjx6d7e67999/v+Lj4xUaGqrU1FSNGTNGDz74oH788Uf5+vpmOT8tLU3lypVzaCtXrpzS0tKcjrNQJhcAABRGo0aN0tChQx3arFZrtud27NjR/nX9+vV1//33Kzg4WJ9++qn69evn0jhJLgAAcDVbpim3sVqtOSYTtxIQEKB77rlHBw4cyPbzoKCgLC95PHHixE1fRJgT5lwAAOBqhs2c4zakp6frl19+Ufny5bP9PDw8XKtWrXJoS0xMVHh4uNN9kVwAAOBqNps5hxNefPFFrV27VocPH9bGjRvVvXt3eXp6qlevXpKkqKgojRo1yn7+4MGD9c0332jixInau3evYmJitGXLFg0cONDpx2VYBACAQujo0aPq1auXTp8+rbJly+pvf/ubNm3apLJly0qSUlJS5OHxvxpDs2bNtHDhQr3yyit66aWXVLNmTSUkJKhu3bpO910o97lgtQiQPVaLAFndidUiV47vNuU+XhXqmHIfV6NyAQCAqzk5pFHQMecCAACYisoFAACudpsrPQoakgsAAFzNpH0uCgqGRQAAgKmoXAAA4GoMiwAAAFOxWgQAACDvqFwAAOBiBsMiAADAVEVsWITkAgAAVytilQvmXAAAAFNRuQAAwNWK2CZaJBcAALgawyIAAAB5R+UCAABXY7UIAAAwFcMiAAAAeUflAgAAV2NYBAAAmMkwitZSVIZFAACAqahcAADgakVsQifJBQAArsacCwAAYKoiVrlgzgUAADAVlQsAAFyNF5cBAABTMSwCAACQd1QuAABwNVaLAAAAUzEsAgAAkHdULgAAcDWGRQAAgKmKWHLBsAgAADAVlQsAAFyMV64DAABz2WzmHLdh/PjxslgseuGFF3I8Jz4+XhaLxeHw9vZ2ui8qFwAAuJqbl6Ju3rxZ7733nurXr3/Lc/38/LRv3z779xaLxen+qFwAAFCIpaenq3fv3vrggw9UqlSpW55vsVgUFBRkP8qVK+d0nyQXAAC4mknDIhkZGbpw4YLDkZGRcdOuBwwYoM6dO6tNmza5CjU9PV3BwcGqXLmyIiMjtXv3bqcfl+QCAABXM2ymHHFxcfL393c44uLicuz2448/1rZt2256zl+FhoZq9uzZWrZsmRYsWCCbzaZmzZrp6NGjTj2uxTAMw6krCoCrpw66OwQgX/Kp8KC7QwDynWtXjrm8jz9WzjTlPh4P9s1SqbBarbJarVnO/fXXX9W4cWMlJiba51q0bNlSDRs21JQpU3LV39WrV1WrVi316tVLY8eOzXWcTOgEAMDVTNpEK6dEIjtbt27VyZMn1ahRI3tbZmam1q1bp2nTpikjI0Oenp43vUfx4sV177336sCBA07FSXIBAICruWG1SOvWrbVr1y6Htr59+yosLEwjRoy4ZWIh/ZmM7Nq1S506dXKqb5ILAAAKIV9fX9WtW9ehrUSJErr77rvt7VFRUapYsaJ9TkZsbKweeOAB1ahRQ+fOndOECRN05MgR9e/f36m+SS4AAHC1fPpukZSUFHl4/G9tx9mzZ/X0008rLS1NpUqV0n333aeNGzeqdu3aTt2XCZ1AEcKETiCrOzKh88spptzHp/MLptzH1ViKCgAATMWwCAAArubm7b/vNJILAABcLZ/OuXAVkgsAAFytiFUumHMBAABMReUCAABXY1gEAACYimERAACAvKNyAQCAqzEsAgAATFXEkguGRQAAgKmoXAAA4GqF7zVeN0VyAQCAqzEsAgAAkHdULgAAcLUiVrkguQAAwNWK2CZaJBcAALhaEatcMOcCAACYisoFAACuxlJUAABgKoZFAAAA8o7KBQAArlbEKhckFwAAuFoRW4rKsAgAADAVlQsAAFzMsLFaBAAAmKmIzblgWAQAAJiKygUAAK5WxCZ0klwAAOBqzLkAAACmKmJzLvJFcnHvvffKYrFkabdYLPL29laNGjXUp08ftWrVyg3RAQAAZ+SLCZ0dOnTQwYMHVaJECbVq1UqtWrVSyZIl9csvv6hJkyZKTU1VmzZttGzZMneHCgCA82w2c44CIl9ULk6dOqVhw4bp1VdfdWh//fXXdeTIEa1YsUKjR4/W2LFjFRkZ6aYoAQDIoyL2VtR8Ubn49NNP1atXryztPXv21KeffipJ6tWrl/bt23enQwMAAE7KF8mFt7e3Nm7cmKV948aN8vb2liTZbDb718jf3p21QHUjOjocD/d62t1hAW734N/uV8LSeKUc3qprV46pa9f27g4Jd0oRGxbJF8nF888/r2eeeUaDBw/WggULtGDBAg0ePFjPPvusBg0aJElavny5GjZs6N5AkWs1qgZrzX8/sh/zZrzl7pAAtytR4i7t3PmTnh/8srtDwZ1mM8w5bsP48eNlsVj0wgsv3PS8xYsXKywsTN7e3qpXr56++uorp/vKF3MuXnnlFVWtWlXTpk3T/PnzJUmhoaH64IMP9MQTT0iSnnnmGT377LPuDBNO8PT0VJm7S7s7DCBf+Wb5t/pm+bfuDgNF0ObNm/Xee++pfv36Nz1v48aN6tWrl+Li4tSlSxctXLhQ3bp107Zt21S3bt1c95cvkgtJ6t27t3r37p3j5z4+PncwGtyulKPH1Kprb1mtXmpQJ0wvPNNX5YMC3R0WALiHG3foTE9PV+/evfXBBx/o9ddfv+m5U6dOVYcOHTR8+HBJ0tixY5WYmKhp06Zp5syZue4zXwyLXLd161b7sMj27dvdHQ7yqH7tUL3+8jDNnPS6Xn1xoI6mnlDUc8N16dLv7g4NANzDjcMiAwYMUOfOndWmTZtbnpuUlJTlvPbt2yspKcmpPvNF5eLkyZPq2bOn1qxZo4CAAEnSuXPn1KpVK3388ccqW7ZsjtdmZGQoIyPDoc0jI0NWq9WVIeMmHgxvYv86tEZV1asdqnaPRuub1d/p0YeZwAYAeZXd3zyr1Zrj37yPP/5Y27Zt0+bNm3N1/7S0NJUrV86hrVy5ckpLS3MqznxRuXj++ed18eJF7d69W2fOnNGZM2f0448/6sKFC/YJnTmJi4uTv7+/w/HG1NyXbuB6fr4lFVy5olKOHnd3KADgFobNZsqR3d+8uLi4bPv89ddfNXjwYH300Ud3fLVlvqhcfPPNN1q5cqVq1aplb6tdu7beffddtWvX7qbXjho1SkOHDnVo87h4zCVxIm9+//0P/XosVQ93aO3uUADAPUx6cVl2f/Nyqlps3bpVJ0+eVKNGjextmZmZWrdunaZNm6aMjAx5eno6XBMUFKQTJ044tJ04cUJBQUFOxZkvkgubzabixYtnaS9evLhst1jXm1056OqVU6bGB+dMmPaBWkbcrwpB5XTy1Gm9++ECeXp6qFObFu4ODXCrEiXuUo0aVe3fVw2pogYN6ujMmbP69Vcqe4WaSRM6bzYEcqPWrVtr165dDm19+/ZVWFiYRowYkSWxkKTw8HCtWrXKYblqYmKiwsPDnYozXyQXDz30kAYPHqxFixapQoUKkqRjx45pyJAhat2a/9otaE6cPKX/G/2Gzl24oNIB/rq3fh199N5klS4V4O7QALdqfF8DrVr5mf37iW/FSJLmzvtU/foPcVNUKKx8fX2zLB8tUaKE7r77bnt7VFSUKlasaB9aGTx4sFq0aKGJEyeqc+fO+vjjj7Vlyxa9//77TvWdL5KLadOmqWvXrgoJCVHlypUl/TlWVLduXS1YsMDN0cFZb8WOcncIQL60dl2SinlVdHcYcAeThkXMlpKSIg+P/02/bNasmRYuXKhXXnlFL730kmrWrKmEhASn9riQJIth5I+3qRiGoVWrVmnPnj2SpFq1auVq2Ux2rp46aGZoQKHhU+FBd4cA5DvXrrh+nt6lmKzvz8qLEjGLTLmPq7m9cmGz2RQfH68lS5bo8OHDslgsqlq1qvz9/WUYhiwWi7tDBAAATnDrUlTDMNS1a1f1799fx44dU7169VSnTh0dOXJEffr0Uffu3d0ZHgAA5sgH7xa5k9xauYiPj9e6deu0atUqtWrVyuGz1atXq1u3bpo3b56ioqLcFCEAACZw4/bf7uDWysWiRYv00ksvZUkspD9XkIwcOVIfffSRGyIDAAB55dbkYufOnerQoUOOn3fs2FE7duy4gxEBAOACDIvcOWfOnMmyh/lflStXTmfPnr2DEQEAYD7jFhtCFjZurVxkZmaqWLGc8xtPT09du3btDkYEAABul1srF4ZhqE+fPjluZXrjm98AACiQCtCQhhncmlxER0ff8hxWigAACjySiztnzpw57uweAIA7g6WoAAAAeef27b8BACj0GBYBAABmMopYcsGwCAAAMBWVCwAAXK2IVS5ILgAAcDV26AQAAMg7KhcAALgawyIAAMBURSy5YFgEAACYisoFAAAuZhhFq3JBcgEAgKsVsWERkgsAAFytiCUXzLkAAACmonIBAICLFbV3i5BcAADgakUsuWBYBAAAmIrKBQAArla0Xi1CcgEAgKsVtTkXDIsAAABTUbkAAMDViljlguQCAABXK2JzLhgWAQAApqJyAQCAixW1CZ0kFwAAuBrDIgAAwEyGzTDlcMaMGTNUv359+fn5yc/PT+Hh4fr6669zPD8+Pl4Wi8Xh8Pb2ztPzUrkAAKAQqlSpksaPH6+aNWvKMAzNnTtXkZGR2r59u+rUqZPtNX5+ftq3b5/9e4vFkqe+SS4AAHA1NwyLPPzwww7fjxs3TjNmzNCmTZtyTC4sFouCgoJuu2+GRQAAcDHDZs6RV5mZmfr444916dIlhYeH53heenq6goODVblyZUVGRmr37t156o/KBQAABURGRoYyMjIc2qxWq6xWa7bn79q1S+Hh4bp8+bJKliyppUuXqnbt2tmeGxoaqtmzZ6t+/fo6f/683nrrLTVr1ky7d+9WpUqVnIqTygUAAK5mM+eIi4uTv7+/wxEXF5djt6GhoUpOTtb333+vZ599VtHR0frpp5+yPTc8PFxRUVFq2LChWrRooSVLlqhs2bJ67733nH5ci2EYhW7x7dVTB90dApAv+VR40N0hAPnOtSvHXN7HqY4tTLmPb8IKpyoXN2rTpo2qV6+e64ThscceU7FixbRo0SKn4mRYBACAAsKZRCI7NpstS3KSk8zMTO3atUudOnVyuh+SCwAAXM0Nq0VGjRqljh07qkqVKrp48aIWLlyoNWvWaPny5ZKkqKgoVaxY0T6sEhsbqwceeEA1atTQuXPnNGHCBB05ckT9+/d3um+SCwAAXOx2Vnrk1cmTJxUVFaXU1FT5+/urfv36Wr58udq2bStJSklJkYfH/6Zenj17Vk8//bTS0tJUqlQp3Xfffdq4cWOOE0BvhjkXQBHCnAsgqzsx5+Jka3PmXASuWmvKfVyN1SIAAMBUDIsAAOBi7hgWcSeSCwAAXM3I2zs6CiqGRQAAgKmoXAAA4GIMiwAAAFMZNoZFAAAA8ozKBQAALsawCAAAMJXBahEAAIC8y1NyceLECT355JOqUKGCihUrJk9PT4cDAAD8j2Ez5ygo8jQs0qdPH6WkpOjVV19V+fLlZbEUrXIPAADOKGqrRfKUXKxfv17fffedGjZsaHI4AAAUPoXvFaE3l6dhkcqVK6sQvkwVAACYIE/JxZQpUzRy5EgdPnzY5HAAACh8DJvFlKOgyNOwSI8ePfT777+revXquuuuu1S8eHGHz8+cOWNKcAAAFAYFKTEwQ56SiylTppgcBgAAKCzylFxER0ebHQcAAIVWUZummOcdOjMzM5WQkKA9e/ZIkurUqaOuXbuyzwUAADdgWCQXDhw4oE6dOunYsWMKDQ2VJMXFxaly5cr68ssvVb16dVODBAAABUeeVosMGjRI1atX16+//qpt27Zp27ZtSklJUdWqVTVo0CCzYwQAoEAzDIspR0GRp8rF2rVrtWnTJpUuXdredvfdd2v8+PGKiIgwLTgAAAqDgrR1txnyVLmwWq26ePFilvb09HR5eXnddlAAAKDgylNy0aVLF/3zn//U999/L8MwZBiGNm3apGeeeUZdu3Y1O0YAAAo0m2Ex5Sgo8pRcvP3226pevbrCw8Pl7e0tb29vRUREqEaNGpo6darZMQIAUKAx5yIXAgICtGzZMu3fv1979+6VJNWqVUs1atQwNTgAAAoDlqI6oWbNmqpZs6ZZsQAAgEIg18nF0KFDNXbsWJUoUUJDhw696bmTJk267cAAACgs2KEzB9u3b9fVq1ftXwMAgNxhWCQH3377bbZfAwAA/FWeVos89dRT2e5zcenSJT311FO3HRQAAIUJS1FzYe7cufrjjz+ytP/xxx+aN2/ebQcFAEBhwlLUm7hw4YJ906yLFy/K29vb/llmZqa++uorBQYGmh4kAAAoOJxKLgICAmSxWGSxWHTPPfdk+dxisWjMmDGmBQcAQGHAapGb+Pbbb2UYhh566CH95z//cXhxmZeXl4KDg1WhQgXTgwQAoCArSPMlzOBUctGiRQtJ0qFDh1SlShVZLEXrhwUAAG4tTxM6V69erc8++yxL++LFizV37tzbDgoAgMLEHRM6Z8yYofr168vPz09+fn4KDw/X119/fdNrFi9erLCwMHl7e6tevXr66quv8vS8eUou4uLiVKZMmSztgYGB+ve//52nQAAAKKwMw5zDGZUqVdL48eO1detWbdmyRQ899JAiIyO1e/fubM/fuHGjevXqpX79+mn79u3q1q2bunXrph9//NHp57UYhvPTTLy9vbV3716FhIQ4tB8+fFi1atXKdpnqnXT11EG39g/kVz4VHnR3CEC+c+3KMZf3saVSN1Pu0/howm1dX7p0aU2YMEH9+vXL8lmPHj106dIlffHFF/a2Bx54QA0bNtTMmTOd6idPlYvAwEDt3LkzS/uOHTt099135+WWAADgFjIyMnThwgWHIyMj45bXZWZm6uOPP9alS5cUHh6e7TlJSUlq06aNQ1v79u2VlJTkdJx5eitqr169NGjQIPn6+qp58+aSpLVr12rw4MHq2bNnXm5pKv7rDMjepsAm7g4BKJLM2gArLi4uy5YPo0ePVkxMTLbn79q1S+Hh4bp8+bJKliyppUuXqnbt2tmem5aWpnLlyjm0lStXTmlpaU7HmafkYuzYsTp8+LBat26tYsX+vIXNZlNUVBRzLgAAuIFZS1FHjRqV5c3kVqs1x/NDQ0OVnJys8+fP67PPPlN0dLTWrl2bY4JhljwlF15eXvrkk080duxY7dixQz4+PqpXr56Cg4PNjg8AAPx/Vqv1psnEjby8vFSjRg1J0n333afNmzdr6tSpeu+997KcGxQUpBMnTji0nThxQkFBQU7Hmafk4rp77rkn2506AQDA/+SXDTptNluOczTCw8O1atUqvfDCC/a2xMTEHOdo3Eyuk4uhQ4dq7NixKlGiRJaSzI0mTZrkdCAAABRW7tihc9SoUerYsaOqVKmiixcvauHChVqzZo2WL18uSYqKilLFihUVFxcnSRo8eLBatGihiRMnqnPnzvr444+1ZcsWvf/++073nevkYvv27bp69ar965ywaycAAO538uRJRUVFKTU1Vf7+/qpfv76WL1+utm3bSpJSUlLk4fG/RaPNmjXTwoUL9corr+ill15SzZo1lZCQoLp16zrdd572ucjvinlVdHcIQL7EahEgq9vdOyI3NgT93ZT7RKRl3R07P7qtORcAAODWbO4O4A7LdXLxyCOP5PqmS5YsyVMwAACg4Mt1cuHv72//2jAMLV26VP7+/mrcuLEkaevWrTp37pxTSQgAAEWBoaI1HzHXycWcOXPsX48YMUKPP/64Zs6cKU9PT0l/bi363HPPyc/Pz/woAQAowGyFbnbjzeXp3SKzZ8/Wiy++aE8sJMnT01NDhw7V7NmzTQsOAIDCwCaLKUdBkafk4tq1a9q7d2+W9r1798pmK2rTVgAAwF/labVI37591a9fP/3yyy9q2rSpJOn777/X+PHj1bdvX1MDBACgoGPORS689dZbCgoK0sSJE5WamipJKl++vIYPH65hw4aZGiAAAAVdUavp3/YmWhcuXJCkfDWRk020gOyxiRaQ1Z3YRCuxXA9T7tP2xCem3MfV8jTnQvpz3sXKlSu1aNEi+5bfx48fV3p6umnBAQBQGBiymHIUFHkaFjly5Ig6dOiglJQUZWRkqG3btvL19dUbb7yhjIwMzZw50+w4AQAosIrasEieKheDBw9W48aNdfbsWfn4+Njbu3fvrlWrVpkWHAAAKHjyVLn47rvvtHHjRnl5eTm0h4SE6NixY6YEBgBAYVHUKhd5Si5sNpsyMzOztB89elS+vr63HRQAAIVJQZovYYY8DYu0a9dOU6ZMsX9vsViUnp6u0aNHq1OnTmbFBgAACqA873PRoUMH1a5dW5cvX9YTTzyh/fv3q0yZMlq0aJHZMQIAUKDZilbhIm/JReXKlbVjxw598skn2rFjh9LT09WvXz/17t3bYYInAABQgXoviBmcTi6uXr2qsLAwffHFF+rdu7d69+7tirgAACg0ithLUZ2fc1G8eHFdvnzZFbEAAIBCIE8TOgcMGKA33nhD165dMzseAAAKHZtJR0GRpzkXmzdv1qpVq7RixQrVq1dPJUqUcPh8yZIlpgQHAEBhYLMw5+KWAgIC9Oijj5odCwAAKAScSi5sNpsmTJign3/+WVeuXNFDDz2kmJgYVogAAHATTOi8iXHjxumll15SyZIlVbFiRb399tsaMGCAq2IDAKBQKGpzLpxKLubNm6fp06dr+fLlSkhI0Oeff66PPvpINltBemQAAOBKTiUXKSkpDtt7t2nTRhaLRcePHzc9MAAACgubxZyjoHBqzsW1a9fk7e3t0Fa8eHFdvXrV1KAAAChM2KHzJgzDUJ8+fWS1Wu1tly9f1jPPPOOwHJWlqAAAFF1OJRfR0dFZ2v7xj3+YFgwAAIVRUVst4lRyMWfOHFfFAQBAoVWQ5kuYIU+baAEAgNwramsq8/RuEQAAgJxQuQAAwMWYcwEAAExV1OZcMCwCAABMRXIBAICLuePdInFxcWrSpIl8fX0VGBiobt26ad++fTe9Jj4+XhaLxeG4cfPM3CC5AADAxdyRXKxdu1YDBgzQpk2blJiYqKtXr6pdu3a6dOnSTa/z8/NTamqq/Thy5IiTPTPnAgCAQumbb75x+D4+Pl6BgYHaunWrmjdvnuN1FotFQUFBt9U3lQsAAFzMsJhzZGRk6MKFCw5HRkZGrmI4f/68JKl06dI3PS89PV3BwcGqXLmyIiMjtXv3bqefl+QCAAAXM2tYJC4uTv7+/g5HXFzcrfu32fTCCy8oIiJCdevWzfG80NBQzZ49W8uWLdOCBQtks9nUrFkzHT161KnntRiGUeiW3xbzqujuEIB8aVNgE3eHAOQ7jY8muLyP6ZXNeQ9XvwOzslQqrFarwwtFs/Pss8/q66+/1vr161WpUqVc93f16lXVqlVLvXr10tixY3N9HXMuAABwMbO2/85NInGjgQMH6osvvtC6deucSiwkqXjx4rr33nt14MABp65jWAQAABczTDqc6tMwNHDgQC1dulSrV69W1apVnY47MzNTu3btUvny5Z26jsoFAAAu5o4dOgcMGKCFCxdq2bJl8vX1VVpamiTJ399fPj4+kqSoqChVrFjRPm8jNjZWDzzwgGrUqKFz585pwoQJOnLkiPr37+9U3yQXAAAUQjNmzJAktWzZ0qF9zpw56tOnjyQpJSVFHh7/G8Q4e/asnn76aaWlpalUqVK67777tHHjRtWuXdupvpnQCRQhTOgEsroTEzonVzFnQueQlAWm3MfVqFwAAOBiZk3oLCiY0AkAAExF5QIAABcrdPMPboHkAgAAF3PHahF3YlgEAACYisoFAAAuVtQmdJJcAADgYkVtzgXDIgAAwFRULgAAcDFbEatdkFwAAOBizLkAAACmKlp1C+ZcAAAAk1G5AADAxRgWucOGDh2abbvFYpG3t7dq1KihyMhIlS5d+g5HBgCAOYraDp1uTy62b9+ubdu2KTMzU6GhoZKkn3/+WZ6engoLC9P06dM1bNgwrV+/3un3yQMAgDvP7XMuIiMj1aZNGx0/flxbt27V1q1bdfToUbVt21a9evXSsWPH1Lx5cw0ZMsTdoQIAkCc2GaYcBYXFMAy3RluxYkUlJiZmqUrs3r1b7dq107Fjx7Rt2za1a9dOp06dytU9i3lVdEWoQIG3KbCJu0MA8p3GRxNc3sfLIU+Ycp9xhxeach9Xc3vl4vz58zp58mSW9t9++00XLlyQJAUEBOjKlSt3OjQAAJAHbk8uIiMj9dRTT2np0qU6evSojh49qqVLl6pfv37q1q2bJOmHH37QPffc495AAQDII5tJR0Hh9gmd7733noYMGaKePXvq2rVrkqRixYopOjpakydPliSFhYXpww8/dGeYAADkWUGaL2EGtycXJUuW1AcffKDJkyfr4MGDkqRq1aqpZMmS9nMaNmzopugAAICz3J5cXFeyZEn7XhZ/TSwAACjoilbdIh/MubDZbIqNjZW/v7+Cg4MVHBysgIAAjR07VjZbQRphAgAge8y5uMNefvllzZo1S+PHj1dERIQkaf369YqJidHly5c1btw4N0cIAMDtYc7FHTZ37lx9+OGH6tq1q72tfv36qlixop577jmSCwAAChi3JxdnzpxRWFhYlvawsDCdOXPGDREBAGCuolW3yAdzLho0aKBp06ZlaZ82bZoaNGjghogAADAXcy7usDfffFOdO3fWypUrFR4eLklKSkrSr7/+qq+++srN0QEAAGe5vXLRokUL7du3T927d9e5c+d07tw5PfLII9q3b58efPBBd4cHAMBtM0z6X0Hh9sqF9OfLy5i4CQAorArSkIYZ3J5c7N+/X8uWLdPhw4dlsVhUrVo1devWTVWrVnV3aAAAIA/cmlzExcXptddek81mU2BgoAzD0G+//aYRI0bo3//+t1588UV3hgcAgCmK2j4Xbptz8e233+qVV17Ryy+/rFOnTik1NVVpaWn67bffNHLkSI0cOVLr1q1zV3gAAJjGMOkoKNxWuZg5c6b69++vmJgYh/bSpUsrNjZWaWlpmjFjhpo3b+6eAAEAQJ64rXLxww8/6Mknn8zx8yeffFKbNm26gxHBTA/+7X4lLI1XyuGtunblmLp2be/ukAC3K/tkB9VOnKJ79yzUvXsWKmzZePm1auTusHAH2GSYchQUbksuTpw4oZCQkBw/r1q1qtLS0u5cQDBViRJ3aefOn/T84JfdHQqQb1xJPa1jcfP1U6dh+qnTi7q4YZdqzBol73squzs0uJg7NtGKi4tTkyZN5Ovrq8DAQHXr1k379u275XWLFy9WWFiYvL29Va9evTztOeW25OLy5cvy8vLK8fPixYvrypUrdzAimOmb5d/qtdFvatmyb9wdCpBvnF+5WedXb1XGoVRlHDquY29+JNvvl1WyUai7Q4OLuWOfi7Vr12rAgAHatGmTEhMTdfXqVbVr106XLl3K8ZqNGzeqV69e6tevn7Zv365u3bqpW7du+vHHH53q262rRT788EOVLFky288uXrx4h6MBgDvIw0OlujSTh4+30rfudXc0KIS++cbxP+7i4+MVGBiorVu35jifcerUqerQoYOGDx8uSRo7dqwSExM1bdo0zZw5M9d9uy25qFKlij744INbnnMrGRkZysjIcGgzDEMWi+W24gMAV/AJC1bYsvHysHop89Jl/fL0eF3ef9TdYcHFzNpEK7u/eVarVVar9ZbXnj9/XtKfCydykpSUpKFDhzq0tW/fXgkJCU7F6bbk4vDhw6bcJy4uTmPGjHFos3iUlMXTz5T7A4CZLv9yTD+1HyJP3xIq1TlcIZMHad/fXybBKOTM2ro7u795o0ePzrLy8kY2m00vvPCCIiIiVLdu3RzPS0tLU7ly5RzaypUr5/QcSLfv0Hm7Ro0alSXLKnV31le4A0B+YFy9pozDf/4f9e+7flGJBjVVrt/DOjJyhpsjQ0GQ3d+83FQtBgwYoB9//FHr1693VWgOCnxykV05iCERAAWGh0UWr+LujgIuZtawSG6HQP5q4MCB+uKLL7Ru3TpVqlTppucGBQXpxIkTDm0nTpxQUFCQU326/a2oKJxKlLhLDRrUUYMGdSRJVUOqqEGDOqpcuYKbIwPcp+LIf6jk/bXlVSlQPmHBqjjyH/INr6szS9e6OzS4mM0wTDmcYRiGBg4cqKVLl2r16tW5emdXeHi4Vq1a5dCWmJio8PBwp/ou8JUL5E+N72ugVSs/s38/8a0YSdLceZ+qX/8hbooKcK9iZQJUdcoLKh5YSpkXL+mPPUe0v/cYXfhuh7tDQyE0YMAALVy4UMuWLZOvr6993oS/v798fHwkSVFRUapYsaLi4uIkSYMHD1aLFi00ceJEde7cWR9//LG2bNmi999/36m+LYbhZCpUABTzqujuEIB8aVNgE3eHAOQ7jY8muLyPfwQ/Ysp9FhxZkutzc5oiMGfOHPXp00eS1LJlS4WEhCg+Pt7++eLFi/XKK6/o8OHDqlmzpt5880116tTJqTjdXrnw9PRUamqqAgMDHdpPnz6twMBAZWZmuikyAADM4Y6tu3NTO1izZk2Wtscee0yPPfbYbfXt9jkXOT18RkbGTXfwBAAA+ZPbKhdvv/22pD/LNjfu1JmZmal169YpLIwlpQCAgs+sfS4KCrclF5MnT5b0Z+Vi5syZ8vT0tH/m5eWlkJAQp7YaBQAgvzJrKWpB4bbk4tChQ5KkVq1aacmSJSpVqpS7QgEAwKUK0uvSzeD2CZ3ffvut/evr8y/YBAsAgILL7RM6JWnevHmqV6+efHx85OPjo/r162v+/PnuDgsAAFO445Xr7uT2ysWkSZP06quvauDAgYqIiJAkrV+/Xs8884xOnTqlIUPYcAkAULAx5+IOe+eddzRjxgxFRUXZ27p27ao6deooJiaG5AIAgALG7clFamqqmjVrlqW9WbNmSk1NdUNEAACYqxBuhn1Tbp9zUaNGDX366adZ2j/55BPVrFnTDREBAGAumwxTjoLC7ZWLMWPGqEePHlq3bp19zsWGDRu0atWqbJMOAACQv7k9uXj00Uf1/fffa/LkyUpISJAk1apVSz/88IPuvfde9wYHAIAJmNDpBvfdd58WLFjg7jAAAHCJgrSM1Axun3MBAAAKF7dVLjw8PG65E6fFYtG1a9fuUEQAALhGQZqMaQa3JRdLly7N8bOkpCS9/fbbstmK2igVAKAwKmpLUd2WXERGRmZp27dvn0aOHKnPP/9cvXv3VmxsrBsiAwDAXEXtP5XzxZyL48eP6+mnn1a9evV07do1JScna+7cuQoODnZ3aAAAwEluTS7Onz+vESNGqEaNGtq9e7dWrVqlzz//XHXr1nVnWAAAmIoXl90hb775pt544w0FBQVp0aJF2Q6TAABQGBS1CZ0Ww02zTDw8POTj46M2bdrI09Mzx/OWLFni9L2LeVW8ndCAQmtTYBN3hwDkO42PJri8jzaV25tyn5W/LjflPq7mtspFVFTULZeiAgBQGLBa5A6Jj493V9cAANxRRW1YJF+sFgEAAIVHvni3CAAAhVlBWulhBpILAABczFbE5lwwLAIAAExF5QIAABcrWnULkgsAAFyuqK0WIbkAAMDFilpywZwLAABgKioXAAC4GDt0AgAAUzEsAgAAcBuoXAAA4GJFbYdOKhcAALiYYRimHM5at26dHn74YVWoUEEWi0UJCQk3PX/NmjWyWCxZjrS0NKf6JbkAAKCQunTpkho0aKB3333Xqev27dun1NRU+xEYGOjU9QyLAADgYu6a0NmxY0d17NjR6esCAwMVEBCQ536pXAAA4GLuGhbJq4YNG6p8+fJq27atNmzY4PT1VC4AACggMjIylJGR4dBmtVpltVpNuX/58uU1c+ZMNW7cWBkZGfrwww/VsmVLff/992rUqFGu70PlAgAAF7PJMOWIi4uTv7+/wxEXF2danKGhofrXv/6l++67T82aNdPs2bPVrFkzTZ482an7ULkAAMDFzFqKOmrUKA0dOtShzayqRU6aNm2q9evXO3UNyQUAAC5mM2m+hJlDILmVnJys8uXLO3UNyQUAAIVUenq6Dhw4YP/+0KFDSk5OVunSpVWlShWNGjVKx44d07x58yRJU6ZMUdWqVVWnTh1dvnxZH374oVavXq0VK1Y41S/JBQAALuauHTq3bNmiVq1a2b+/PqQSHR2t+Ph4paamKiUlxf75lStXNGzYMB07dkx33XWX6tevr5UrVzrcIzcsRiF8VVsxr4ruDgHIlzYFNnF3CEC+0/hogsv7qBXY1JT77Dn5gyn3cTVWiwAAAFMxLAIAgIsVtReXkVwAAOBiZq0WKSgYFgEAAKaicgEAgIsxLAIAAEzFsAgAAMBtoHIBAICLMSwCAABMZRg2d4dwR5FcAADgYrYiVrlgzgUAADAVlQsAAFysEL7G66ZILgAAcDGGRQAAAG4DlQsAAFyMYREAAGAqdugEAAC4DVQuAABwMXboBAAApipqcy4YFgEAAKaicgEAgIsVtX0uSC4AAHCxojYsQnIBAICLsRQVAADgNlC5AADAxRgWAQAApipqEzoZFgEAAKaicgEAgIsxLAIAAEzFahEAAIDbQOUCAAAX48VlAADAVAyLAAAA3AYqFwAAuBirRQAAgKmK2pwLhkUAAHAxwzBMOZy1bt06Pfzww6pQoYIsFosSEhJuec2aNWvUqFEjWa1W1ahRQ/Hx8U73S3IBAEAhdenSJTVo0EDvvvturs4/dOiQOnfurFatWik5OVkvvPCC+vfvr+XLlzvVL8MiAAC4mLvmXHTs2FEdO3bM9fkzZ85U1apVNXHiRElSrVq1tH79ek2ePFnt27fP9X2oXAAA4GKGSYerJSUlqU2bNg5t7du3V1JSklP3oXIBAEABkZGRoYyMDIc2q9Uqq9Vqyv3T0tJUrlw5h7Zy5crpwoUL+uOPP+Tj45Or+xTK5OLalWPuDgH685cgLi5Oo0aNMu0fPlAY8LtR9Jj1dykmJkZjxoxxaBs9erRiYmJMub9ZGBaBy2RkZGjMmDFZsmygqON3A3k1atQonT9/3uEYNWqUafcPCgrSiRMnHNpOnDghPz+/XFctpEJauQAAoDAycwgkO+Hh4frqq68c2hITExUeHu7UfahcAABQSKWnpys5OVnJycmS/lxqmpycrJSUFEl/VkKioqLs5z/zzDM6ePCg/u///k979+7V9OnT9emnn2rIkCFO9UtyAQBAIbVlyxbde++9uvfeeyVJQ4cO1b333qvXXntNkpSammpPNCSpatWq+vLLL5WYmKgGDRpo4sSJ+vDDD51ahipJFqOobXiOO4ZJa0D2+N1AYUdyAQAATMWwCAAAMBXJBQAAMBXJBQAAMBXJBQAAMBXJBZzSp08fWSwWWSwWFS9eXOXKlVPbtm01e/Zs2Ww2d4cH5Nn1f9vjx493aE9ISJDFYnHqXiEhIZoyZUquzrv+++Tj46OQkBA9/vjjWr16tVP9AfkNyQWc1qFDB6Wmpurw4cP6+uuv1apVKw0ePFhdunTRtWvX3B0ekGfe3t564403dPbs2TvWZ2xsrFJTU7Vv3z7NmzdPAQEBatOmjcaNG3fHYgDMRnIBp1mtVgUFBalixYpq1KiRXnrpJS1btkxff/214uPjJUkpKSmKjIxUyZIl5efnp8cffzzLfvWvv/66AgMD5evrq/79+2vkyJFq2LDhnX8g4P9r06aNgoKCFBcXd9Pz/vOf/6hOnTqyWq0KCQnRxIkT7Z+1bNlSR44c0ZAhQ+xViZvx9fVVUFCQqlSpoubNm+v999/Xq6++qtdee0379u2zn7d27Vo1bdpUVqtV5cuX18iRIx2S+YsXL6p3794qUaKEypcvr8mTJ6tly5Z64YUX8vbDAG4DyQVM8dBDD6lBgwZasmSJbDabIiMjdebMGa1du1aJiYk6ePCgevToYT//o48+0rhx4/TGG29o69atqlKlimbMmOHGJwAkT09P/fvf/9Y777yjo0ePZnvO1q1b9fjjj6tnz57atWuXYmJi9Oqrr9oT6yVLlqhSpUr2ikRqaqrTcQwePFiGYWjZsmWSpGPHjqlTp05q0qSJduzYoRkzZmjWrFl6/fXX7dcMHTpUGzZs0H//+18lJibqu+++07Zt25z/IQAm4MVlME1YWJh27typVatWadeuXTp06JAqV64sSZo3b57q1KmjzZs3q0mTJnrnnXfUr18/9e3bV5L02muvacWKFUpPT3fnIwDq3r27GjZsqNGjR2vWrFlZPp80aZJat26tV199VZJ0zz336KefftKECRPUp08flS5dWp6envaKRF6ULl1agYGBOnz4sCRp+vTpqly5sqZNmyaLxaKwsDAdP35cI0aM0GuvvaZLly5p7ty5WrhwoVq3bi1JmjNnjipUqJC3HwJwm6hcwDSGYchisWjPnj2qXLmyPbGQpNq1aysgIEB79uyRJO3bt09NmzZ1uP7G7wF3eeONNzR37lz7v9e/2rNnjyIiIhzaIiIitH//fmVmZpoWw/Xfp+t9hoeHOwyxREREKD09XUePHtXBgwd19epVh98hf39/hYaGmhYP4AySC5hmz549qlq1qrvDAG5b8+bN1b59e40aNcot/Z8+fVq//fYbv08osEguYIrVq1dr165devTRR1WrVi39+uuv+vXXX+2f//TTTzp37pxq164tSQoNDdXmzZsd7nHj94A7jR8/Xp9//rmSkpIc2mvVqqUNGzY4tG3YsEH33HOPPD09JUleXl63VcWYOnWqPDw81K1bN3ufSUlJ+uuroDZs2CBfX19VqlRJ1apVU/HixR1+h86fP6+ff/45zzEAt4M5F3BaRkaG0tLSlJmZqRMnTuibb75RXFycunTpoqioKHl4eKhevXrq3bu3pkyZomvXrum5555TixYt1LhxY0nS888/r6efflqNGzdWs2bN9Mknn2jnzp2qVq2am58O+NP1f8Nvv/22Q/uwYcPUpEkTjR07Vj169FBSUpKmTZum6dOn288JCQnRunXr1LNnT1mtVpUpUybHfi5evKi0tDRdvXpVhw4d0oIFC/Thhx8qLi5ONWrUkCQ999xzmjJlip5//nkNHDhQ+/bt0+jRozV06FB5eHjI19dX0dHRGj58uH2+xujRo+Xh4eH0Hh2AKQzACdHR0YYkQ5JRrFgxo2zZskabNm2M2bNnG5mZmfbzjhw5YnTt2tUoUaKE4evrazz22GNGWlqaw71iY2ONMmXKGCVLljSeeuopY9CgQcYDDzxwpx8JMAzjz3/bkZGRDm2HDh0yvLy8jBv/r/Kzzz4zateubRQvXtyoUqWKMWHCBIfPk5KSjPr16xtWqzXLtX8VHBxs/33y8vIyqlSpYjz++OPG6tWrs5y7Zs0ao0mTJoaXl5cRFBRkjBgxwrh69ar98wsXLhhPPPGEcddddxlBQUHGpEmTjKZNmxojR47Mw08DuD28ch35Rtu2bRUUFKT58+e7OxSgwLt06ZIqVqyoiRMnql+/fu4OB0UMwyJwi99//10zZ85U+/bt5enpqUWLFmnlypVKTEx0d2hAgbR9+3bt3btXTZs21fnz5xUbGytJioyMdHNkKIpILuAWFotFX331lcaNG6fLly8rNDRU//nPf9SmTRt3hwYUWG+99Zb27dsnLy8v3Xffffruu+9uOt8DcBWGRQAAgKlYigoAAExFcgEAAExFcgEAAExFcgEAAExFcgHAaRaLRQkJCe4OA0A+RXIB5HNJSUny9PRU586dnbouJCREU6ZMcU1QAHATJBdAPjdr1iw9//zzWrdunY4fP+7ucADglkgugHwsPT1dn3zyiZ599ll17txZ8fHxDp9//vnnatKkiby9vVWmTBl1795dktSyZUsdOXJEQ4YMkcVisb+8KiYmRg0bNnS4x5QpUxQSEmL/fvPmzWrbtq3KlCkjf39/tWjRQtu2bXPlYwIoZEgugHzs008/VVhYmEJDQ/WPf/xDs2fPtr92+8svv1T37t3VqVMnbd++XatWrVLTpk0lSUuWLFGlSpUUGxur1NRUpaam5rrPixcvKjo6WuvXr9emTZtUs2ZNderUSRcvXnTJMwIofNj+G8jHZs2apX/84x+SpA4dOuj8+fNau3atWrZsqXHjxqlnz54aM2aM/fwGDRpIkkqXLi1PT0/5+voqKCjIqT4feughh+/ff/99BQQEaO3aterSpcttPhGAooDKBZBP7du3Tz/88IN69eolSSpWrJh69OihWbNmSZKSk5PVunVr0/s9ceKEnn76adWsWVP+/v7y8/NTenq6UlJSTO8LQOFE5QLIp2bNmqVr166pQoUK9jbDMGS1WjVt2jT5+Pg4fU8PDw/d+Dqhq1evOnwfHR2t06dPa+rUqQoODpbValV4eLiuXLmStwcBUORQuQDyoWvXrmnevHmaOHGikpOT7ceOHTtUoUIFLVq0SPXr19eqVatyvIeXl5cyMzMd2sqWLau0tDSHBCM5OdnhnA0bNmjQoEHq1KmT6tSpI6vVqlOnTpn6fAAKNyoXQD70xRdf6OzZs+rXr5/8/f0dPnv00Uc1a9YsTZgwQa1bt1b16tXVs2dPXbt2TV999ZVGjBgh6c99LtatW6eePXvKarWqTJkyatmypX777Te9+eab+vvf/65vvvlGX3/9tfz8/Oz3r1mzpubPn6/GjRvrwoULGj58eJ6qJACKLioXQD40a9YstWnTJktiIf2ZXGzZskWlS5fW4sWL9d///lcNGzbUQw89pB9++MF+XmxsrA4fPqzq1aurbNmykqRatWpp+vTpevfdd9WgQQP98MMPevHFF7P0ffbsWTVq1EhPPvmkBg0apMDAQNc+MIBCxWLcOAALAABwG6hcAAAAU5FcAAAAU5FcAAAAU5FcAAAAU5FcAAAAU5FcAAAAU5FcAAAAU5FcAAAAU5FcAAAAU5FcAAAAU5FcAAAAU5FcAAAAU/0/Oo1Z+u8c4GkAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "actual = np.array(\n",
    "    ['Dog','Dog','Dog','Not Dog','Dog','Not Dog','Dog','Dog','Not Dog','Not Dog'])\n",
    "predicted = np.array(\n",
    "    ['Dog','Not Dog','Dog','Not Dog','Dog','Dog','Dog','Dog','Not Dog','Not Dog'])\n",
    "\n",
    "# confusion_matrix\n",
    "conf_matrix = confusion_matrix(actual, predicted)\n",
    "\n",
    "sns.heatmap(conf_matrix, annot=True, fmt='g', xticklabels=['Dog', 'Not Dog'], yticklabels=['Dog', 'Not Dog'])\n",
    "plt.ylabel('Prediction')\n",
    "plt.xlabel('Actual')\n",
    "plt.title('Confusion Matrix')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the confusion matrix we can find:\n",
    "\n",
    "- ***Accuracy:*** It uses the ratio of total correct instances to the total instances\n",
    "\n",
    "![accuracy confusion_matrix](acc1.png)\n",
    "\n",
    "acc = (5+3)/(5+3+1+1) = 8/10 = 0.8\n",
    "\n",
    "- ***Precision:*** It's a measure of how accurate a model's positive predictions are\n",
    ">It's defined as the ratio of true positive predictions to the total number of positive predictions made by the model\n",
    "\n",
    "![precision](precision.png)\n",
    "\n",
    "precison = 5 / (5 + 1) = 5/6 = 0.8333\n",
    "\n",
    "- ***Recall:*** It measures the effectiveness of a classification model in identifying all relevant instances from a dataset\n",
    ">It's the ration of the number of true positives(TP) to the sum of true positives(TP) and false positives(FP)\n",
    "\n",
    "![recall](recall.png)\n",
    "\n",
    "recall = 5 / (5 + 1) = 5/6 = 0.8333\n",
    "\n",
    "- ***F1-Score:*** It's used to evaluate the overall performance of a classification mode;l\n",
    ">It's the harmonic mean of precision and recall\n",
    "\n",
    "![f1-score](f1_score.png)\n",
    "\n",
    "F1-Score = (2 * 0.8333 * 0.8333) / (0.8333 + 0.8333) = 0.8333\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhIAAAHHCAYAAADqJrG+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAABHeElEQVR4nO3dd3hUZdrH8d8kJJOQSgKkIJBQJKAgCpaA0gwgRWkiIEIoimIEIRTFV5SiRpGOoKJIcQEVRVasINUSihAQQemQFZKgSAKhpJ73D5ZZhwRNhpnMMH4/e53ryjznzHnukxW5vZ9yTIZhGAIAALCBh7MDAAAA1y4SCQAAYDMSCQAAYDMSCQAAYDMSCQAAYDMSCQAAYDMSCQAAYDMSCQAAYDMSCQAAYDMSCcCB9u/frzZt2igoKEgmk0krVqyw6/2PHDkik8mkBQsW2PW+17IWLVqoRYsWzg4D+McgkYDbO3jwoB599FHVqFFDPj4+CgwMVNOmTTVjxgydP3/eoX3Hx8dr165devHFF/Xuu++qcePGDu2vLPXr108mk0mBgYHF/h73798vk8kkk8mkyZMnl/r+x48f17hx47Rjxw47RAvAUco5OwDAkT777DN1795dZrNZffv21Y033qjc3Fx9++23GjVqlHbv3q25c+c6pO/z588rOTlZ//d//6cnnnjCIX1Ur15d58+fl5eXl0Pu/3fKlSunc+fOaeXKlXrggQeszi1evFg+Pj66cOGCTfc+fvy4xo8fr6ioKDVs2LDE31u1apVN/QGwDYkE3Nbhw4fVs2dPVa9eXWvXrlVERITlXEJCgg4cOKDPPvvMYf3/9ttvkqTg4GCH9WEymeTj4+Ow+/8ds9mspk2baunSpUUSiSVLlqhDhw766KOPyiSWc+fOqXz58vL29i6T/gBcxNAG3NakSZOUnZ2tefPmWSURl9SqVUtPPvmk5XN+fr4mTpyomjVrymw2KyoqSs8884xycnKsvhcVFaWOHTvq22+/1W233SYfHx/VqFFDixYtslwzbtw4Va9eXZI0atQomUwmRUVFSbo4JHDp5z8bN26cTCaTVdvq1at15513Kjg4WP7+/qpTp46eeeYZy/krzZFYu3at7rrrLvn5+Sk4OFidOnXSzz//XGx/Bw4cUL9+/RQcHKygoCD1799f586du/Iv9jIPPvigvvjiC2VmZlratm7dqv379+vBBx8scv0ff/yhkSNHqn79+vL391dgYKDatWunnTt3Wq5Zv369br31VklS//79LUMkl56zRYsWuvHGG7Vt2zY1a9ZM5cuXt/xeLp8jER8fLx8fnyLP37ZtW1WoUEHHjx8v8bMCKIpEAm5r5cqVqlGjhpo0aVKi6x9++GE999xzuuWWWzRt2jQ1b95cSUlJ6tmzZ5FrDxw4oPvvv1+tW7fWlClTVKFCBfXr10+7d++WJHXt2lXTpk2TJPXq1Uvvvvuupk+fXqr4d+/erY4dOyonJ0cTJkzQlClTdN999+m77777y+99/fXXatu2rU6cOKFx48YpMTFR33//vZo2baojR44Uuf6BBx7QmTNnlJSUpAceeEALFizQ+PHjSxxn165dZTKZtHz5ckvbkiVLFBMTo1tuuaXI9YcOHdKKFSvUsWNHTZ06VaNGjdKuXbvUvHlzy1/qdevW1YQJEyRJgwYN0rvvvqt3331XzZo1s9zn5MmTateunRo2bKjp06erZcuWxcY3Y8YMVapUSfHx8SooKJAkvfnmm1q1apVmzZqlyMjIEj8rgGIYgBvKysoyJBmdOnUq0fU7duwwJBkPP/ywVfvIkSMNScbatWstbdWrVzckGRs3brS0nThxwjCbzcaIESMsbYcPHzYkGa+++qrVPePj443q1asXieH55583/vxHctq0aYYk47fffrti3Jf6mD9/vqWtYcOGRuXKlY2TJ09a2nbu3Gl4eHgYffv2LdLfgAEDrO7ZpUsXIzQ09Ip9/vk5/Pz8DMMwjPvvv9+4++67DcMwjIKCAiM8PNwYP358sb+DCxcuGAUFBUWew2w2GxMmTLC0bd26tcizXdK8eXNDkvHGG28Ue6558+ZWbV999ZUhyXjhhReMQ4cOGf7+/kbnzp3/9hkB/D0qEnBLp0+fliQFBASU6PrPP/9ckpSYmGjVPmLECEkqMpeiXr16uuuuuyyfK1WqpDp16ujQoUM2x3y5S3Mr/v3vf6uwsLBE30lLS9OOHTvUr18/hYSEWNobNGig1q1bW57zzx577DGrz3fddZdOnjxp+R2WxIMPPqj169crPT1da9euVXp6erHDGtLFeRUeHhf/1VNQUKCTJ09ahm22b99e4j7NZrP69+9fomvbtGmjRx99VBMmTFDXrl3l4+OjN998s8R9AbgyEgm4pcDAQEnSmTNnSnT90aNH5eHhoVq1alm1h4eHKzg4WEePHrVqr1atWpF7VKhQQadOnbIx4qJ69Oihpk2b6uGHH1ZYWJh69uypDz744C+Tiktx1qlTp8i5unXr6vfff9fZs2et2i9/lgoVKkhSqZ6lffv2CggI0Pvvv6/Fixfr1ltvLfK7vKSwsFDTpk1T7dq1ZTabVbFiRVWqVEk//vijsrKyStxnlSpVSjWxcvLkyQoJCdGOHTs0c+ZMVa5cucTfBXBlJBJwS4GBgYqMjNRPP/1Uqu9dPtnxSjw9PYttNwzD5j4ujd9f4uvrq40bN+rrr79Wnz599OOPP6pHjx5q3bp1kWuvxtU8yyVms1ldu3bVwoUL9fHHH1+xGiFJL730khITE9WsWTP961//0ldffaXVq1frhhtuKHHlRbr4+ymNlJQUnThxQpK0a9euUn0XwJWRSMBtdezYUQcPHlRycvLfXlu9enUVFhZq//79Vu0ZGRnKzMy0rMCwhwoVKlitcLjk8qqHJHl4eOjuu+/W1KlTtWfPHr344otau3at1q1bV+y9L8W5d+/eIud++eUXVaxYUX5+flf3AFfw4IMPKiUlRWfOnCl2guolH374oVq2bKl58+apZ8+eatOmjeLi4or8Tkqa1JXE2bNn1b9/f9WrV0+DBg3SpEmTtHXrVrvdH/gnI5GA2xo9erT8/Pz08MMPKyMjo8j5gwcPasaMGZIuluYlFVlZMXXqVElShw4d7BZXzZo1lZWVpR9//NHSlpaWpo8//tjquj/++KPIdy9tzHT5ktRLIiIi1LBhQy1cuNDqL+affvpJq1atsjynI7Rs2VITJ07Ua6+9pvDw8Cte5+npWaTasWzZMh07dsyq7VLCU1zSVVpPPfWUUlNTtXDhQk2dOlVRUVGKj4+/4u8RQMmxIRXcVs2aNbVkyRL16NFDdevWtdrZ8vvvv9eyZcvUr18/SdJNN92k+Ph4zZ07V5mZmWrevLm2bNmihQsXqnPnzldcWmiLnj176qmnnlKXLl00dOhQnTt3Tq+//rquv/56q8mGEyZM0MaNG9WhQwdVr15dJ06c0Jw5c3TdddfpzjvvvOL9X331VbVr106xsbEaOHCgzp8/r1mzZikoKEjjxo2z23NczsPDQ88+++zfXtexY0dNmDBB/fv3V5MmTbRr1y4tXrxYNWrUsLquZs2aCg4O1htvvKGAgAD5+fnp9ttvV3R0dKniWrt2rebMmaPnn3/eshx1/vz5atGihcaOHatJkyaV6n4ALuPkVSOAw+3bt8945JFHjKioKMPb29sICAgwmjZtasyaNcu4cOGC5bq8vDxj/PjxRnR0tOHl5WVUrVrVGDNmjNU1hnFx+WeHDh2K9HP5ssMrLf80DMNYtWqVceONNxre3t5GnTp1jH/9619Fln+uWbPG6NSpkxEZGWl4e3sbkZGRRq9evYx9+/YV6ePyJZJff/210bRpU8PX19cIDAw07r33XmPPnj1W11zq7/LlpfPnzzckGYcPH77i79QwrJd/XsmVln+OGDHCiIiIMHx9fY2mTZsaycnJxS7b/Pe//23Uq1fPKFeunNVzNm/e3LjhhhuK7fPP9zl9+rRRvXp145ZbbjHy8vKsrhs+fLjh4eFhJCcn/+UzAPhrJsMoxYwqAACAP2GOBAAAsBmJBAAAsBmJBAAAsBmJBAAAsBmJBAAAsBmJBAAAsBmJBAAAsJlb7myZ0bK5s0MAXFKV7w44OwTA5eTnHvv7i65S3u+H7HIfr4o1/v6iMkZFAgAA2MwtKxIAALiUwgJnR+AwJBIAADiaUejsCByGRAIAAEcrdN9EgjkSAAC4oYKCAo0dO1bR0dHy9fVVzZo1NXHiRP35XZ2GYei5555TRESEfH19FRcXp/3795eqHxIJAAAczDAK7XKUxiuvvKLXX39dr732mn7++We98sormjRpkmbNmmW5ZtKkSZo5c6beeOMNbd68WX5+fmrbtq0uXLhQ4n4Y2gAAwNGcMLTx/fffq1OnTurQoYMkKSoqSkuXLtWWLVskXaxGTJ8+Xc8++6w6deokSVq0aJHCwsK0YsUK9ezZs0T9UJEAAMANNWnSRGvWrNG+ffskSTt37tS3336rdu3aSZIOHz6s9PR0xcXFWb4TFBSk22+/XcnJySXuh4oEAACOZqdVGzk5OcrJybFqM5vNMpvNRa59+umndfr0acXExMjT01MFBQV68cUX1bt3b0lSenq6JCksLMzqe2FhYZZzJUFFAgAARysssMuRlJSkoKAgqyMpKanYLj/44AMtXrxYS5Ys0fbt27Vw4UJNnjxZCxcutOujUZEAAOAaMWbMGCUmJlq1FVeNkKRRo0bp6aeftsx1qF+/vo4ePaqkpCTFx8crPDxckpSRkaGIiAjL9zIyMtSwYcMSx0RFAgAARzMK7XKYzWYFBgZaHVdKJM6dOycPD+u/5j09PVX434mf0dHRCg8P15o1ayznT58+rc2bNys2NrbEj0ZFAgAAR3PCqo17771XL774oqpVq6YbbrhBKSkpmjp1qgYMGCBJMplMGjZsmF544QXVrl1b0dHRGjt2rCIjI9W5c+cS90MiAQCAG5o1a5bGjh2rxx9/XCdOnFBkZKQeffRRPffcc5ZrRo8erbNnz2rQoEHKzMzUnXfeqS+//FI+Pj4l7sdk/HmLKzfBa8SB4vEacaCosniNeM7BTXa5j7nmHXa5jz1RkQAAwNHc+F0bJBIAADiaG7/9k1UbAADAZlQkAABwtMICZ0fgMCQSAAA4GkMbAAAARVGRAADA0Vi1AQAAbMbQBgAAQFFUJAAAcDSGNgAAgK0Mw32XfzK0AQAAbEZFAgAAR3PjyZYkEgAAOBpzJAAAgM3cuCLBHAkAAGAzKhIAADgaL+0CAAA2Y2gDAACgKCoSAAA4Gqs2AACAzRjaAAAAKIqKBAAAjsbQBgAAsJkbJxIMbQAAAJtRkQAAwMHc+TXiJBIAADiaGw9tkEgAAOBoLP8EAAAoiooEAACOxtAGAACwGUMbAAAARVGRAADA0RjaAAAANmNoAwAAoCgqEgAAOBpDGwAAwGZunEgwtAEAgBuKioqSyWQqciQkJEiSLly4oISEBIWGhsrf31/dunVTRkZGqfshkQAAwNGMQvscpbB161alpaVZjtWrV0uSunfvLkkaPny4Vq5cqWXLlmnDhg06fvy4unbtWupHY2gDAABHc8LQRqVKlaw+v/zyy6pZs6aaN2+urKwszZs3T0uWLFGrVq0kSfPnz1fdunW1adMm3XHHHSXuh4oEAACOZqeKRE5Ojk6fPm115OTk/G33ubm5+te//qUBAwbIZDJp27ZtysvLU1xcnOWamJgYVatWTcnJyaV6NBIJAACuEUlJSQoKCrI6kpKS/vZ7K1asUGZmpvr16ydJSk9Pl7e3t4KDg62uCwsLU3p6eqliYmgDAABHs9PQxpgxY5SYmGjVZjab//Z78+bNU7t27RQZGWmXOP6MRAIAAEez086WZrO5RInDnx09elRff/21li9fbmkLDw9Xbm6uMjMzraoSGRkZCg8PL9X9GdoAAMCNzZ8/X5UrV1aHDh0sbY0aNZKXl5fWrFljadu7d69SU1MVGxtbqvu7REXC09NTaWlpqly5slX7yZMnVblyZRUUFDgpMgAA7MBJG1IVFhZq/vz5io+PV7ly//srPygoSAMHDlRiYqJCQkIUGBioIUOGKDY2tlQrNiQXSSQMwyi2PScnR97e3mUcDQAAduakROLrr79WamqqBgwYUOTctGnT5OHhoW7duiknJ0dt27bVnDlzSt2HUxOJmTNnSpJMJpPefvtt+fv7W84VFBRo48aNiomJcVZ4AABc09q0aXPF/1j38fHR7NmzNXv27Kvqw6mJxLRp0yRdrEi88cYb8vT0tJzz9vZWVFSU3njjDWeFBwCAfVzhL3N34NRE4vDhw5Kkli1bavny5apQoYIzwwEAwDHc+KVdLjFHYt26dc4OAQAA2MAlEomCggItWLBAa9as0YkTJ1R4Wea2du1aJ0UGAIAdUJFwrCeffFILFixQhw4ddOONN8pkMjk7JAAA7MdOG1K5IpdIJN577z198MEHat++vbNDAQDA/ty4IuESO1t6e3urVq1azg4DAACUkkskEiNGjNCMGTOuuNYVAIBrmmHY53BBLjG08e2332rdunX64osvdMMNN8jLy8vq/J9fNAIAwDXHjYc2XCKRCA4OVpcuXZwdBgAAKCWXSCTmz5/v7BAAAHAcKhIAAMBmLP90vA8//FAffPCBUlNTlZuba3Vu+/btTooKAAD8FZdYtTFz5kz1799fYWFhSklJ0W233abQ0FAdOnRI7dq1c3Z4AABcFaPQsMvhilwikZgzZ47mzp2rWbNmydvbW6NHj9bq1as1dOhQZWVlOTs8AACuTmGhfQ4X5BKJRGpqqpo0aSJJ8vX11ZkzZyRJffr00dKlS50ZGgAA+AsukUiEh4frjz/+kCRVq1ZNmzZtknTxNeNsUgUAuOYZhfY5XJBLJBKtWrXSJ598Iknq37+/hg8frtatW6tHjx7sLwEAuPYVGvY5XJBLrNqYO3eu5dXhCQkJCg0N1ffff6/77rtPjz76qJOjAwDgKrno/AZ7cIlEwsPDQx4e/yuO9OzZUz179nRiRAAAoCRcIpGQpMzMTG3ZskUnTpywVCcu6du3r5OiAgDADqhIONbKlSvVu3dvZWdnKzAwUCaTyXLOZDKRSAAArm1uvHDAJSZbjhgxQgMGDFB2drYyMzN16tQpy3FpNQcAAHA9LpFIHDt2TEOHDlX58uWdHQps4BffT2HrNlgdoQsXWc77drxXFaZNV6VPP1fYug0y+fk7MVrANYwelaD83GOaMnm8s0NBWXDjDalcYmijbdu2+uGHH1SjRg1nhwIb5R8+pFMjRlg+GwUFlp9NZrNytmxRzpYtChjEKhygcaOb9MjDD2nnj3ucHQrKiosu3bQHl0gkOnTooFGjRmnPnj2qX7++vLy8rM7fd999TooMJWUUFKjwVPHDUOc++lCS5HVTwzKMCHBNfn7ltWjRa3ps8Gg9M2aos8MBrppLJBKPPPKIJGnChAlFzplMJhX86b9u4ZrKVblOFZd9JCM3V3l7div7rbkqPHHC2WEBLmfWzJf0xedrtGbtNyQS/yQuuiulPbhEInH5ck9cW/J+/llZr7ysgv+kyiM0VH59+ylkxiydHNBPxvnzzg4PcBkPPHCfbr75Rt0R28HZoaCsMbThunJycpSTk2PdVlgos4dLzCP9R8jdsvl/Hw4dUt6en1XxvfdlbtlSFz7/3HmBAS7kuusiNW3KBN3TvleRf2cB1zKXSCRmzpxZbLvJZJKPj49q1aqlZs2aydPTs8g1SUlJGj/eetbziOrVNCo6yhGhogSMs9kq+PVXlYus4uxQAJdxyy31FRZWSVs3f2lpK1eunO666w4lPN5P5f2jqc66McON/791iURi2rRp+u2333Tu3DlVqFBBknTq1CmVL19e/v7+OnHihGrUqKF169apatWqVt8dM2aMEhMTrdoy76Vs6EwmH195RkaqYDV7gACXrF37rW66uZVV29tvTdXevQf16uTZJBHuzo2HNlyi/v/SSy/p1ltv1f79+3Xy5EmdPHlS+/bt0+23364ZM2YoNTVV4eHhGj58eJHvms1mBQYGWh0Ma5Qt/8cGy+umm+QRFi6vG25Q0MQXpMJCXVjztSTJo0KIytWspXJVLlYoytWooXI1a8kUEODMsIEylZ19Vrt377U6zp09p5MnT2n37r3ODg+O5savEXeJisSzzz6rjz76SDVr1rS01apVS5MnT1a3bt106NAhTZo0Sd26dXNilLgSz0qVFPTsc/IIDFRhVqbydu3SHwmDZWRlSZJ877tP/v36W64PmTlLkpT1cpIufPVlsfcEAFwbXCKRSEtLU35+fpH2/Px8paenS5IiIyN15syZsg4NJZA1seiy3T87u3CBzi5cUDbBANeQu1t3d3YIKCsMbThWy5Yt9eijjyolJcXSlpKSosGDB6tVq4tjirt27VJ0dLSzQgQAwHZuvEW2SyQS8+bNU0hIiBo1aiSz2Syz2azGjRsrJCRE8+bNkyT5+/trypQpTo4UAAD8mUskEuHh4Vq9erX27NmjZcuWadmyZdqzZ49WrVqlsLAwSRerFm3atHFypAAA2KDQsM9RSseOHdNDDz2k0NBQ+fr6qn79+vrhhx8s5w3D0HPPPaeIiAj5+voqLi5O+/fvL1UfLjFH4pKYmBjFxMQ4OwwAAOzLCSsuTp06paZNm6ply5b64osvVKlSJe3fv9+yzYIkTZo0STNnztTChQsVHR2tsWPHqm3bttqzZ498fHxK1I/TEonExERNnDhRfn5+RfaBuNzUqVPLKCoAANzDK6+8oqpVq2r+/PmWtj/PNTQMQ9OnT9ezzz6rTp06SZIWLVqksLAwrVixQj179ixRP05LJFJSUpSXl2f5+UpMJlNZhQQAgGPYadVGca+FuDS38HKffPKJ2rZtq+7du2vDhg2qUqWKHn/8ccuLMg8fPqz09HTFxcVZvhMUFKTbb79dycnJrp9IrFu3rtifAQBwN/baIru410I8//zzGjduXJFrDx06pNdff12JiYl65plntHXrVg0dOlTe3t6Kj4+3bK9waS7iJWFhYZZzJeFScyQAAMCVFfdaiOKqEdLFN2s3btxYL730kiTp5ptv1k8//aQ33nhD8fHxdovJaYlE165dS3zt8uXLHRgJAAAOZqehjSsNYxQnIiJC9erVs2qrW7euPvroI0kXV0xKUkZGhiIiIizXZGRkqGHDhiWOyWmJRFBQkLO6BgCgbDlhZ8umTZtq717r97js27dP1atXl3Rx4mV4eLjWrFljSRxOnz6tzZs3a/DgwSXux2mJxJ9nkQIA4NacsPxz+PDhatKkiV566SU98MAD2rJli+bOnau5c+dKuriYYdiwYXrhhRdUu3Zty/LPyMhIde7cucT9MEcCAAA3dOutt+rjjz/WmDFjNGHCBEVHR2v69Onq3bu35ZrRo0fr7NmzGjRokDIzM3XnnXfqyy+/LPEeEpJkMgzDJd4k8uGHH+qDDz5QamqqcnNzrc5t3769VPfKaNncnqEBbqPKdwecHQLgcvJzjzm8j+zE++xyH/+pn9jlPvbkEltkz5w5U/3791dYWJhSUlJ02223KTQ0VIcOHVK7du2cHR4AAFfFKDTscrgil0gk5syZo7lz52rWrFny9vbW6NGjtXr1ag0dOlRZWVnODg8AAFyBSyQSqampatKkiSTJ19dXZ86ckST16dNHS5cudWZoAABcPSe9tKssuEQiER4erj/++EOSVK1aNW3atEnSxe07XWQKBwAAtisstM/hglwikWjVqpU++eTiBJL+/ftr+PDhat26tXr06KEuXbo4OToAAHAlLrH8c+7cuSr8b6aVkJCgihUr6rvvvtN9992nxx57zMnRAQBwlVx0WMIeXCKR8PDwUG5urrZv364TJ07I19fX8jayL7/8Uvfee6+TIwQA4CqQSDjWl19+qT59+ujkyZNFzplMJhUUFDghKgAA8HdcYo7EkCFD9MADDygtLU2FhYVWB0kEAOBaZxiGXQ5X5BIViYyMDCUmJhZ5JzoAAG7BjYc2XKIicf/992v9+vXODgMAAMdw430kXKIi8dprr6l79+765ptvVL9+fXl5eVmdHzp0qJMiAwAAf8UlEomlS5dq1apV8vHx0fr162UymSznTCYTiQQA4Jrmqu/JsAeXSCT+7//+T+PHj9fTTz8tDw+XGG0BAMB+3DiRcIm/tXNzc9WjRw+SCAAArjEu8Td3fHy83n//fWeHAQCAYxTa6XBBLjG0UVBQoEmTJumrr75SgwYNiky2nDp1qpMiAwDg6jFHwsF27dqlm2++WZL0008/WZ3788RLAADgWlwikVi3bp2zQwAAwHGoSAAAAJu56PwGe3CJyZYAAODaREUCAAAHY7IlAACwnRsPbZBIAADgYO5ckWCOBAAAsBkVCQAAHI2hDQAAYCvDjRMJhjYAAIDNqEgAAOBoblyRIJEAAMDBGNoAAAAoBhUJAAAczY0rEiQSAAA4mDsPbZBIAADgYO6cSDBHAgAA2IyKBAAADkZFAgAA2M4w2ecohXHjxslkMlkdMTExlvMXLlxQQkKCQkND5e/vr27duikjI6PUj0YiAQCAm7rhhhuUlpZmOb799lvLueHDh2vlypVatmyZNmzYoOPHj6tr166l7oOhDQAAHMxZQxvlypVTeHh4kfasrCzNmzdPS5YsUatWrSRJ8+fPV926dbVp0ybdcccdJe6DigQAAA5mFJrscpTW/v37FRkZqRo1aqh3795KTU2VJG3btk15eXmKi4uzXBsTE6Nq1aopOTm5VH1QkQAA4BqRk5OjnJwcqzaz2Syz2Vzk2ttvv10LFixQnTp1lJaWpvHjx+uuu+7STz/9pPT0dHl7eys4ONjqO2FhYUpPTy9VTFQkAABwMKPQPkdSUpKCgoKsjqSkpGL7bNeunbp3764GDRqobdu2+vzzz5WZmakPPvjArs9GRQIAAAczSrni4krGjBmjxMREq7biqhHFCQ4O1vXXX68DBw6odevWys3NVWZmplVVIiMjo9g5FX+FigQAANcIs9mswMBAq6OkiUR2drYOHjyoiIgINWrUSF5eXlqzZo3l/N69e5WamqrY2NhSxWRTIpGRkaE+ffooMjJS5cqVk6enp9UBAAD+x15DG6UxcuRIbdiwQUeOHNH333+vLl26yNPTU7169VJQUJAGDhyoxMRErVu3Ttu2bVP//v0VGxtbqhUbko1DG/369VNqaqrGjh2riIgImUz2KdkAAOCObFlxcbV+/fVX9erVSydPnlSlSpV05513atOmTapUqZIkadq0afLw8FC3bt2Uk5Ojtm3bas6cOaXux2QYhlHaLwUEBOibb75Rw4YNS91hWcho2dzZIQAuqcp3B5wdAuBy8nOPObyP1MZ32+U+1X5Y8/cXlTGbhjaqVq0qG/IPAADgZmxKJKZPn66nn35aR44csXM4AAC4H2dtSFUWbJoj0aNHD507d041a9ZU+fLl5eXlZXX+jz/+sEtwAAC4A1dNAuzBpkRi+vTpdg4DAABci2xKJOLj4+0dBwAAbsudpxXavLNlQUGBVqxYoZ9//lnSxVeV3nfffewjAQDAZRjauMyBAwfUvn17HTt2THXq1JF0cf/vqlWr6rPPPlPNmjXtGiQAAHBNNq3aGDp0qGrWrKn//Oc/2r59u7Zv367U1FRFR0dr6NCh9o4RAIBrmmGY7HK4IpsqEhs2bNCmTZsUEhJiaQsNDdXLL7+spk2b2i04AADcQWm3t76W2FSRMJvNOnPmTJH27OxseXt7X3VQAADg2mBTItGxY0cNGjRImzdvlmEYMgxDmzZt0mOPPab77rvP3jECAHBNKzRMdjlckU2JxMyZM1WzZk3FxsbKx8dHPj4+atq0qWrVqqUZM2bYO0YAAK5pzJG4THBwsP79739r//79+uWXXyRJdevWVa1atewaHAAA7oDln1dQu3Zt1a5d216xAACAa0yJE4nExERNnDhRfn5+SkxM/Mtrp06detWBAQDgLtjZUlJKSory8vIsPwMAgJJhaEPSunXriv0ZAAD8c9m0amPAgAHF7iNx9uxZDRgw4KqDAgDAnbD88zILFy7U+fPni7SfP39eixYtuuqgAABwJyz//K/Tp09bNqA6c+aMfHx8LOcKCgr0+eefq3LlynYPEgAAuKZSJRLBwcEymUwymUy6/vrri5w3mUwaP3683YIDAMAdsGrjv9atWyfDMNSqVSt99NFHVi/t8vb2VvXq1RUZGWn3IAEAuJa56vwGeyhVItG8eXNJ0uHDh1WtWjWZTO77iwEAAH/PpsmWa9eu1YcfflikfdmyZVq4cOFVBwUAgDtx58mWNiUSSUlJqlixYpH2ypUr66WXXrrqoAAAcCeGYZ/DFdn0ro3U1FRFR0cXaa9evbpSU1OvOigAANyJO8+RsKkiUblyZf34449F2nfu3KnQ0NCrDgoAAFwbbKpI9OrVS0OHDlVAQICaNWsmSdqwYYOefPJJ9ezZ064B2qLG5qPODgFwSeePf+PsEIB/JFed32APNiUSEydO1JEjR3T33XerXLmLtygsLFTfvn2ZIwEAwGXceWjDpkTC29tb77//viZOnKidO3fK19dX9evXV/Xq1e0dHwAAcGE2JRKXXH/99cXucAkAAP7HRRdc2EWJE4nExERNnDhRfn5+SkxM/Mtrp06detWBAQDgLhjakJSSkqK8vDzLz1fCbpcAAPxzlDiRWLduXbE/AwCAv8aqDQAAYLNCZwfgQCVOJLp27Vrimy5fvtymYAAAwLWlxIlEUFCQ5WfDMPTxxx8rKChIjRs3liRt27ZNmZmZpUo4AAD4JzDkvkMbJd4ie/78+ZYjLCxMDzzwgA4fPqzly5dr+fLlOnTokHr27Fnsy7wAAPgnKzTsc1yNl19+WSaTScOGDbO0XbhwQQkJCQoNDZW/v7+6deumjIyMUt3XpndtvPPOOxo5cqQ8PT0tbZ6enkpMTNQ777xjyy0BAHBbhTLZ5bDV1q1b9eabb6pBgwZW7cOHD9fKlSu1bNkybdiwQcePHy/1yIJNiUR+fr5++eWXIu2//PKLCgvdeUoJAADXluzsbPXu3VtvvfWWKlSoYGnPysrSvHnzNHXqVLVq1UqNGjXS/Pnz9f3332vTpk0lvr9Nqzb69++vgQMH6uDBg7rtttskSZs3b9bLL7+s/v3723JLAADclr3mSOTk5CgnJ8eqzWw2y2w2X/E7CQkJ6tChg+Li4vTCCy9Y2rdt26a8vDzFxcVZ2mJiYlStWjUlJyfrjjvuKFFMNiUSkydPVnh4uKZMmaK0tDRJUkREhEaNGqURI0bYcksAANyWvWr1SUlJGj9+vFXb888/r3HjxhV7/Xvvvaft27dr69atRc6lp6fL29tbwcHBVu1hYWFKT08vcUw2JRIeHh4aPXq0Ro8erdOnT0uSAgMDbbkVAAAooTFjxhR5TcWVqhH/+c9/9OSTT2r16tXy8fFxWEw2zZGQLs6T+Prrr7V06VLLttjHjx9Xdna23YIDAMAdGDLZ5TCbzQoMDLQ6rpRIbNu2TSdOnNAtt9yicuXKqVy5ctqwYYNmzpypcuXKKSwsTLm5ucrMzLT6XkZGhsLDw0v8bDZVJI4ePap77rlHqampysnJUevWrRUQEKBXXnlFOTk5euONN2y5LQAAbskZyxDuvvtu7dq1y6qtf//+iomJ0VNPPaWqVavKy8tLa9asUbdu3SRJe/fuVWpqqmJjY0vcj02JxJNPPqnGjRtr586dCg0NtbR36dJFjzzyiC23BAAAdhQQEKAbb7zRqs3Pz0+hoaGW9oEDByoxMVEhISEKDAzUkCFDFBsbW+KJlpKNicQ333yj77//Xt7e3lbtUVFROnbsmC23BADAbbnqxgjTpk2Th4eHunXrppycHLVt21Zz5swp1T1sSiQKCwtVUFBQpP3XX39VQECALbcEAMBtucoW2evXr7f67OPjo9mzZ2v27Nk239OmyZZt2rTR9OnTLZ9NJpOys7P1/PPPq3379jYHAwAAri027yNxzz33qF69erpw4YIefPBB7d+/XxUrVtTSpUvtHSMAANe0QtcoSDiETYlE1apVtXPnTr3//vvauXOnsrOzNXDgQPXu3Vu+vr72jhEAgGva1bwnw9WVOpHIy8tTTEyMPv30U/Xu3Vu9e/d2RFwAALiNq3xxp0sr9RwJLy8vXbhwwRGxAACAa4xNky0TEhL0yiuvKD8/397xAADgdgrtdLgim+ZIbN26VWvWrNGqVatUv359+fn5WZ1fvny5XYIDAMAdFJqYI2ElODjYsp0mAAD45ypVIlFYWKhXX31V+/btU25urlq1aqVx48axUgMAgL/AZMv/evHFF/XMM8/I399fVapU0cyZM5WQkOCo2AAAcAvuPEeiVInEokWLNGfOHH311VdasWKFVq5cqcWLF6uw0FUfDwAAOFKpEonU1FSrLbDj4uJkMpl0/PhxuwcGAIC7KDTZ53BFpZojkZ+fLx8fH6s2Ly8v5eXl2TUoAADcCTtb/pdhGOrXr5/MZrOl7cKFC3rsscesloCy/BMAgH+GUiUS8fHxRdoeeughuwUDAIA7cudVG6VKJObPn++oOAAAcFuuOr/BHmzakAoAAJScO69ttOldGwAAABIVCQAAHI45EgAAwGbuPEeCoQ0AAGAzKhIAADiYO0+2JJEAAMDB3DmRYGgDAADYjIoEAAAOZrjxZEsSCQAAHIyhDQAAgGJQkQAAwMHcuSJBIgEAgIOxsyUAALAZO1sCAAAUg4oEAAAOxhwJAABgM3dOJBjaAAAANqMiAQCAg7FqAwAA2IxVGwAA4Jry+uuvq0GDBgoMDFRgYKBiY2P1xRdfWM5fuHBBCQkJCg0Nlb+/v7p166aMjIxS90MiAQCAgxXa6SiN6667Ti+//LK2bdumH374Qa1atVKnTp20e/duSdLw4cO1cuVKLVu2TBs2bNDx48fVtWvXUj+byTAMtxu68Ssf5ewQAJeUmbrW2SEALserYg2H95FU/SG73GfM0X9d1fdDQkL06quv6v7771elSpW0ZMkS3X///ZKkX375RXXr1lVycrLuuOOOEt+TigQAAG6uoKBA7733ns6ePavY2Fht27ZNeXl5iouLs1wTExOjatWqKTk5uVT3ZrIlAAAOVmindRs5OTnKycmxajObzTKbzcVev2vXLsXGxurChQvy9/fXxx9/rHr16mnHjh3y9vZWcHCw1fVhYWFKT08vVUxUJAAAcDB7zZFISkpSUFCQ1ZGUlHTFfuvUqaMdO3Zo8+bNGjx4sOLj47Vnzx67PhsVCQAAHMxekxHHjBmjxMREq7YrVSMkydvbW7Vq1ZIkNWrUSFu3btWMGTPUo0cP5ebmKjMz06oqkZGRofDw8FLFREUCAIBrhNlstiznvHT8VSJxucLCQuXk5KhRo0by8vLSmjVrLOf27t2r1NRUxcbGliomKhIAADiYM961MWbMGLVr107VqlXTmTNntGTJEq1fv15fffWVgoKCNHDgQCUmJiokJESBgYEaMmSIYmNjS7ViQyKRAADA4Zyxs+WJEyfUt29fpaWlKSgoSA0aNNBXX32l1q1bS5KmTZsmDw8PdevWTTk5OWrbtq3mzJlT6n7YRwL4B2EfCaCosthH4rmo3na5z4Qji+1yH3uiIgEAgIPZa/mnKyKRAADAwdw3jWDVBgAAuApUJAAAcDBnrNooKy6TSOzfv1/r1q3TiRMnVFho/St/7rnnnBQVAABXjzkSDvbWW29p8ODBqlixosLDw2Uy/W+djMlkIpEAAMBFuUQi8cILL+jFF1/UU0895exQAACwO/etR7hIInHq1Cl1797d2WEAAOAQ7jxHwiVWbXTv3l2rVq1ydhgAADhEoQy7HK7IJSoStWrV0tixY7Vp0ybVr19fXl5eVueHDh3qpMgAAMBfcYktsqOjo694zmQy6dChQ6W6H1tkA8Vji2ygqLLYInt4VE+73Gfakffsch97comKxOHDh50dAgAADsMcCQAAgGK4REUiMTGx2HaTySQfHx/VqlVLnTp1UkhISBlHBgDA1TNcdKKkPbhEIpGSkqLt27eroKBAderUkSTt27dPnp6eiomJ0Zw5czRixAh9++23qlevnpOjBQCgdBjacLBOnTopLi5Ox48f17Zt27Rt2zb9+uuvat26tXr16qVjx46pWbNmGj58uLNDBQAAf+ISqzaqVKmi1atXF6k27N69W23atNGxY8e0fft2tWnTRr///vvf3o9VG0DxWLUBFFUWqzYej3rALveZc+QDu9zHnlyiIpGVlaUTJ04Uaf/tt990+vRpSVJwcLByc3PLOjQAAK6aYafDFblEItGpUycNGDBAH3/8sX799Vf9+uuv+vjjjzVw4EB17txZkrRlyxZdf/31zg0UAABYcYnJlm+++aaGDx+unj17Kj8/X5JUrlw5xcfHa9q0aZKkmJgYvf32284ME6UQERmmFyY+rdZtWqh8eV8dOnhEjz42Sinbdzk7NKBMFBQUaM68xfp01Vr9fvKUKlUMUef2rfVov14ymUzKy8/XrLkL9U3yD/r1eJr8/fx0x603a/hj/VW5Uqizw4eduer21vbgEomEv7+/3nrrLU2bNs2yi2WNGjXk7+9vuaZhw4ZOig6lFRwcqDVrPtLGjcnq0qWffv/tpGrVilbmqSxnhwaUmXn/Wqb3V3ymF58doVrR1bX7l3169sVp8vf300PdO+nChRzt2XtQj/brpTq1auj0mTN6ecabeuKp8frgnZnODh925s6rNlwikbjE399fDRo0cHYYuEqJiYP166/H9dijoyxtR4/+6sSIgLK346ef1fKuO9S8yW2SpCoRYfp89Qbt2rNXkhTg76e3Z7xk9Z1nEger18PDlJZ+QhHhlcs8ZjgO+0g4QNeuXbVgwQIFBgaqa9euf3nt8uXLyygq2EP7DnFa8/VGvfuv2brrztt1PC1Dc+e+qwXzXW+PeMBRGt5YVx9+8oWOpP6qqGrX6Zf9h7T9x90aPeSRK34nO/ucTCaTAgL8yjBS4Oo4LZEICgqSyWSy/GyrnJwc5eTkWLUZhmG5N8pedHQ1PfzIQ5o1621NfnWObmnUQJMnj1Nebp4WL/7I2eEBZeLhPg/o7LlzuvfBQfL08FBBYaGGDopXx7atir0+JydX015/R+3jmsvfj0TC3TC04QDz588v9ufSSkpK0vjx463aypULkrdXsM33xNXx8DBp+/ZdGvf8q5KknTt3q1696zXw4d4kEvjH+HLtRn26ap1eGTdataKr65f9h/TKjDdVuWKIOrVvbXVtXn6+Rox9SYZhaOyoJ5wUMRzJnYc2XGL559UYM2aMsrKyrA6vcrZXOHD10tNP6Jdf9lu17d17UFWrRjopIqDsTZk9Tw8/9IDax7XQ9TWjdd89d6tvjy56+13rDYUuJRHHM07orekvUY3ANcclEomMjAz16dNHkZGRKleunDw9Pa2Ov2I2mxUYGGh1MKzhXJuSt6l2beud4mrXilZq6jEnRQSUvQsXcmTysP53kYeHhwr/tJnwpSQi9T/H9fb0lxQcFFjWYaKMFNrpcEUusWqjX79+Sk1N1dixYxUREUEicI2b9do8rV37kUaOelzLP/pMjRvfpP4DemnIE2OcHRpQZlo0vV1vLXxPEWGVVSu6un7ed0CL3l+uLh3aSLqYRCT+34vas++AZk8ar8LCQv1+8g9JUlBggLy8vJwZPuys0Plvo3AYl3jXRkBAgL755hu77RXBuzac7552rTRh/GjVrBWtI0f+o1mz3mbVhgvgXRtl5+zZc5r11iKt2ZisP05lqlLFELVv3UKD+z8oLy8vHUvLUNv7+xX73XdmvaLbbmEpfFkpi3dt9Kn+16sTS+rdo663itElEol69epp8eLFuvnmm+1yPxIJoHgkEkBRZZFIPGSnROJfLphIuMQcienTp+vpp5/WkSNHnB0KAAB2VyjDLocrcok5Ej169NC5c+dUs2ZNlS9fvsjY4B9//OGkyAAAwF9xiURi+vTpzg4BAACHced9JFwikYiPj3d2CAAAOIyrLt20B5eYIyFJBw8e1LPPPqtevXrpxIkTkqQvvvhCu3fvdnJkAABcHXeeI+ESicSGDRtUv359bd68WcuXL1d2drYkaefOnXr++eedHB0AALgSl0gknn76ab3wwgtavXq1vL29Le2tWrXSpk2bnBgZAABXz7DT/1yRSyQSu3btUpcuXYq0V65cWb///rsTIgIAwH6csUV2UlKSbr31VgUEBKhy5crq3Lmz9u7da3XNhQsXlJCQoNDQUPn7+6tbt27KyMgoVT8ukUgEBwcrLS2tSHtKSoqqVKnihIgAALi2bdiwQQkJCdq0aZNWr16tvLw8tWnTRmfPnrVcM3z4cK1cuVLLli3Thg0bdPz4cXXtWrrNs1xi1UbPnj311FNPadmyZTKZTCosLNR3332nkSNHqm/fvs4ODwCAq+KMTaS//PJLq88LFixQ5cqVtW3bNjVr1kxZWVmaN2+elixZolatWkmS5s+fr7p162rTpk264447StSPS1QkXnrpJcXExKhq1arKzs5WvXr1dNddd6lJkyZ69tlnnR0eAABXxV6rNnJycnT69GmrIycnp0QxZGVlSZJCQkIkSdu2bVNeXp7i4uIs18TExKhatWpKTk4u8bO5RCLh7e2tt956S4cOHdKnn36qxYsXa9++fXr33Xf/9jXiAAD8UyQlJSkoKMjqSEpK+tvvFRYWatiwYWratKluvPFGSVJ6erq8vb0VHBxsdW1YWJjS09NLHJNLDG1I0rx58zRt2jTt379fklS7dm0NGzZMDz/8sJMjAwDg6thrQ6oxY8YoMTHRqs1sNv/t9xISEvTTTz/p22+/tVMk/+MSicRzzz2nqVOnasiQIYqNjZUkJScna/jw4UpNTdWECROcHCEAALaz19JNs9lcosThz5544gl9+umn2rhxo6677jpLe3h4uHJzc5WZmWlVlcjIyFB4eHiJ7+8SicTrr7+ut956S7169bK03XfffWrQoIGGDBlCIgEAQCkZhqEhQ4bo448/1vr16xUdHW11vlGjRvLy8tKaNWvUrVs3SdLevXuVmppq+Y/6knCJRCIvL0+NGzcu0t6oUSPl5+c7ISIAAOzHGdtbJyQkaMmSJfr3v/+tgIAAy7yHoKAg+fr6KigoSAMHDlRiYqJCQkIUGBhoGRko6YoNyUUmW/bp00evv/56kfa5c+eqd+/eTogIAAD7MQzDLkdpvP7668rKylKLFi0UERFhOd5//33LNdOmTVPHjh3VrVs3NWvWTOHh4Vq+fHmp+nFaReLPk0VMJpPefvttrVq1ypIFbd68WampqewjAQC45jnj7Z8lSTx8fHw0e/ZszZ492+Z+nJZIpKSkWH1u1KiRpItvAZWkihUrqmLFirz9EwAAF+a0RGLdunXO6hoAgDLlqi/csgeXmGwJAIA7c8Zky7LiEpMtAQDAtYmKBAAADuaMl3aVFRIJAAAcjKENAACAYlCRAADAwVi1AQAAbFboxnMkGNoAAAA2oyIBAICDuW89gkQCAACHc+dVGyQSAAA4mDsnEsyRAAAANqMiAQCAg7GzJQAAsBlDGwAAAMWgIgEAgIOxsyUAALCZO8+RYGgDAADYjIoEAAAO5s6TLUkkAABwMIY2AAAAikFFAgAAB2NoAwAA2IzlnwAAwGaFzJEAAAAoiooEAAAOxtAGAACwGUMbAAAAxaAiAQCAgzG0AQAAbMbQBgAAQDGoSAAA4GAMbQAAAJsxtAEAAFAMEgkAABzMsNP/Smvjxo269957FRkZKZPJpBUrVljHZRh67rnnFBERIV9fX8XFxWn//v2l6oNEAgAABzOMQrscpXX27FnddNNNmj17drHnJ02apJkzZ+qNN97Q5s2b5efnp7Zt2+rChQsl7oM5EgAAOJizXiPerl07tWvXrthzhmFo+vTpevbZZ9WpUydJ0qJFixQWFqYVK1aoZ8+eJeqDigQAANeInJwcnT592urIycmx6V6HDx9Wenq64uLiLG1BQUG6/fbblZycXOL7kEgAAOBghmHY5UhKSlJQUJDVkZSUZFNM6enpkqSwsDCr9rCwMMu5kmBoAwAAB7PX0MaYMWOUmJho1WY2m+1yb1uRSAAAcI0wm812SxzCw8MlSRkZGYqIiLC0Z2RkqGHDhiW+D0MbAAA4mL2GNuwpOjpa4eHhWrNmjaXt9OnT2rx5s2JjY0t8HyoSAAA4mLN2tszOztaBAwcsnw8fPqwdO3YoJCRE1apV07Bhw/TCCy+odu3aio6O1tixYxUZGanOnTuXuA8SCQAA3NQPP/ygli1bWj5fml8RHx+vBQsWaPTo0Tp79qwGDRqkzMxM3Xnnnfryyy/l4+NT4j5Mhr1rJS7Ar3yUs0MAXFJm6lpnhwC4HK+KNRzeR3hwXbvcJz3zZ7vcx56oSAAA4GBu+N/sFky2BAAANqMiAQCAgzlri+yyQCIBAICDufPQBokEAAAO5qzln2WBORIAAMBmVCQAAHAwhjYAAIDN3HmyJUMbAADAZlQkAABwMIY2AACAzVi1AQAAUAwqEgAAOJjhxpMtSSQAAHAwhjYAAACKQUUCAAAHY9UGAACwGXMkAACAzdy5IsEcCQAAYDMqEgAAOJg7VyRIJAAAcDD3TSMY2gAAAFfBZLhzvQVOlZOTo6SkJI0ZM0Zms9nZ4QAugz8bcCckEnCY06dPKygoSFlZWQoMDHR2OIDL4M8G3AlDGwAAwGYkEgAAwGYkEgAAwGYkEnAYs9ms559/nslkwGX4swF3wmRLAABgMyoSAADAZiQSAADAZiQSAADAZiQSKJV+/fqpc+fOls8tWrTQsGHDnBYP4Ehl8c/35X+mgGsNL+3CVVm+fLm8vLycHUaxoqKiNGzYMBIduLQZM2a49Zsh4f5IJHBVQkJCnB0CcE0LCgpydgjAVWFow421aNFCQ4YM0bBhw1ShQgWFhYXprbfe0tmzZ9W/f38FBASoVq1a+uKLLyRJBQUFGjhwoKKjo+Xr66s6depoxowZf9vHn/+LPy0tTR06dJCvr6+io6O1ZMkSRUVFafr06ZZrTCaT3n77bXXp0kXly5dX7dq19cknn1jOlySOS+XgyZMnKyIiQqGhoUpISFBeXp4lrqNHj2r48OEymUwymUxX+dvEP1V+fr6eeOIJBQUFqWLFiho7dqylgpCTk6ORI0eqSpUq8vPz0+23367169dbvrtgwQIFBwfrq6++Ut26deXv76977rlHaWlplmsuH9o4c+aMevfuLT8/P0VERGjatGlF/pxFRUXppZde0oABAxQQEKBq1app7ty5jv5VAMUikXBzCxcuVMWKFbVlyxYNGTJEgwcPVvfu3dWkSRNt375dbdq0UZ8+fXTu3DkVFhbquuuu07Jly7Rnzx4999xzeuaZZ/TBBx+UuL++ffvq+PHjWr9+vT766CPNnTtXJ06cKHLd+PHj9cADD+jHH39U+/bt1bt3b/3xxx+SVOI41q1bp4MHD2rdunVauHChFixYoAULFki6OORy3XXXacKECUpLS7P6FzdQGgsXLlS5cuW0ZcsWzZgxQ1OnTtXbb78tSXriiSeUnJys9957Tz/++KO6d++ue+65R/v377d8/9y5c5o8ebLeffddbdy4UampqRo5cuQV+0tMTNR3332nTz75RKtXr9Y333yj7du3F7luypQpaty4sVJSUvT4449r8ODB2rt3r/1/AcDfMeC2mjdvbtx5552Wz/n5+Yafn5/Rp08fS1taWpohyUhOTi72HgkJCUa3bt0sn+Pj441OnTpZ9fHkk08ahmEYP//8syHJ2Lp1q+X8/v37DUnGtGnTLG2SjGeffdbyOTs725BkfPHFF1d8luLiqF69upGfn29p6969u9GjRw/L5+rVq1v1C5RW8+bNjbp16xqFhYWWtqeeesqoW7eucfToUcPT09M4duyY1XfuvvtuY8yYMYZhGMb8+fMNScaBAwcs52fPnm2EhYVZPv/5z9Tp06cNLy8vY9myZZbzmZmZRvny5S1/zgzj4j/bDz30kOVzYWGhUblyZeP111+3y3MDpcEcCTfXoEEDy8+enp4KDQ1V/fr1LW1hYWGSZKkazJ49W++8845SU1N1/vx55ebmqmHDhiXqa+/evSpXrpxuueUWS1utWrVUoUKFv4zLz89PgYGBVpWLksRxww03yNPT0/I5IiJCu3btKlGsQEndcccdVkNjsbGxmjJlinbt2qWCggJdf/31Vtfn5OQoNDTU8rl8+fKqWbOm5XNERESxVTpJOnTokPLy8nTbbbdZ2oKCglSnTp0i1/75z5DJZFJ4ePgV7ws4EomEm7t8RYXJZLJqu/QvyMLCQr333nsaOXKkpkyZotjYWAUEBOjVV1/V5s2byySuwsJCSSpxHH91D8DRsrOz5enpqW3btlkltJLk7+9v+bm4f04NO6zS4J9/uAoSCVh89913atKkiR5//HFL28GDB0v8/Tp16ig/P18pKSlq1KiRJOnAgQM6depUmcZxibe3twoKCkr9PeDPLk9gN23apNq1a+vmm29WQUGBTpw4obvuussufdWoUUNeXl7aunWrqlWrJknKysrSvn371KxZM7v0Adgbky1hUbt2bf3www/66quvtG/fPo0dO1Zbt24t8fdjYmIUFxenQYMGacuWLUpJSdGgQYPk6+tbqlUTVxvHJVFRUdq4caOOHTum33//vdTfByQpNTVViYmJ2rt3r5YuXapZs2bpySef1PXXX6/evXurb9++Wr58uQ4fPqwtW7YoKSlJn332mU19BQQEKD4+XqNGjdK6deu0e/duDRw4UB4eHqw8gssikYDFo48+qq5du6pHjx66/fbbdfLkSauqQEksWrRIYWFhatasmbp06aJHHnlEAQEB8vHxKdM4JGnChAk6cuSIatasqUqVKpX6+4B0cSXS+fPnddtttykhIUFPPvmkBg0aJEmaP3+++vbtqxEjRqhOnTrq3LmzVTXBFlOnTlVsbKw6duyouLg4NW3aVHXr1i3VnyGgLPEacTjUr7/+qqpVq+rrr7/W3Xff7exwgGvO2bNnVaVKFU2ZMkUDBw50djhAEcyRgF2tXbtW2dnZql+/vtLS0jR69GhFRUUxvguUUEpKin755RfddtttysrK0oQJEyRJnTp1cnJkQPFIJGBXeXl5euaZZ3To0CEFBASoSZMmWrx4scu+jwNwRZMnT9bevXvl7e2tRo0a6ZtvvlHFihWdHRZQLIY2AACAzZhsCQAAbEYiAQAAbEYiAQAAbEYiAQAAbEYiAaDUTCaTVqxY4ewwALgAEgnAxSUnJ8vT01MdOnQo1feioqI0ffp0xwQFAP9FIgG4uHnz5mnIkCHauHGjjh8/7uxwAMAKiQTgwrKzs/X+++9r8ODB6tChgxYsWGB1fuXKlbr11lvl4+OjihUrqkuXLpKkFi1a6OjRoxo+fLhMJpPlhU/jxo1Tw4YNre4xffp0RUVFWT5v3bpVrVu3VsWKFRUUFKTmzZtr+/btjnxMANcwEgnAhX3wwQeKiYlRnTp19NBDD+mdd97RpT3kPvvsM3Xp0kXt27dXSkqK1qxZo9tuu02StHz5cl133XWaMGGC0tLSlJaWVuI+z5w5o/j4eH377beWV2a3b99eZ86cccgzAri2sUU24MLmzZunhx56SJJ0zz33KCsrSxs2bFCLFi304osvqmfPnho/frzl+ptuukmSFBISIk9PTwUEBCg8PLxUfbZq1crq89y5cxUcHKwNGzaoY8eOV/lEANwNFQnARe3du1dbtmxRr169JEnlypVTjx49NG/ePEnSjh07HPJG1YyMDD3yyCOqXbu2goKCFBgYqOzsbKWmptq9LwDXPioSgIuaN2+e8vPzFRkZaWkzDENms1mvvfaafH19S31PDw8PXf56nby8PKvP8fHxOnnypGbMmKHq1avLbDYrNjZWubm5tj0IALdGRQJwQfn5+Vq0aJGmTJmiHTt2WI6dO3cqMjJSS5cuVYMGDbRmzZor3sPb21sFBQVWbZUqVVJ6erpVMrFjxw6ra7777jsNHTpU7du31w033CCz2azff//drs8HwH1QkQBc0KeffqpTp05p4MCBCgoKsjrXrVs3zZs3T6+++qruvvtu1axZUz179lR+fr4+//xzPfXUU5Iu7iOxceNG9ezZU2azWRUrVlSLFi3022+/adKkSbr//vv15Zdf6osvvlBgYKDl/rVr19a7776rxo0b6/Tp0xo1apRN1Q8A/wxUJAAXNG/ePMXFxRVJIqSLicQPP/ygkJAQLVu2TJ988okaNmyoVq1aacuWLZbrJkyYoCNHjqhmzZqqVKmSJKlu3bqaM2eOZs+erZtuuklbtmzRyJEji/R96tQp3XLLLerTp4+GDh2qypUrO/aBAVyzTMblA6YAAAAlREUCAADYjEQCAADYjEQCAADYjEQCAADYjEQCAADYjEQCAADYjEQCAADYjEQCAADYjEQCAADYjEQCAADYjEQCAADYjEQCAADY7P8BnC1Gp1f6E/cAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.9300699300699301\n",
      "Precision:  0.9534883720930233\n",
      "Recall:  0.9318181818181818\n",
      "F1_score:  0.942528735632184\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "\n",
    "# loading the dataset\n",
    "x, y = load_breast_cancer(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.25)\n",
    "\n",
    "# training\n",
    "tree = DecisionTreeClassifier(random_state=23)\n",
    "tree.fit(X_train, y_train)\n",
    "\n",
    "# prediction\n",
    "y_pred = tree.predict(X_test)\n",
    "\n",
    "# Confusion Matrix\n",
    "cm = confusion_matrix(y_test, y_pred)\n",
    "\n",
    "sns.heatmap(cm, annot=True, fmt='g',\n",
    "            xticklabels=['malignant', 'benign'],\n",
    "            yticklabels=['malignant', 'benign'])\n",
    "plt.ylabel('Prediction')\n",
    "plt.xlabel('Actual')\n",
    "plt.title('Confusion Matrix')\n",
    "plt.show()\n",
    "\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy: \", accuracy)\n",
    "precision = precision_score(y_test, y_pred)\n",
    "print(\"Precision: \", precision)\n",
    "recall = recall_score(y_test, y_pred)\n",
    "print(\"Recall: \", recall)\n",
    "F1_score = f1_score(y_test, y_pred)\n",
    "print(\"F1_score: \", F1_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.Performance Metrics for Regression Problems\n",
    "\n",
    "### a. Mean Absolute Error (MAE)\n",
    "\n",
    "It measures the average absolute difference between actual and predicted values, absolute meaning taking a number as positive\n",
    "\n",
    "![mean absolute error](mae.png)\n",
    "\n",
    "where:-\n",
    "- Y, the actual outcome\n",
    "- Y', the prdicted outcome\n",
    "- N, the total number of data points\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Limitations of M.A.E\n",
    "\n",
    "it's not differentiable, thus we have to use optimizers such as Gradient Descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Absolute Error:  1.8\n"
     ]
    }
   ],
   "source": [
    "actual = [2, 3, 5, 5, 9]\n",
    "calculated = [3, 3, 8, 7, 6]\n",
    "\n",
    "n = 5\n",
    "sum = 0\n",
    "\n",
    "for i in range(n):\n",
    "    sum += abs(actual[i] - calculated[i])\n",
    "\n",
    "error = sum/n\n",
    "\n",
    "print(\"Mean Absolute Error: \", error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Absolute Error:  0.0979020979020979\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "# loading the dataset\n",
    "x, y = load_breast_cancer(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.25)\n",
    "\n",
    "# training\n",
    "tree = DecisionTreeClassifier(random_state=23)\n",
    "tree.fit(X_train, y_train)\n",
    "\n",
    "# prediction\n",
    "y_pred = tree.predict(X_test)\n",
    "\n",
    "mae = mean_absolute_error(y_test, y_pred)\n",
    "print(\"Mean Absolute Error: \", mae)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b. Mean Squared Error (MSE)\n",
    "\n",
    "It's also known as ***Mean Squared Deviation***\n",
    "\n",
    "It measures the average of the squared difference between predicted values and the actual values given by the model\n",
    "\n",
    "Due to squared difference it penalizes small error, this leads to over estimation of how bad the model really is\n",
    "\n",
    "![mean squared error](mse.png)\n",
    "\n",
    "where:- \n",
    "- y, actual value\n",
    "- y', the predicted value\n",
    "- n, the total number of data points\n",
    "\n",
    "<code>\n",
    "from sklearn.metrics import mean_squared_error\n",
    "</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Squared Error:  0.055944055944055944\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "# loading the dataset\n",
    "x, y = load_breast_cancer(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.25)\n",
    "\n",
    "# training\n",
    "tree = DecisionTreeClassifier(random_state=23)\n",
    "tree.fit(X_train, y_train)\n",
    "\n",
    "# prediction\n",
    "y_pred = tree.predict(X_test)\n",
    "\n",
    "mse = mean_squared_error(y_test, y_pred)\n",
    "print(\"Mean Squared Error: \", mse)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}