getting_started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started with the BRIDGE dataset\n",
    "\n",
    "This notebook walks through some examples of how to use the tabular MDPs in the BRIDGE dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, install the `effective-horizon` package, either from the local repository or from PyPI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "nbreg": {
     "diff_ignore": [
      "/outputs/0/text"
     ]
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Obtaining file:///home/cassidy/sh/Programs/Python/effective-horizon\r\n",
      "  Installing build dependencies ... \u001b[?25l-\b \b\\\b \b|\b \bdone\r\n",
      "\u001b[?25h  Checking if build backend supports build_editable ... \u001b[?25ldone\r\n",
      "\u001b[?25h  Getting requirements to build editable ... \u001b[?25l-\b \bdone\r\n",
      "\u001b[?25h  Installing backend dependencies ... \u001b[?25l-\b \b\\\b \bdone\r\n",
      "\u001b[?25h  Preparing editable metadata (pyproject.toml) ... \u001b[?25l-\b \bdone\r\n",
      "\u001b[?25hRequirement already satisfied: sacred>=0.8.2 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (0.8.4)\r\n",
      "Requirement already satisfied: torch>=1.10.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (2.1.0)\r\n",
      "Requirement already satisfied: minigrid<2.3,>=2.2 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (2.2.1)\r\n",
      "Requirement already satisfied: gymnasium[atari] in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (0.29.1)\r\n",
      "Requirement already satisfied: numpy>=1.20.3 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (1.26.0)\r\n",
      "Requirement already satisfied: scipy>=1.8 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (1.10.1)\r\n",
      "Requirement already satisfied: autorom>=0.4.2 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from autorom[accept-rom-license]>=0.4.2->effective-horizon==0.1.0) (0.6.1)\r\n",
      "Requirement already satisfied: GPUtil>=1.4.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (1.4.0)\r\n",
      "Requirement already satisfied: procgen>=0.10.7 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (0.10.7)\r\n",
      "Requirement already satisfied: torchvision>=0.11.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (0.16.0)\r\n",
      "Requirement already satisfied: tensorflow-probability in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (0.22.0)\r\n",
      "Requirement already satisfied: tqdm in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (4.66.1)\r\n",
      "Requirement already satisfied: opencv-python in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (4.9.0.80)\r\n",
      "Requirement already satisfied: tensorboard in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from effective-horizon==0.1.0) (2.16.2)\r\n",
      "Requirement already satisfied: click in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from autorom>=0.4.2->autorom[accept-rom-license]>=0.4.2->effective-horizon==0.1.0) (8.1.7)\r\n",
      "Requirement already satisfied: requests in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from autorom>=0.4.2->autorom[accept-rom-license]>=0.4.2->effective-horizon==0.1.0) (2.31.0)\r\n",
      "Requirement already satisfied: AutoROM.accept-rom-license in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from autorom[accept-rom-license]>=0.4.2->effective-horizon==0.1.0) (0.6.1)\r\n",
      "Requirement already satisfied: pygame>=2.2.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from minigrid<2.3,>=2.2->effective-horizon==0.1.0) (2.5.2)\r\n",
      "Requirement already satisfied: gym<1.0.0,>=0.15.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from procgen>=0.10.7->effective-horizon==0.1.0) (0.26.2)\r\n",
      "Requirement already satisfied: gym3<1.0.0,>=0.3.3 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from procgen>=0.10.7->effective-horizon==0.1.0) (0.3.3)\r\n",
      "Requirement already satisfied: filelock<4.0.0,>=3.0.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from procgen>=0.10.7->effective-horizon==0.1.0) (3.12.4)\r\n",
      "Requirement already satisfied: docopt<1.0,>=0.3 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from sacred>=0.8.2->effective-horizon==0.1.0) (0.6.2)\r\n",
      "Requirement already satisfied: jsonpickle>=1.2 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from sacred>=0.8.2->effective-horizon==0.1.0) (3.0.2)\r\n",
      "Requirement already satisfied: munch<3.0,>=2.5 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from sacred>=0.8.2->effective-horizon==0.1.0) (2.5.0)\r\n",
      "Requirement already satisfied: wrapt<2.0,>=1.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from sacred>=0.8.2->effective-horizon==0.1.0) (1.15.0)\r\n",
      "Requirement already satisfied: py-cpuinfo>=4.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from sacred>=0.8.2->effective-horizon==0.1.0) (9.0.0)\r\n",
      "Requirement already satisfied: colorama>=0.4 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from sacred>=0.8.2->effective-horizon==0.1.0) (0.4.6)\r\n",
      "Requirement already satisfied: packaging>=18.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from sacred>=0.8.2->effective-horizon==0.1.0) (23.2)\r\n",
      "Requirement already satisfied: GitPython in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from sacred>=0.8.2->effective-horizon==0.1.0) (3.1.37)\r\n",
      "Requirement already satisfied: typing-extensions in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (4.5.0)\r\n",
      "Requirement already satisfied: sympy in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (1.12)\r\n",
      "Requirement already satisfied: networkx in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (3.1)\r\n",
      "Requirement already satisfied: jinja2 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (3.1.2)\r\n",
      "Requirement already satisfied: fsspec in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (2023.9.2)\r\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (12.1.105)\r\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (12.1.105)\r\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (12.1.105)\r\n",
      "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (8.9.2.26)\r\n",
      "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (12.1.3.1)\r\n",
      "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (11.0.2.54)\r\n",
      "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (10.3.2.106)\r\n",
      "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (11.4.5.107)\r\n",
      "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (12.1.0.106)\r\n",
      "Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (2.18.1)\r\n",
      "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (12.1.105)\r\n",
      "Requirement already satisfied: triton==2.1.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torch>=1.10.0->effective-horizon==0.1.0) (2.1.0)\r\n",
      "Requirement already satisfied: nvidia-nvjitlink-cu12 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.10.0->effective-horizon==0.1.0) (12.2.140)\r\n",
      "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from torchvision>=0.11.1->effective-horizon==0.1.0) (10.0.1)\r\n",
      "Requirement already satisfied: cloudpickle>=1.2.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from gymnasium[atari]->effective-horizon==0.1.0) (2.2.1)\r\n",
      "Requirement already satisfied: farama-notifications>=0.0.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from gymnasium[atari]->effective-horizon==0.1.0) (0.0.4)\r\n",
      "Requirement already satisfied: importlib-metadata>=4.8.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from gymnasium[atari]->effective-horizon==0.1.0) (6.8.0)\r\n",
      "Requirement already satisfied: shimmy<1.0,>=0.1.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from shimmy[atari]<1.0,>=0.1.0; extra == \"atari\"->gymnasium[atari]->effective-horizon==0.1.0) (0.2.1)\r\n",
      "Requirement already satisfied: absl-py>=0.4 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from tensorboard->effective-horizon==0.1.0) (2.0.0)\r\n",
      "Requirement already satisfied: grpcio>=1.48.2 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from tensorboard->effective-horizon==0.1.0) (1.59.0)\r\n",
      "Requirement already satisfied: markdown>=2.6.8 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from tensorboard->effective-horizon==0.1.0) (3.5.2)\r\n",
      "Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from tensorboard->effective-horizon==0.1.0) (4.24.4)\r\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from tensorboard->effective-horizon==0.1.0) (68.0.0)\r\n",
      "Requirement already satisfied: six>1.9 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from tensorboard->effective-horizon==0.1.0) (1.16.0)\r\n",
      "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from tensorboard->effective-horizon==0.1.0) (0.7.2)\r\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from tensorboard->effective-horizon==0.1.0) (3.0.1)\r\n",
      "Requirement already satisfied: decorator in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from tensorflow-probability->effective-horizon==0.1.0) (5.1.1)\r\n",
      "Requirement already satisfied: gast>=0.3.2 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from tensorflow-probability->effective-horizon==0.1.0) (0.5.4)\r\n",
      "Requirement already satisfied: dm-tree in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from tensorflow-probability->effective-horizon==0.1.0) (0.1.8)\r\n",
      "Requirement already satisfied: gym-notices>=0.0.4 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from gym<1.0.0,>=0.15.0->procgen>=0.10.7->effective-horizon==0.1.0) (0.0.8)\r\n",
      "Requirement already satisfied: cffi<2.0.0,>=1.13.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from gym3<1.0.0,>=0.3.3->procgen>=0.10.7->effective-horizon==0.1.0) (1.16.0)\r\n",
      "Requirement already satisfied: imageio<3.0.0,>=2.6.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from gym3<1.0.0,>=0.3.3->procgen>=0.10.7->effective-horizon==0.1.0) (2.31.5)\r\n",
      "Requirement already satisfied: imageio-ffmpeg<0.4.0,>=0.3.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from gym3<1.0.0,>=0.3.3->procgen>=0.10.7->effective-horizon==0.1.0) (0.3.0)\r\n",
      "Requirement already satisfied: glfw<2.0.0,>=1.8.6 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from gym3<1.0.0,>=0.3.3->procgen>=0.10.7->effective-horizon==0.1.0) (1.12.0)\r\n",
      "Requirement already satisfied: moderngl<6.0.0,>=5.5.4 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from gym3<1.0.0,>=0.3.3->procgen>=0.10.7->effective-horizon==0.1.0) (5.8.2)\r\n",
      "Requirement already satisfied: zipp>=0.5 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from importlib-metadata>=4.8.0->gymnasium[atari]->effective-horizon==0.1.0) (3.17.0)\r\n",
      "Requirement already satisfied: ale-py~=0.8.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from shimmy[atari]<1.0,>=0.1.0; extra == \"atari\"->gymnasium[atari]->effective-horizon==0.1.0) (0.8.1)\r\n",
      "Requirement already satisfied: MarkupSafe>=2.1.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from werkzeug>=1.0.1->tensorboard->effective-horizon==0.1.0) (2.1.3)\r\n",
      "Requirement already satisfied: gitdb<5,>=4.0.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from GitPython->sacred>=0.8.2->effective-horizon==0.1.0) (4.0.10)\r\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from requests->autorom>=0.4.2->autorom[accept-rom-license]>=0.4.2->effective-horizon==0.1.0) (3.3.0)\r\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from requests->autorom>=0.4.2->autorom[accept-rom-license]>=0.4.2->effective-horizon==0.1.0) (3.4)\r\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from requests->autorom>=0.4.2->autorom[accept-rom-license]>=0.4.2->effective-horizon==0.1.0) (2.0.6)\r\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from requests->autorom>=0.4.2->autorom[accept-rom-license]>=0.4.2->effective-horizon==0.1.0) (2023.7.22)\r\n",
      "Requirement already satisfied: mpmath>=0.19 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from sympy->torch>=1.10.0->effective-horizon==0.1.0) (1.3.0)\r\n",
      "Requirement already satisfied: importlib-resources in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from ale-py~=0.8.1->shimmy[atari]<1.0,>=0.1.0; extra == \"atari\"->gymnasium[atari]->effective-horizon==0.1.0) (6.1.0)\r\n",
      "Requirement already satisfied: pycparser in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from cffi<2.0.0,>=1.13.0->gym3<1.0.0,>=0.3.3->procgen>=0.10.7->effective-horizon==0.1.0) (2.21)\r\n",
      "Requirement already satisfied: smmap<6,>=3.0.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from gitdb<5,>=4.0.1->GitPython->sacred>=0.8.2->effective-horizon==0.1.0) (5.0.1)\r\n",
      "Requirement already satisfied: glcontext<3,>=2.3.6 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from moderngl<6.0.0,>=5.5.4->gym3<1.0.0,>=0.3.3->procgen>=0.10.7->effective-horizon==0.1.0) (2.4.0)\r\n",
      "Building wheels for collected packages: effective-horizon\r\n",
      "  Building editable for effective-horizon (pyproject.toml) ... \u001b[?25l-\b \bdone\r\n",
      "\u001b[?25h  Created wheel for effective-horizon: filename=effective_horizon-0.1.0-0.editable-py3-none-any.whl size=8584 sha256=4fabfcc6351b56a7b31c00dbde8b0290571ef2fc65de52b12eb6a852ed97c303\r\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-37o4_8oj/wheels/41/59/be/6f885a1d514659d5ee63e55292b47bd8503f7ed6bc71c33d69\r\n",
      "Successfully built effective-horizon\r\n",
      "Installing collected packages: effective-horizon\r\n",
      "  Attempting uninstall: effective-horizon\r\n",
      "    Found existing installation: effective-horizon 0.1.0\r\n",
      "    Uninstalling effective-horizon-0.1.0:\r\n",
      "      Successfully uninstalled effective-horizon-0.1.0\r\n",
      "Successfully installed effective-horizon-0.1.0\r\n",
      "Requirement already satisfied: requests in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (2.31.0)\r\n",
      "Requirement already satisfied: matplotlib in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (3.8.0)\r\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from requests) (3.3.0)\r\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from requests) (3.4)\r\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from requests) (2.0.6)\r\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from requests) (2023.7.22)\r\n",
      "Requirement already satisfied: contourpy>=1.0.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from matplotlib) (1.1.1)\r\n",
      "Requirement already satisfied: cycler>=0.10 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from matplotlib) (0.12.1)\r\n",
      "Requirement already satisfied: fonttools>=4.22.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from matplotlib) (4.43.1)\r\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from matplotlib) (1.4.5)\r\n",
      "Requirement already satisfied: numpy<2,>=1.21 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from matplotlib) (1.26.0)\r\n",
      "Requirement already satisfied: packaging>=20.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from matplotlib) (23.2)\r\n",
      "Requirement already satisfied: pillow>=6.2.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from matplotlib) (10.0.1)\r\n",
      "Requirement already satisfied: pyparsing>=2.3.1 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from matplotlib) (3.1.1)\r\n",
      "Requirement already satisfied: python-dateutil>=2.7 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from matplotlib) (2.8.2)\r\n",
      "Requirement already satisfied: importlib-resources>=3.2.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from matplotlib) (6.1.0)\r\n",
      "Requirement already satisfied: zipp>=3.1.0 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from importlib-resources>=3.2.0->matplotlib) (3.17.0)\r\n",
      "Requirement already satisfied: six>=1.5 in /home/cassidy/anaconda3/envs/effective_horizon/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)\r\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import warnings\n",
    "\n",
    "if os.path.exists(\"pyproject.toml\"):\n",
    "    !pip install -e .\n",
    "else:\n",
    "    !pip install effective-horizon\n",
    "\n",
    "!pip install requests matplotlib\n",
    "\n",
    "warnings.warn = lambda *args,**kwargs: None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's load a tabular version of an MDP. The repository includes a few small MDPs, but the rest are included in the [full BRIDGE dataset](https://zenodo.org/records/10966777). The `load_mdp` function will load transitions and rewards matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((169, 6), (169, 6), array([ 82, 156,   0,   0,   0,   0]))"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from effective_horizon.mdp_utils import load_mdp\n",
    "\n",
    "from io import BytesIO\n",
    "import requests\n",
    "\n",
    "mdp_file = \"data/mdps/MiniGrid-KeyCorridorS3R1-v0/consolidated.npz\"\n",
    "if not os.path.exists(mdp_file):\n",
    "    response = requests.get(\"https://github.com/cassidylaidlaw/effective-horizon/raw/main/data/mdps/MiniGrid-KeyCorridorS3R1-v0/consolidated.npz\", stream=True)\n",
    "    mdp_file = BytesIO(response.raw.read())\n",
    "\n",
    "transitions, rewards = load_mdp(mdp_file)\n",
    "transitions.shape, rewards.shape, transitions[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both the `transitions` and `rewards` matrices are of size `(num_states, num_actions)`. As we can see, this MDP has 169 states (168 non-terminal states plus the terminal state) and 6 actions. `transitions[0]` gives the states that are reached by taking each action in the initial state, which is always at index 0.\n",
    "\n",
    "Now, let's use value iteration to determine an optimal policy for this MDP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "nbreg": {
     "diff_ignore": [
      "/outputs/1/text"
     ]
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(100, 169)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Value iteration: 100%|██████████| 100/100 [00:00<00:00, 5006.69it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(1.0, 1.0)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from effective_horizon.mdp_utils import get_sparse_mdp, run_value_iteration\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "sparse_transitions, rewards_vector = get_sparse_mdp(transitions, rewards)\n",
    "vi_results = run_value_iteration(sparse_transitions, rewards_vector, horizon=100)\n",
    "\n",
    "optimal_policy = np.argmax(vi_results.optimal_qs, axis=2)\n",
    "print(optimal_policy.shape)\n",
    "\n",
    "total_reward = 0\n",
    "optimal_actions = []\n",
    "state = 0\n",
    "for t in range(100):\n",
    "    action = optimal_policy[t, state]\n",
    "    optimal_actions.append(action)\n",
    "    reward = rewards[state, action]\n",
    "    total_reward += reward\n",
    "    state = transitions[state, action]\n",
    "total_reward, vi_results.optimal_values[0, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`run_value_iteration` returns the optimal Q and value functions as well as the random policy's Q and value functions (which can be accessed by `vi_results.random_qs`). By taking the arg-max of the optimal Q function, we can get an optimal policy. Running this optimal policy in the environment gives reward equal to the optimal value at the initial state.\n",
    "\n",
    "We can also access the same environment through the [Gymnasium](https://gymnasium.farama.org/) API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n"
     ]
    }
   ],
   "source": [
    "import effective_horizon\n",
    "import gymnasium as gym\n",
    "\n",
    "env = gym.make(\"BRIDGE/MiniGrid-KeyCorridorS3R1-v0\")\n",
    "obs, info = env.reset()\n",
    "\n",
    "total_reward = 0\n",
    "for t in range(100):\n",
    "    action = optimal_actions[t]\n",
    "    obs, reward, terminated, truncated, info = env.step(action)\n",
    "    total_reward += reward\n",
    "    if terminated:\n",
    "        break\n",
    "\n",
    "print(total_reward)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tabular and Gymnasium versions of the environments are identical, as shown by the fact that the optimal policy calculated from the tabular version also achieves optimal return in the Gymnasium environment. The Gymnasium environments can be used to run deep RL algorithms and compare their results to properties computed from the tabular versions!\n",
    "\n",
    "For visualization purposes, it's also possible to view what the screen observation looks like for a particular state in the tabular MDP. For example, let's find which state has the lowest non-zero value under the random policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "nbreg": {
     "diff_ignore": [
      "/outputs/1/data/text/plain",
      "/outputs/2/data/image/png"
     ]
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "84 39 5.671526e-12\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x78f585863520>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAEECAYAAACIpFYUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAy5ElEQVR4nO3deXRddb3//+c+c+Y0TTO1aRug0AKlQIFaUVTaL4UvKlz4KnDrEpUrDi0CdcC6BC5ctAh3AQutqHcp4E8R5XsFFK/4LQWKSCi1pQICoUDpnHRIMydn2p/fH5+c04YkbdMm+wx9PViHNPvsc/Le2Scnr3z2Z3CMMQYRERERj/gyXYCIiIgcXRQ+RERExFMKHyIiIuIphQ8RERHxlMKHiIiIeErhQ0RERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTYxY+li9fztSpU4lEIsyZM4eXXnpprL6UiIiI5JAxCR+//e1vWbJkCTfffDPr1q1j1qxZLFiwgJ07d47FlxMREZEc4ozFwnJz5szhzDPP5Ec/+hEArutSX1/PNddcw7e//e0DPtZ1XbZv305JSQmO44x2aSIiIjIGjDF0dnZSV1eHz3fgto3AaH/xWCzG2rVrWbp0aXqbz+dj/vz5NDY2Dto/Go0SjUbTn2/bto0TTzxxtMsSERERD2zZsoVJkyYdcJ9RDx+7d+8mmUxSXV09YHt1dTVvvvnmoP2XLVvGLbfcMmj75ZdfTigUGu3yRI46lZWVg34eRQ7EdV22bdtGR0dHpkuRHBKLxXj44YcpKSk56L6jHj5GaunSpSxZsiT9eUdHB/X19YRCIYUPkVEQDoeJRCK6jCmHLJlMEg6H9R4sh+VQ3mtGPXxUVlbi9/tpaWkZsL2lpYWamppB+4fDYcLh8GiXISIiIllq1Ee7hEIhZs+ezcqVK9PbXNdl5cqVzJ07d7S/nIiIiOSYMbnssmTJEq688krOOOMMzjrrLO655x66u7v5/Oc/PxZfTkRERHLImISPyy67jF27dnHTTTfR3NzMqaeeypNPPqlObyIiIjJ2HU4XL17M4sWLx+rpRUREJEdpbRcRERHxlMKHiIiIeErhQ0RERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8ZTCh4iIiHhK4UNEREQ8pfAhIiIinlL4EBEREU8pfIiIiIinFD5ERETEUwofIiIi4imFDxEREfGUwoeIiIh4SuFDREREPKXwISIiIp5S+BARERFPKXyIiIiIpxQ+RERExFMKHyIiIuIphQ8RERHxlMKHiIiIeErhQ0RERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8dSIwseyZcs488wzKSkpoaqqiosvvpimpqYB+/T19bFo0SLGjx9PcXExl156KS0tLaNatIiIiOSuEYWPVatWsWjRIl588UVWrFhBPB7nvPPOo7u7O73P9ddfzx//+EceeeQRVq1axfbt27nkkktGvXARERHJTYGR7Pzkk08O+PyBBx6gqqqKtWvXcs4559De3s7Pf/5zHnroIc4991wA7r//fmbMmMGLL77IBz7wgdGrXERERHLSEfX5aG9vB6CiogKAtWvXEo/HmT9/fnqf6dOnM3nyZBobG4d8jmg0SkdHx4CbiIiI5K/DDh+u63Lddddx9tlnc/LJJwPQ3NxMKBSivLx8wL7V1dU0NzcP+TzLli2jrKwsfauvrz/ckkRERCQHHHb4WLRoEa+99hoPP/zwERWwdOlS2tvb07ctW7Yc0fOJiIhIdhtRn4+UxYsX88QTT/Dcc88xadKk9PaamhpisRhtbW0DWj9aWlqoqakZ8rnC4TDhcPhwyhAREZEcNKKWD2MMixcv5tFHH+Xpp5+moaFhwP2zZ88mGAyycuXK9LampiY2b97M3LlzR6diERERyWkjavlYtGgRDz30EI8//jglJSXpfhxlZWUUFBRQVlbGVVddxZIlS6ioqKC0tJRrrrmGuXPnaqSLiIiIACMMH/fddx8AH/3oRwdsv//++/nc5z4HwN13343P5+PSSy8lGo2yYMECfvzjH49KsSIiIpL7RhQ+jDEH3ScSibB8+XKWL19+2EWJiIhI/tLaLiIiIuIphQ8RERHxlMKHiIiIeErhQ0RERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8ZTCh4iIiHhK4UNEREQ8pfAhIiIinlL4EBEREU8pfIiIiIinFD5ERETEUwofIiIi4qlApgsQGQ3RaAE9PeMwxsl0KVmnqKgg0yVkKUNhYYyCglimC8k6iYTLrl2JTJcheUzhQ/JCS8s01q37OIlEKNOlZJ2zz36LhoZ/4jgm06VkFccxTJ++nenTt2e6lKwTjTq0t4fZuTOY6VIkTyl85CDXdYnH4xijXyYpXV0B2tpq+8NHDHCBJHC0fY8cwN//MQj46e1tAezrJpHQX7MpPp8hEumloqIbRw1mA/T2OoTDfowJkEwm9brZj+M4BINBfD71WjgSCh85qK+vj1deeYWOjo5Ml5I1urvHk0wmgTZgLdAO7AB6MllWBgSAWqAEmAlMSd/T0dHB5s2b+79P4vcbTjqpO9NlZL333nuPd999N9NlZI1wOMzMmTOpqKjIdCk5TeEjB8XjcZqbm9m9e3emS8kiu7GtHX3AJmAn8DbQ2X9/vv9pm2rhCQHHAeOAqQP2iMVi7N69W3/F9vP7obd34DZj7O1o5DgM2QLU3t7O5s2bvS8oSxUWFnL88cdnuoycp/AheSYKbMe2emzGho8KbEtAPosBu7AD2EJAN9CV0YpyUUcHvPMOxOOZrsRbZWVwzDEQUpcp8YjCh+SZOLbVIxVAOoEIR0f42IltASnq//xou+R05Lq64M03B7eI5Lu6Opg8WeFDvKPwIYfE54NIBMJhqK+H4mKYOBHKywfvm0xCNAqxGOzaBT09sH077N1r/6KMeTKy0Y+99FAMnA5M9+KLZlAL9pJTD7blQ0Qkeyl8yCEJBqG0FCoqYN48G0DOPReGuvQZi0FrK3R2wpo10NwMq1bB66/bbd6FjxrsiI/zgY958UUzaAPQjG39yPf+LSKS6xQ+5IAiEdvKUV4Oxx0H48fDtGlQUwPjxkHBEPNXBQKQSNiP9fVQWGjDSEEBbNwIGzbY+8e236PTf/NhX+ZBYAvwLvk1/LYaOB57fD40abGI5AKFDzmgqio46SQbPC65xLZ81NbaIBEOD/2YQMCGlVRLSTIJZ54J7e3w+9/Dgw/aSzFtbV6PLPgf4HYgn0Z7XAIsy3QRIiIjovAhQ/L57LC70lLbt2PiRNsprbzc9ow/UMc0x7HDGP1+e7nGGHBd2wIycaJtDdmzx44s8HbKiU5gK3bysXzRih1iLCKSOxQ+ZEiFhbZ149RT4Yor7OWWujobOgKH8aopLrbP97/+FzQ0QGMj3H23HV0gIiJHF4UPGVIoZANDZSVMnQolJTY8+P0jfy7HsYHF77d9RQoKbCfUSAT6+sa674ccCj+2t4jLvnaUfOoZI9nOsf85/v7u0s7+W9N7mf7/m/Sr0+Aa+6o1++0h2U/hQwZxHDuK5eSTYdYs2+oRDg89++FIRSL2kk5DA3z0o7BjB/zjH2oBySQHOAE4BjtN2RbsoN02dEFHxl7ACVPgH0fIV8SEyHGEfcUEfRH8TpCIv5SIrzS9b9LE6E60kjRx4qaXpImzJ/oeXYmdRJOd9LlaciJXKHzIII4D1dUwYwZMmgRFRYMvtYyko2gqtDiO7QMSDNoWlRNOsK0gb76p8JFJDnZQ8knARqAD2xLSgcKHjD2fEyDiL6PQX051ZAZFgfFE+gNIcaCK4sAEnP43kZjbQ1tsK3G3lz63k7jbR9IkSJoYrknQ53ai1o/coPAhgziOne3wrLPsyJahFm90XXj3Xdtx9N137RDalIoKO7qlrMz2EykqGvz44mLbuuL3a1bFbDAOmISdG3UCdmL2HdgWkNb+jy3sWylH5HCFfEUEnQLKgnVURo4l5CuiOFBJyFdIeaiekK+QgBPG7wQI+QoHPNZPgEJ/Ba4/TsQtxzVxAk6I6vAJ9Cbb6Um20p3Yzc6+t4ibPvqSHZi86mCeP44ofNx+++0sXbqUa6+9lnvuuQewK65+/etf5+GHHyYajbJgwQJ+/OMfU11dPRr1igccx14W+eAHh7/UkkjAW2/ZVosVK+wtZdo0uOYamDLFjo4ZKnyUlsL06fb5I5ExOQw5RA42fEzpvxnsPKnbsSGkCXsJpgeFDzlyYV8xhf7xTCo8jRml5xPyFxLxl+LDx8EmyPM5QYoC4wdsqwwfCxhibg/RZBctfW8STXbRk2wj7vaQMAof2eiww8eaNWv46U9/yimnnDJg+/XXX8+f/vQnHnnkEcrKyli8eDGXXHIJf/vb3464WPFOaoXL4cKH68KmTba/xo4d9vOUri47m2l3t50jZMKEwY8PhWwLSXn54XVilbGRmpotgF0NJwhMBsZjL8U0YC/HdGKX8OvCDlzuRY3dMjwHH4X+CoK+AirDxzIuVM/4cAMhv23l8OHDcQ4+QZ4zzBuSMeB3ggR9BRQHq5hYeCo9yVb8vQH63E56Eq0kTN9oH5YcgcMKH11dXSxcuJD/+q//4rbbbktvb29v5+c//zkPPfQQ5557LgD3338/M2bM4MUXX+QDH/jAoOeKRqNEo9H05x0d6jCUC+JxO1z2//7fwaNV9uyBP/3JzufxkY/YVpT3Kyy0o2g6O3XZJRuFgar+f0/EBosPY4PGhv7bLmwfkW7sxO4atCTD8TkBJkSOoyxYR0PRXOoKT8FHAL8TYLSWA/A7Ifz+IFX+46kMN9CV2EOTfwUd8Wa29KyjM9E8Kl9HRsdhzcW8aNEiLrzwQubPnz9g+9q1a4nH4wO2T58+ncmTJ9PY2Djkcy1btoyysrL0rb6+/nBKkgyIx4ceKuu6dmG5vr6BLSL78/n2Db8djVE0MrocbEuHH7tMXRgowPYJKccGk1qgvv82qf/zsv591JlMABz8RHylFPorKA3UUBacSEFgHEGngIAvhOP4cBxn2BaNQ/46/c/hOD78ToCgr4Cwr5iSYA1lwTqKA+Mp8JcTcIaZllk8N+L3iIcffph169axZs2aQfc1NzcTCoUof99Sp9XV1TQ3D506ly5dypIlS9Kfd3R0KICIZLE6oBLbChLDXnJJ9Q95DdtBdQuwJ1MFStYI+QqoKziF4uAEji/5GBWhqQR9QywINQYi/mKOKf4gsWQ3PifAnthGdva9SWtskydfXw5sROFjy5YtXHvttaxYsYLIKPUSDIfDhIdbJESyluPYSydlZbaFI3XlzOezQ2mLiuyIluH6c7iunVo9kfB6fRc5Eg62FWT/n9hUf49O7IgYHzaIxLEBxe3/mOjfT6f7aODgw0fAiVAUqKQkUEVhYDyFgXGeVeBzAhT4ywg4IYoDE4i5PbT7tuHD3z9NmQaSZ9KIwsfatWvZuXMnp59+enpbMpnkueee40c/+hF/+ctfiMVitLW1DWj9aGlpoaamZtSKlswLBuG88+yMpX/9Kzz/vJ2IrLjYDs9dsMDOETLcIKfubti+HTZvhljM29pldIWwl2EqsJdkYtjWjy5gNzaQtAJvs6+lRG/7+S3oRCgKjKc0WMdxJR+mNFg7aJSKV/xOiEmFpzIhchyuidOX7CDqdtGTbM1IPWKNKHzMmzePV199dcC2z3/+80yfPp0bbriB+vp6gsEgK1eu5NJLLwWgqamJzZs3M3fu3NGrWsac69rbcKNe/H7bkdTvt/N8hEJ2yGxxsZ0R9fjj7SJyhYVDP38sZle19X5xORltfiB1mkuxLRvV2LlBtmFbSYLYeUMcbAABtYLkM58TJOwrpchfQXlwEmWhugzW4qckWEWRGU9xoIqIvwyXJCQd9ArMnBGFj5KSEk4++eQB24qKihg/fnx6+1VXXcWSJUuoqKigtLSUa665hrlz5w450kWykzHwzjuwapUNENOmDR0+pkyxw2WLi+Hss20H0nDYzuFxwgl2+1BzfIANHa+9ZoNLn0bA5Z0QNpTUYYfsNgDTsHOFbMWOkNmCHbbbhW0NkfxREqzihNJ5lASqCPuLM10OYNeJqY5MJ+CE2Nq7no54syYgy6BR75R+99134/P5uPTSSwdMMia5wxjYtg1eftm2fhx77OBZTn0+e0mluhqOO2745xquE3t3t50VVZdd8o+DbekIAhFs59SUHuykZXuxl1782L4gCh/5pdA/jvrC2RT6ywn6hmn+9JxDRXgKJcEqet12nE6fwkcGHXH4ePbZZwd8HolEWL58OcuXLz/Sp5YMSYWPf/zDtmLMnm0vqYTDA0PISEfHGWOH50aj0NICr75qJyhTy0f+Ge6lEcBO316EbfDuwAaR1MRle9i3qF0C22lVDeO5I+BECPoKKPCX2WnSfZEBq9Jmmt8Jgg/CvhIKA+OIu71E3S70KvOehuPLIKnLLjt3wrhxMG8elJSMzmRg0Si0t9vZUZ9/3vb7GG4uEMk/IeykZQDHYd/yU5ddNrFvqO6G/m3J/pvkhpCviJJAFUWBSiL+kkFrs2RawAkTcMIUBsZREqihL9lG3O3F1RR5nlP4kCHFYtDTA1u3wpo1dlTLrFl2FdpgcOjF5oZjjB1Sm0zCli12PZi33rJfQ8FjfwHshYgS7Gorqb8Y49hxI7H+f+fuNy01fXuKwXZITa0vMwk7UVkQGz5a+j/2su/o4x7WKyMT9hVREqyh0F+RnkAsW+xfS8hXSGmwBp/joyPRrIaPDFD4kCH19EBvLzz3HGzYAKedBt/4BlRV2daQkbaCdHXZ53zqKXjgAdi71z6/7K8Qe0HiRGAuNoiAvSixEvuruB17YSJ/pOYNKcTOlmqwl1y6gZexLSEb+z+2Yi/TSHYqDdUytWiO/cWexb9eSgLVTCk6k93Rd9kV3UDSqOOZ17L31SEZZYy9dXfbyy87dthLJX19tgWjsNDeDhRCkkm7dkuqj8fevbaDaUuLDSJq9UjxYYPGeOwg1SnY5dz2Dx9TsN0338P+anbJ5RaQlP1bQnzYFo+UILazqh/b8hFiX9tQanbVJHaBO/3hmh38TpCQv4igL5JVrR7v53cChHz9dR7eKiNyhBQ+5ICiUWhttZ1P77wTKivhnHPsRGJnnWUXhxtOTw+sWGFDy+rV9lJLayvs3m2Dh2Y2BfurtwT7d//FwAJsCJnIvl/LUWA29u/+XwAvYntK9Hhcq7fCwAxs1JrFvotPe7CtH1uwHVPfJN/agnJXwAlT4Csh6CvIqo6m7+d3QhT4Swn7CrO6znym8CEHlJpsrK0Nmppg1y47c2k8Du+b8mWQRMKOmnn7bfjnP+GNNzwpOcc42CXbirEXHU5i3/JtqTfFRP8+bdi5RIuxf/fnd/jwYyctSzHYiFbc/zHWv09qKhm3f58kmsAsU3z4CfjCdlRJFv9S9zl+Ak7I1pnFLTT5TOFDDkkiYUep9PXB//t/dhbTmTPhxBOHf0w0CmvX2laPnTu9qzW3hLB/1zcA07HdLoPv28eH/TUcwLaAOMBaYJ13ZWaJEux3rBy7im4PtodManG7buyomVT3XF3J91bYX0xZsI6AE87qFoWQr5DSUB2diZ34dNklIxQ+5JC4rg0efX02hJSU2D4cB5JI2NEtGzZ4U2Nu8mPHeJyAbdUoZPBfjD5sfw8H2zrShe37cXTZf1G7Uux3K46dRbWXfZOXdWJ7ybgofHgt4ISJ+MvwOcOsKJklAr4wAcKEfEXq85EhCh8iIlkgELDLEYRCdlTZ4S723dJiL3eKZDOFDxGRLBAI2BmFS0pgxgz778Px6qt2xWh16JZspvAhklFJ7LiNAHA8thdDANuzIXX5xcWOeOkFNmMvMOz2vNJMM9jvQurWif1ube//uAPb56MVe7kll+esHGol6UOR6cCRMH30JtsIOBFCvgIcJzsvaSTcaHpqdZMHQ9ZzkcKHSEbFgH9gp9E6ATgFO35j/wlUXOyv2jZsJ9OVHK1TbXVhw8Uu4F3ssNuXsaEjNdolP2ZAyU3RZDcd8R1E/GUEfXVZ2+U05nbTEW+hO7EH1+jVkgkKHyIZZbCzVDjYVo1XsVNr1UG6I1wUO4ajFTvLaRdHQ1fKJDZUxLHfgdQ8H7ux0WsnNo719N+f61zXLjnQ2wt79hz+govd3aNb10i4JEi4UZK+7F4S0DUuCRMlabK7znym8CGSUQbbqtENPAY0YqfWev/06k8DzewLIfn/11oUeAMbNFJHnZpkLIkNI6kZTvNBPG5DR2urHZo+kvWT9heLZe7yS8LE6HU78bsRTBb/Uk+aGH3JTmJuT1bXmc8UPmRIpaVQXDz8/cXFdpE5GQ2pCwWt2F+lRdgZTvcPH5uxrR6d5NM6r2a/W5J9E8en1nbZzb72ntTMpvl6wSm1ACPYIJKLkiZONNlFxFeMMSZr5xlLmjgxt5u426c+Hxmi8CGD+P3wmc/AJZcMv08gANOne1fT0SF1AeEf2L/195/hdE//fbncjXJoUexFpB3Yni+d2C64qVVt+/r/HedouNiU29pj29nUvZqayAxqCk5kYN+l7NGZ2Mmm7pfoSuzqv/QiXlP4kEEcB6ZNg3PP1czD3kr036LYsJF/9p/2PNVBNDWOZy+wtf/jhv5tXeRTO0/+i7k9dMRbKA3WYIxrWz8gKxaZM/tdi0rV2Zdsw6jDaUYofIiIZ2LY1owebOtGOzZsdGBbPFqxLR0d7LsEI7kj5nbTmWimKzGRvmQn4PSvHJsdM57aTqYxehKtdCaaibs9uIq3GaHwIQd0sI5rWfAHjWSh4V42CfaNVlmP7UK7i/ztx3G0SZg+Esk++pLtxNweAk6IgO8wp2odA0kTJ5a083v0JPbioksumaLwIYO4LqxadeB9QiGYNw9OOMGbmiR3GPaNROnCtmJ0Y4fG9mAvrfQA2/rvO8wRpZLFehKtbO5eQ0mwmslFs/H7D3O61lFlaI2+x+7oO+yJvquOphmm8CGDuC784Q/wxBPD71NSApWVCh8ytBg2VGzHdp3dgZ0MrJd943W07H3+6kzs4q3OZxgXmkx15AQiWRA+DIaWvibe6nyanuQehY8MU/iQIbmuvQ0nkTjw/XL0SGKDRgLbhyOK7bvRhe0229L/eQ82lCRRX45855o4fW4HPclW9sa34JKgKFBJyFeYgVqSdCd2E3W76Ey00JdsJ+72oeibWQofInJEYth+G53YQcK7gfewl1lc9oWNBGrtOFrETR8d8R0k3F7e6fwrJcEajis5h4rQZM9rSZoYW3vW0xrbxPbeV2mPb9PEYllA4UNEDpnBho0oNlTEsJdStmPDx05sK0db/+dytDIYkiRMlO7EbsChO7GbsK+YkK+QgBMe8+G3rkkQTXYTdbvoSuyiM95C1O3S6JYsofAhIiOyHduy0YodLtuN7dMRxV5aSaDJwMSKuT1s632l/3KLoTxUz6TCU6kKHz/mX7sv2cXG7hfojO/kve5G2uPb+i+3SDZQ+BCRQVKrw+7/Mdl/Sy3qths7cqUbO2Q2/+ZelSNlcIm6nSRMlM5EC47jpyIxhXiwF5/jx+8E+/d0jqglZN8EYgaXpJ3m3e2kM95CR3wH3Yk99Cbbj/h4ZPQofIjIIFFsyOjDDontxk593oq9nNLZv083NnSoIVsOxDUJdvVtoC22lZ7EHrb1rKcychz1hacTcEL9LSNHdhkmYWIk3D7a4lt5r3s1PYm9NPe9TtTtVPDIQgofIpKWGoUSxwaMLuyllb3YzqTbM1SX5DaDS3dyDyTtRF+d8V34nADV4engMwR8EXwGhgsgqVYRM+Ssh3aba+LE3V664rvY3vMKPcm9tMe3kTD5su5xflH4EDnKGWyLRmrZ+mb29ePoY9+Sdh2ZKlDyStTtwjVJtvSsozfZRshXSHGgipCvgHHhyYR8RQScED4n0N9BtSj9WNfE6U22kzSJ/qnS47THt9OTaKU32U5PspXuxB7a4ttImD6SRhcDs5XCh4jQhu2/sRF4FdvisQP145DRF3O7idFNd3I3O/peJeQroiw4iUJ/OVPNXIoC44n4iu2aMAHfgPCRJEFPspW420ef20nc7WNLzzr2RN+lO7GbzkRLBo9MRkLhQwZxHDjjDJgxY/i1WyIRaGjwti4ZGwYbNF7FdiJtw7Z0aCIw8ULSJOhLtuOaOM29rxP2FxN0IvidABF/6YDZURP9i8Il+y+xJE2cttgWepN7ibu9GTwKGSmFDxnE74f/839g0aLh93Ecu76L5D4DNGGXsU+NbkltFxlrydRIGBz2xjbvN+rFwRmiD8i+CcLsv4xx+6dK1ys2lyh8yJBCISgs1Kq1RwsXtXRIJpn+/1xliKOEb6QP2LZtG5/5zGcYP348BQUFzJw5k7///e/p+40x3HTTTdTW1lJQUMD8+fPZsGHDqBYtIiIiuWtELR979+7l7LPP5mMf+xh//vOfmTBhAhs2bGDcuHHpfe644w7uvfdeHnzwQRoaGrjxxhtZsGABr7/+OpFIZNQPQMbG3r2wefORtXzs2AF9WTWhYClQT37NSlHJYfwNISKSUSMKHz/4wQ+or6/n/vvvT29r2K/XoTGGe+65h+9+97tcdNFFAPzyl7+kurqaxx57jMsvv3yUypaxlEzCr34FzzxzZM8Ti8Gbb45OTaPjfwMzMl3EKKsCFOpFJLeMKHz84Q9/YMGCBXzqU59i1apVTJw4ka9+9at88YtfBGDjxo00Nzczf/789GPKysqYM2cOjY2NQ4aPaDRKNLpvEpiODs0mkGnGwLvv2lvuSq2f6mKnzIphf1FXZbKoMZJaTSW1fqw66ohIdhtR+Hj33Xe57777WLJkCd/5zndYs2YNX/va1wiFQlx55ZU0NzcDUF1dPeBx1dXV6fveb9myZdxyyy2HWb7IcJJACzZ4PImdwSKf7QTewC7tVg1oKJKIZK8RhQ/XdTnjjDP4/ve/D8Bpp53Ga6+9xk9+8hOuvPLKwypg6dKlLFmyJP15R0cH9fX1h/VcIvsk2bew+8vk/8Tgndhpwgww7iD7iohk1ojCR21tLSeeeOKAbTNmzOC///u/AaipqQGgpaWF2tra9D4tLS2ceuqpQz5nOBwmHA6PpAyRAwhi//JPYlsBOoGiAz4iP4Swx+0D6oDxQGFGK8pFxcVw4om2v9LRpLQUApp4QTw0opfb2WefTVNT04Btb731FlOmTAFs59OamhpWrlyZDhsdHR2sXr2ar3zlK6NTscgBhYFa7Es7xtGzIkkIGzpCwGSgAijOaEW5qLQUTjkl01V4T/P5iNdGFD6uv/56PvjBD/L973+fT3/607z00kv87Gc/42c/+xlgVx687rrruO2225g2bVp6qG1dXR0XX3zxWNQv8j4RYCr20kMJcLRNuRzAhq9i7NBiGQnH0S9iES+MKHyceeaZPProoyxdupRbb72VhoYG7rnnHhYuXJje51vf+hbd3d1cffXVtLW18aEPfYgnn3xSc3yIR8qBD2H7PiQ5+qZLdLCXXnxoAmMRyVYjfnf6+Mc/zsc//vFh73cch1tvvZVbb731iAqT4fn9fkpLS0km82myrCPlwy6Ppn4O71dY2A5AIBCguLiYREJr1YJdwygehz17Ml1J9olGHaLRIAAFBQVUVFRgzNEW5IdWUFBAQB1kjpi+gzmooKCAM844Q+FjgCDw/6HZPgebNKkAxymktLSUk046Sb9E9rN7N/zlL5muIvskEi7NzbuALhoaGgYMIDjaOY5DUdHR0Il9bCl85KBUy4e839BzyRztCgsnAIUEAgH9xfY+8Ti0tma6iuyTTCaJRvfiOA4FBQUUFBRkuiTJM/ozUURERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8ZTCh4iIiHhK4UNEREQ8pfAhIiIinlL4EBEREU8pfIiIiIinFD5ERETEUwofIiIi4imFDxEREfGUwoeIiIh4SuFDREREPKXwISIiIp5S+BARERFPKXyIiIiIpxQ+RERExFMKHyIiIuIphQ8RERHxlMKHiIiIeErhQ0RERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8dSIwkcymeTGG2+koaGBgoICjj32WP7jP/4DY0x6H2MMN910E7W1tRQUFDB//nw2bNgw6oWLiIhIbhpR+PjBD37Afffdx49+9CPeeOMNfvCDH3DHHXfwwx/+ML3PHXfcwb333stPfvITVq9eTVFREQsWLKCvr2/UixcREZHcExjJzi+88AIXXXQRF154IQBTp07lN7/5DS+99BJgWz3uuecevvvd73LRRRcB8Mtf/pLq6moee+wxLr/88lEuX0RERHLNiFo+PvjBD7Jy5UreeustAP7xj3/w/PPPc8EFFwCwceNGmpubmT9/fvoxZWVlzJkzh8bGxiGfMxqN0tHRMeAmIiIi+WtELR/f/va36ejoYPr06fj9fpLJJN/73vdYuHAhAM3NzQBUV1cPeFx1dXX6vvdbtmwZt9xyy+HULiIiIjloRC0fv/vd7/j1r3/NQw89xLp163jwwQf5z//8Tx588MHDLmDp0qW0t7enb1u2bDns5xIREZHsN6KWj29+85t8+9vfTvfdmDlzJps2bWLZsmVceeWV1NTUANDS0kJtbW36cS0tLZx66qlDPmc4HCYcDh9m+SIiIpJrRtTy0dPTg8838CF+vx/XdQFoaGigpqaGlStXpu/v6Ohg9erVzJ07dxTKFRERkVw3opaPT3ziE3zve99j8uTJnHTSSbz88svcddddfOELXwDAcRyuu+46brvtNqZNm0ZDQwM33ngjdXV1XHzxxWNRv4iIiOSYEYWPH/7wh9x444189atfZefOndTV1fGlL32Jm266Kb3Pt771Lbq7u7n66qtpa2vjQx/6EE8++SSRSGTUixcREZHc45j9pyfNAh0dHZSVlfHZz36WUCiU6XJEct6ECROoqanBcZxMlyI5IplMsnXrVtrb2zNdiuSQWCzGL3/5S9rb2yktLT3gvlrbRURERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8ZTCh4iIiHhK4UNEREQ8pfAhIiIinlL4EBEREU8pfIiIiIinFD5ERETEUwofIiIi4imFDxEREfGUwoeIiIh4SuFDREREPKXwISIiIp5S+BARERFPKXyIiIiIpxQ+RERExFMKHyIiIuIphQ8RERHxlMKHiIiIeErhQ0RERDyl8CEiIiKeCmS6gPczxgAQi8UyXIlIfohGo/T19WW6DMkhrusSjUb1Piwjknq9pH6PH4hjDmUvD23dupX6+vpMlyEiIiKHYcuWLUyaNOmA+2Rd+HBdl6amJk488US2bNlCaWlppksadR0dHdTX1+fl8eXzsUF+H5+OLXfl8/Hp2HKHMYbOzk7q6urw+Q7cqyPrLrv4fD4mTpwIQGlpaV6ckOHk8/Hl87FBfh+fji135fPx6dhyQ1lZ2SHtpw6nIiIi4imFDxEREfFUVoaPcDjMzTffTDgcznQpYyKfjy+fjw3y+/h0bLkrn49Px5afsq7DqYiIiOS3rGz5EBERkfyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTWRk+li9fztSpU4lEIsyZM4eXXnop0yWN2LJlyzjzzDMpKSmhqqqKiy++mKampgH7fPSjH8VxnAG3L3/5yxmq+ND9+7//+6C6p0+fnr6/r6+PRYsWMX78eIqLi7n00ktpaWnJYMUjM3Xq1EHH5zgOixYtAnLrvD333HN84hOfoK6uDsdxeOyxxwbcb4zhpptuora2loKCAubPn8+GDRsG7NPa2srChQspLS2lvLycq666iq6uLg+PYngHOr54PM4NN9zAzJkzKSoqoq6ujs9+9rNs3759wHMMdb5vv/12j49ksIOdu8997nOD6j7//PMH7JOt5+5gxzbUz5/jONx5553pfbL1vB3Ke/+hvEdu3ryZCy+8kMLCQqqqqvjmN79JIpHw8lDGVNaFj9/+9rcsWbKEm2++mXXr1jFr1iwWLFjAzp07M13aiKxatYpFixbx4osvsmLFCuLxOOeddx7d3d0D9vviF7/Ijh070rc77rgjQxWPzEknnTSg7ueffz593/XXX88f//hHHnnkEVatWsX27du55JJLMljtyKxZs2bAsa1YsQKAT33qU+l9cuW8dXd3M2vWLJYvXz7k/XfccQf33nsvP/nJT1i9ejVFRUUsWLBgwCq4Cxcu5J///CcrVqzgiSee4LnnnuPqq6/26hAO6EDH19PTw7p167jxxhtZt24dv//972lqauKTn/zkoH1vvfXWAefzmmuu8aL8AzrYuQM4//zzB9T9m9/8ZsD92XruDnZs+x/Tjh07+MUvfoHjOFx66aUD9svG83Yo7/0He49MJpNceOGFxGIxXnjhBR588EEeeOABbrrppkwc0tgwWeass84yixYtSn+eTCZNXV2dWbZsWQarOnI7d+40gFm1alV620c+8hFz7bXXZq6ow3TzzTebWbNmDXlfW1ubCQaD5pFHHklve+ONNwxgGhsbPapwdF177bXm2GOPNa7rGmNy97wB5tFHH01/7rquqampMXfeeWd6W1tbmwmHw+Y3v/mNMcaY119/3QBmzZo16X3+/Oc/G8dxzLZt2zyr/VC8//iG8tJLLxnAbNq0Kb1typQp5u677x7b4o7QUMd25ZVXmosuumjYx+TKuTuU83bRRReZc889d8C2XDhvxgx+7z+U98j/+Z//MT6fzzQ3N6f3ue+++0xpaamJRqPeHsAYyaqWj1gsxtq1a5k/f356m8/nY/78+TQ2NmawsiPX3t4OQEVFxYDtv/71r6msrOTkk09m6dKl9PT0ZKK8EduwYQN1dXUcc8wxLFy4kM2bNwOwdu1a4vH4gHM4ffp0Jk+enJPnMBaL8atf/YovfOELOI6T3p6r521/GzdupLm5ecC5KisrY86cOelz1djYSHl5OWeccUZ6n/nz5+Pz+Vi9erXnNR+p9vZ2HMehvLx8wPbbb7+d8ePHc9ppp3HnnXfmTPP2s88+S1VVFSeccAJf+cpX2LNnT/q+fDl3LS0t/OlPf+Kqq64adF8unLf3v/cfyntkY2MjM2fOpLq6Or3PggUL6Ojo4J///KeH1Y+drFrVdvfu3SSTyQHfcIDq6mrefPPNDFV15FzX5brrruPss8/m5JNPTm//13/9V6ZMmUJdXR2vvPIKN9xwA01NTfz+97/PYLUHN2fOHB544AFOOOEEduzYwS233MKHP/xhXnvtNZqbmwmFQoPe3Kurq2lubs5MwUfgscceo62tjc997nPpbbl63t4vdT6G+nlL3dfc3ExVVdWA+wOBABUVFTl3Pvv6+rjhhhu44oorBqwg+rWvfY3TTz+diooKXnjhBZYuXcqOHTu46667MljtwZ1//vlccsklNDQ08M477/Cd73yHCy64gMbGRvx+f96cuwcffJCSkpJBl25z4bwN9d5/KO+Rzc3NQ/5cpu7LB1kVPvLVokWLeO211wb0iwAGXHudOXMmtbW1zJs3j3feeYdjjz3W6zIP2QUXXJD+9ymnnMKcOXOYMmUKv/vd7ygoKMhgZaPv5z//ORdccAF1dXXpbbl63o5m8XicT3/60xhjuO+++wbct2TJkvS/TznlFEKhEF/60pdYtmxZVq+5cfnll6f/PXPmTE455RSOPfZYnn32WebNm5fBykbXL37xCxYuXEgkEhmwPRfO23Dv/ZJlHU4rKyvx+/2Dev22tLRQU1OToaqOzOLFi3niiSd45plnmDRp0gH3nTNnDgBvv/22F6WNmvLyco4//njefvttampqiMVitLW1DdgnF8/hpk2beOqpp/i3f/u3A+6Xq+ctdT4O9PNWU1MzqLN3IpGgtbU1Z85nKnhs2rSJFStWDGj1GMqcOXNIJBK899573hQ4So455hgqKyvTr8N8OHd//etfaWpqOujPIGTfeRvuvf9Q3iNramqG/LlM3ZcPsip8hEIhZs+ezcqVK9PbXNdl5cqVzJ07N4OVjZwxhsWLF/Poo4/y9NNP09DQcNDHrF+/HoDa2toxrm50dXV18c4771BbW8vs2bMJBoMDzmFTUxObN2/OuXN4//33U1VVxYUXXnjA/XL1vDU0NFBTUzPgXHV0dLB69er0uZo7dy5tbW2sXbs2vc/TTz+N67rp0JXNUsFjw4YNPPXUU4wfP/6gj1m/fj0+n2/QJYtst3XrVvbs2ZN+Heb6uQPb8jh79mxmzZp10H2z5bwd7L3/UN4j586dy6uvvjogPKaC84knnujNgYy1DHd4HeThhx824XDYPPDAA+b11183V199tSkvLx/Q6zcXfOUrXzFlZWXm2WefNTt27Ejfenp6jDHGvP322+bWW281f//7383GjRvN448/bo455hhzzjnnZLjyg/v6179unn32WbNx40bzt7/9zcyfP99UVlaanTt3GmOM+fKXv2wmT55snn76afP3v//dzJ0718ydOzfDVY9MMpk0kydPNjfccMOA7bl23jo7O83LL79sXn75ZQOYu+66y7z88svp0R633367KS8vN48//rh55ZVXzEUXXWQaGhpMb29v+jnOP/98c9ppp5nVq1eb559/3kybNs1cccUVmTqkAQ50fLFYzHzyk580kyZNMuvXrx/wc5gaMfDCCy+Yu+++26xfv96888475le/+pWZMGGC+exnP5vhIzvwsXV2dppvfOMbprGx0WzcuNE89dRT5vTTTzfTpk0zfX196efI1nN3sNelMca0t7ebwsJCc9999w16fDaft4O99xtz8PfIRCJhTj75ZHPeeeeZ9evXmyeffNJMmDDBLF26NBOHNCayLnwYY8wPf/hDM3nyZBMKhcxZZ51lXnzxxUyXNGLAkLf777/fGGPM5s2bzTnnnGMqKipMOBw2xx13nPnmN79p2tvbM1v4IbjssstMbW2tCYVCZuLEieayyy4zb7/9dvr+3t5e89WvftWMGzfOFBYWmn/5l38xO3bsyGDFI/eXv/zFAKapqWnA9lw7b88888yQr8Mrr7zSGGOH2954442murrahMNhM2/evEHHvGfPHnPFFVeY4uJiU1paaj7/+c+bzs7ODBzNYAc6vo0bNw77c/jMM88YY4xZu3atmTNnjikrKzORSMTMmDHDfP/73x/wCzxTDnRsPT095rzzzjMTJkwwwWDQTJkyxXzxi18c9Edatp67g70ujTHmpz/9qSkoKDBtbW2DHp/N5+1g7/3GHNp75HvvvWcuuOACU1BQYCorK83Xv/51E4/HPT6aseMYY8wYNaqIiIiIDJJVfT5EREQk/yl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8ZTCh4iIiHhK4UNEREQ8pfAhIiIinlL4EBEREU8pfIiIiIinFD5ERETEU/8/6kMDPbe6OYgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "t, state = np.unravel_index(\n",
    "    np.argmin(np.where(vi_results.random_values > 0, vi_results.random_values, np.inf)),\n",
    "    vi_results.random_values.shape,\n",
    ")\n",
    "print(t, state, vi_results.random_values[t, state])\n",
    "\n",
    "mdp = np.load(mdp_file)\n",
    "plt.imshow(mdp[\"screens\"][mdp[\"screen_mapping\"][state]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this state on the 85th timestep, there is less than a $10^{-11}$ chance of getting the optimal reward!\n",
    "\n",
    "We can even load the state of the MDP from the tabular version into the Gym environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "nbreg": {
     "diff_ignore": [
      "/outputs/0/data/text/plain",
      "/outputs/1/data/image/png"
     ]
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x78f5857da6a0>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAEECAYAAACIpFYUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAy5ElEQVR4nO3deXRddb3//+c+c+Y0TTO1aRug0AKlQIFaUVTaL4UvKlz4KnDrEpUrDi0CdcC6BC5ctAh3AQutqHcp4E8R5XsFFK/4LQWKSCi1pQICoUDpnHRIMydn2p/fH5+c04YkbdMm+wx9PViHNPvsc/Le2Scnr3z2Z3CMMQYRERERj/gyXYCIiIgcXRQ+RERExFMKHyIiIuIphQ8RERHxlMKHiIiIeErhQ0RERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTYxY+li9fztSpU4lEIsyZM4eXXnpprL6UiIiI5JAxCR+//e1vWbJkCTfffDPr1q1j1qxZLFiwgJ07d47FlxMREZEc4ozFwnJz5szhzDPP5Ec/+hEArutSX1/PNddcw7e//e0DPtZ1XbZv305JSQmO44x2aSIiIjIGjDF0dnZSV1eHz3fgto3AaH/xWCzG2rVrWbp0aXqbz+dj/vz5NDY2Dto/Go0SjUbTn2/bto0TTzxxtMsSERERD2zZsoVJkyYdcJ9RDx+7d+8mmUxSXV09YHt1dTVvvvnmoP2XLVvGLbfcMmj75ZdfTigUGu3yRI46lZWVg34eRQ7EdV22bdtGR0dHpkuRHBKLxXj44YcpKSk56L6jHj5GaunSpSxZsiT9eUdHB/X19YRCIYUPkVEQDoeJRCK6jCmHLJlMEg6H9R4sh+VQ3mtGPXxUVlbi9/tpaWkZsL2lpYWamppB+4fDYcLh8GiXISIiIllq1Ee7hEIhZs+ezcqVK9PbXNdl5cqVzJ07d7S/nIiIiOSYMbnssmTJEq688krOOOMMzjrrLO655x66u7v5/Oc/PxZfTkRERHLImISPyy67jF27dnHTTTfR3NzMqaeeypNPPqlObyIiIjJ2HU4XL17M4sWLx+rpRUREJEdpbRcRERHxlMKHiIiIeErhQ0RERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8ZTCh4iIiHhK4UNEREQ8pfAhIiIinlL4EBEREU8pfIiIiIinFD5ERETEUwofIiIi4imFDxEREfGUwoeIiIh4SuFDREREPKXwISIiIp5S+BARERFPKXyIiIiIpxQ+RERExFMKHyIiIuIphQ8RERHxlMKHiIiIeErhQ0RERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8dSIwseyZcs488wzKSkpoaqqiosvvpimpqYB+/T19bFo0SLGjx9PcXExl156KS0tLaNatIiIiOSuEYWPVatWsWjRIl588UVWrFhBPB7nvPPOo7u7O73P9ddfzx//+EceeeQRVq1axfbt27nkkktGvXARERHJTYGR7Pzkk08O+PyBBx6gqqqKtWvXcs4559De3s7Pf/5zHnroIc4991wA7r//fmbMmMGLL77IBz7wgdGrXERERHLSEfX5aG9vB6CiogKAtWvXEo/HmT9/fnqf6dOnM3nyZBobG4d8jmg0SkdHx4CbiIiI5K/DDh+u63Lddddx9tlnc/LJJwPQ3NxMKBSivLx8wL7V1dU0NzcP+TzLli2jrKwsfauvrz/ckkRERCQHHHb4WLRoEa+99hoPP/zwERWwdOlS2tvb07ctW7Yc0fOJiIhIdhtRn4+UxYsX88QTT/Dcc88xadKk9PaamhpisRhtbW0DWj9aWlqoqakZ8rnC4TDhcPhwyhAREZEcNKKWD2MMixcv5tFHH+Xpp5+moaFhwP2zZ88mGAyycuXK9LampiY2b97M3LlzR6diERERyWkjavlYtGgRDz30EI8//jglJSXpfhxlZWUUFBRQVlbGVVddxZIlS6ioqKC0tJRrrrmGuXPnaqSLiIiIACMMH/fddx8AH/3oRwdsv//++/nc5z4HwN13343P5+PSSy8lGo2yYMECfvzjH49KsSIiIpL7RhQ+jDEH3ScSibB8+XKWL19+2EWJiIhI/tLaLiIiIuIphQ8RERHxlMKHiIiIeErhQ0RERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8ZTCh4iIiHhK4UNEREQ8pfAhIiIinlL4EBEREU8pfIiIiIinFD5ERETEUwofIiIi4qlApgsQGQ3RaAE9PeMwxsl0KVmnqKgg0yVkKUNhYYyCglimC8k6iYTLrl2JTJcheUzhQ/JCS8s01q37OIlEKNOlZJ2zz36LhoZ/4jgm06VkFccxTJ++nenTt2e6lKwTjTq0t4fZuTOY6VIkTyl85CDXdYnH4xijXyYpXV0B2tpq+8NHDHCBJHC0fY8cwN//MQj46e1tAezrJpHQX7MpPp8hEumloqIbRw1mA/T2OoTDfowJkEwm9brZj+M4BINBfD71WjgSCh85qK+vj1deeYWOjo5Ml5I1urvHk0wmgTZgLdAO7AB6MllWBgSAWqAEmAlMSd/T0dHB5s2b+79P4vcbTjqpO9NlZL333nuPd999N9NlZI1wOMzMmTOpqKjIdCk5TeEjB8XjcZqbm9m9e3emS8kiu7GtHX3AJmAn8DbQ2X9/vv9pm2rhCQHHAeOAqQP2iMVi7N69W3/F9vP7obd34DZj7O1o5DgM2QLU3t7O5s2bvS8oSxUWFnL88cdnuoycp/AheSYKbMe2emzGho8KbEtAPosBu7AD2EJAN9CV0YpyUUcHvPMOxOOZrsRbZWVwzDEQUpcp8YjCh+SZOLbVIxVAOoEIR0f42IltASnq//xou+R05Lq64M03B7eI5Lu6Opg8WeFDvKPwIYfE54NIBMJhqK+H4mKYOBHKywfvm0xCNAqxGOzaBT09sH077N1r/6KMeTKy0Y+99FAMnA5M9+KLZlAL9pJTD7blQ0Qkeyl8yCEJBqG0FCoqYN48G0DOPReGuvQZi0FrK3R2wpo10NwMq1bB66/bbd6FjxrsiI/zgY958UUzaAPQjG39yPf+LSKS6xQ+5IAiEdvKUV4Oxx0H48fDtGlQUwPjxkHBEPNXBQKQSNiP9fVQWGjDSEEBbNwIGzbY+8e236PTf/NhX+ZBYAvwLvk1/LYaOB57fD40abGI5AKFDzmgqio46SQbPC65xLZ81NbaIBEOD/2YQMCGlVRLSTIJZ54J7e3w+9/Dgw/aSzFtbV6PLPgf4HYgn0Z7XAIsy3QRIiIjovAhQ/L57LC70lLbt2PiRNsprbzc9ow/UMc0x7HDGP1+e7nGGHBd2wIycaJtDdmzx44s8HbKiU5gK3bysXzRih1iLCKSOxQ+ZEiFhbZ149RT4Yor7OWWujobOgKH8aopLrbP97/+FzQ0QGMj3H23HV0gIiJHF4UPGVIoZANDZSVMnQolJTY8+P0jfy7HsYHF77d9RQoKbCfUSAT6+sa674ccCj+2t4jLvnaUfOoZI9nOsf85/v7u0s7+W9N7mf7/m/Sr0+Aa+6o1++0h2U/hQwZxHDuK5eSTYdYs2+oRDg89++FIRSL2kk5DA3z0o7BjB/zjH2oBySQHOAE4BjtN2RbsoN02dEFHxl7ACVPgH0fIV8SEyHGEfcUEfRH8TpCIv5SIrzS9b9LE6E60kjRx4qaXpImzJ/oeXYmdRJOd9LlaciJXKHzIII4D1dUwYwZMmgRFRYMvtYyko2gqtDiO7QMSDNoWlRNOsK0gb76p8JFJDnZQ8knARqAD2xLSgcKHjD2fEyDiL6PQX051ZAZFgfFE+gNIcaCK4sAEnP43kZjbQ1tsK3G3lz63k7jbR9IkSJoYrknQ53ai1o/coPAhgziOne3wrLPsyJahFm90XXj3Xdtx9N137RDalIoKO7qlrMz2EykqGvz44mLbuuL3a1bFbDAOmISdG3UCdmL2HdgWkNb+jy3sWylH5HCFfEUEnQLKgnVURo4l5CuiOFBJyFdIeaiekK+QgBPG7wQI+QoHPNZPgEJ/Ba4/TsQtxzVxAk6I6vAJ9Cbb6Um20p3Yzc6+t4ibPvqSHZi86mCeP44ofNx+++0sXbqUa6+9lnvuuQewK65+/etf5+GHHyYajbJgwQJ+/OMfU11dPRr1igccx14W+eAHh7/UkkjAW2/ZVosVK+wtZdo0uOYamDLFjo4ZKnyUlsL06fb5I5ExOQw5RA42fEzpvxnsPKnbsSGkCXsJpgeFDzlyYV8xhf7xTCo8jRml5xPyFxLxl+LDx8EmyPM5QYoC4wdsqwwfCxhibg/RZBctfW8STXbRk2wj7vaQMAof2eiww8eaNWv46U9/yimnnDJg+/XXX8+f/vQnHnnkEcrKyli8eDGXXHIJf/vb3464WPFOaoXL4cKH68KmTba/xo4d9vOUri47m2l3t50jZMKEwY8PhWwLSXn54XVilbGRmpotgF0NJwhMBsZjL8U0YC/HdGKX8OvCDlzuRY3dMjwHH4X+CoK+AirDxzIuVM/4cAMhv23l8OHDcQ4+QZ4zzBuSMeB3ggR9BRQHq5hYeCo9yVb8vQH63E56Eq0kTN9oH5YcgcMKH11dXSxcuJD/+q//4rbbbktvb29v5+c//zkPPfQQ5557LgD3338/M2bM4MUXX+QDH/jAoOeKRqNEo9H05x0d6jCUC+JxO1z2//7fwaNV9uyBP/3JzufxkY/YVpT3Kyy0o2g6O3XZJRuFgar+f0/EBosPY4PGhv7bLmwfkW7sxO4atCTD8TkBJkSOoyxYR0PRXOoKT8FHAL8TYLSWA/A7Ifz+IFX+46kMN9CV2EOTfwUd8Wa29KyjM9E8Kl9HRsdhzcW8aNEiLrzwQubPnz9g+9q1a4nH4wO2T58+ncmTJ9PY2Djkcy1btoyysrL0rb6+/nBKkgyIx4ceKuu6dmG5vr6BLSL78/n2Db8djVE0MrocbEuHH7tMXRgowPYJKccGk1qgvv82qf/zsv591JlMABz8RHylFPorKA3UUBacSEFgHEGngIAvhOP4cBxn2BaNQ/46/c/hOD78ToCgr4Cwr5iSYA1lwTqKA+Mp8JcTcIaZllk8N+L3iIcffph169axZs2aQfc1NzcTCoUof99Sp9XV1TQ3D506ly5dypIlS9Kfd3R0KICIZLE6oBLbChLDXnJJ9Q95DdtBdQuwJ1MFStYI+QqoKziF4uAEji/5GBWhqQR9QywINQYi/mKOKf4gsWQ3PifAnthGdva9SWtskydfXw5sROFjy5YtXHvttaxYsYLIKPUSDIfDhIdbJESyluPYSydlZbaFI3XlzOezQ2mLiuyIluH6c7iunVo9kfB6fRc5Eg62FWT/n9hUf49O7IgYHzaIxLEBxe3/mOjfT6f7aODgw0fAiVAUqKQkUEVhYDyFgXGeVeBzAhT4ywg4IYoDE4i5PbT7tuHD3z9NmQaSZ9KIwsfatWvZuXMnp59+enpbMpnkueee40c/+hF/+ctfiMVitLW1DWj9aGlpoaamZtSKlswLBuG88+yMpX/9Kzz/vJ2IrLjYDs9dsMDOETLcIKfubti+HTZvhljM29pldIWwl2EqsJdkYtjWjy5gNzaQtAJvs6+lRG/7+S3oRCgKjKc0WMdxJR+mNFg7aJSKV/xOiEmFpzIhchyuidOX7CDqdtGTbM1IPWKNKHzMmzePV199dcC2z3/+80yfPp0bbriB+vp6gsEgK1eu5NJLLwWgqamJzZs3M3fu3NGrWsac69rbcKNe/H7bkdTvt/N8hEJ2yGxxsZ0R9fjj7SJyhYVDP38sZle19X5xORltfiB1mkuxLRvV2LlBtmFbSYLYeUMcbAABtYLkM58TJOwrpchfQXlwEmWhugzW4qckWEWRGU9xoIqIvwyXJCQd9ArMnBGFj5KSEk4++eQB24qKihg/fnx6+1VXXcWSJUuoqKigtLSUa665hrlz5w450kWykzHwzjuwapUNENOmDR0+pkyxw2WLi+Hss20H0nDYzuFxwgl2+1BzfIANHa+9ZoNLn0bA5Z0QNpTUYYfsNgDTsHOFbMWOkNmCHbbbhW0NkfxREqzihNJ5lASqCPuLM10OYNeJqY5MJ+CE2Nq7no54syYgy6BR75R+99134/P5uPTSSwdMMia5wxjYtg1eftm2fhx77OBZTn0+e0mluhqOO2745xquE3t3t50VVZdd8o+DbekIAhFs59SUHuykZXuxl1782L4gCh/5pdA/jvrC2RT6ywn6hmn+9JxDRXgKJcEqet12nE6fwkcGHXH4ePbZZwd8HolEWL58OcuXLz/Sp5YMSYWPf/zDtmLMnm0vqYTDA0PISEfHGWOH50aj0NICr75qJyhTy0f+Ge6lEcBO316EbfDuwAaR1MRle9i3qF0C22lVDeO5I+BECPoKKPCX2WnSfZEBq9Jmmt8Jgg/CvhIKA+OIu71E3S70KvOehuPLIKnLLjt3wrhxMG8elJSMzmRg0Si0t9vZUZ9/3vb7GG4uEMk/IeykZQDHYd/yU5ddNrFvqO6G/m3J/pvkhpCviJJAFUWBSiL+kkFrs2RawAkTcMIUBsZREqihL9lG3O3F1RR5nlP4kCHFYtDTA1u3wpo1dlTLrFl2FdpgcOjF5oZjjB1Sm0zCli12PZi33rJfQ8FjfwHshYgS7Gorqb8Y49hxI7H+f+fuNy01fXuKwXZITa0vMwk7UVkQGz5a+j/2su/o4x7WKyMT9hVREqyh0F+RnkAsW+xfS8hXSGmwBp/joyPRrIaPDFD4kCH19EBvLzz3HGzYAKedBt/4BlRV2daQkbaCdHXZ53zqKXjgAdi71z6/7K8Qe0HiRGAuNoiAvSixEvuruB17YSJ/pOYNKcTOlmqwl1y6gZexLSEb+z+2Yi/TSHYqDdUytWiO/cWexb9eSgLVTCk6k93Rd9kV3UDSqOOZ17L31SEZZYy9dXfbyy87dthLJX19tgWjsNDeDhRCkkm7dkuqj8fevbaDaUuLDSJq9UjxYYPGeOwg1SnY5dz2Dx9TsN0338P+anbJ5RaQlP1bQnzYFo+UILazqh/b8hFiX9tQanbVJHaBO/3hmh38TpCQv4igL5JVrR7v53cChHz9dR7eKiNyhBQ+5ICiUWhttZ1P77wTKivhnHPsRGJnnWUXhxtOTw+sWGFDy+rV9lJLayvs3m2Dh2Y2BfurtwT7d//FwAJsCJnIvl/LUWA29u/+XwAvYntK9Hhcq7fCwAxs1JrFvotPe7CtH1uwHVPfJN/agnJXwAlT4Csh6CvIqo6m7+d3QhT4Swn7CrO6znym8CEHlJpsrK0Nmppg1y47c2k8Du+b8mWQRMKOmnn7bfjnP+GNNzwpOcc42CXbirEXHU5i3/JtqTfFRP8+bdi5RIuxf/fnd/jwYyctSzHYiFbc/zHWv09qKhm3f58kmsAsU3z4CfjCdlRJFv9S9zl+Ak7I1pnFLTT5TOFDDkkiYUep9PXB//t/dhbTmTPhxBOHf0w0CmvX2laPnTu9qzW3hLB/1zcA07HdLoPv28eH/TUcwLaAOMBaYJ13ZWaJEux3rBy7im4PtodManG7buyomVT3XF3J91bYX0xZsI6AE87qFoWQr5DSUB2diZ34dNklIxQ+5JC4rg0efX02hJSU2D4cB5JI2NEtGzZ4U2Nu8mPHeJyAbdUoZPBfjD5sfw8H2zrShe37cXTZf1G7Uux3K46dRbWXfZOXdWJ7ybgofHgt4ISJ+MvwOcOsKJklAr4wAcKEfEXq85EhCh8iIlkgELDLEYRCdlTZ4S723dJiL3eKZDOFDxGRLBAI2BmFS0pgxgz778Px6qt2xWh16JZspvAhklFJ7LiNAHA8thdDANuzIXX5xcWOeOkFNmMvMOz2vNJMM9jvQurWif1ube//uAPb56MVe7kll+esHGol6UOR6cCRMH30JtsIOBFCvgIcJzsvaSTcaHpqdZMHQ9ZzkcKHSEbFgH9gp9E6ATgFO35j/wlUXOyv2jZsJ9OVHK1TbXVhw8Uu4F3ssNuXsaEjNdolP2ZAyU3RZDcd8R1E/GUEfXVZ2+U05nbTEW+hO7EH1+jVkgkKHyIZZbCzVDjYVo1XsVNr1UG6I1wUO4ajFTvLaRdHQ1fKJDZUxLHfgdQ8H7ux0WsnNo719N+f61zXLjnQ2wt79hz+govd3aNb10i4JEi4UZK+7F4S0DUuCRMlabK7znym8CGSUQbbqtENPAY0YqfWev/06k8DzewLIfn/11oUeAMbNFJHnZpkLIkNI6kZTvNBPG5DR2urHZo+kvWT9heLZe7yS8LE6HU78bsRTBb/Uk+aGH3JTmJuT1bXmc8UPmRIpaVQXDz8/cXFdpE5GQ2pCwWt2F+lRdgZTvcPH5uxrR6d5NM6r2a/W5J9E8en1nbZzb72ntTMpvl6wSm1ACPYIJKLkiZONNlFxFeMMSZr5xlLmjgxt5u426c+Hxmi8CGD+P3wmc/AJZcMv08gANOne1fT0SF1AeEf2L/195/hdE//fbncjXJoUexFpB3Yni+d2C64qVVt+/r/HedouNiU29pj29nUvZqayAxqCk5kYN+l7NGZ2Mmm7pfoSuzqv/QiXlP4kEEcB6ZNg3PP1czD3kr036LYsJF/9p/2PNVBNDWOZy+wtf/jhv5tXeRTO0/+i7k9dMRbKA3WYIxrWz8gKxaZM/tdi0rV2Zdsw6jDaUYofIiIZ2LY1owebOtGOzZsdGBbPFqxLR0d7LsEI7kj5nbTmWimKzGRvmQn4PSvHJsdM57aTqYxehKtdCaaibs9uIq3GaHwIQd0sI5rWfAHjWSh4V42CfaNVlmP7UK7i/ztx3G0SZg+Esk++pLtxNweAk6IgO8wp2odA0kTJ5a083v0JPbioksumaLwIYO4LqxadeB9QiGYNw9OOMGbmiR3GPaNROnCtmJ0Y4fG9mAvrfQA2/rvO8wRpZLFehKtbO5eQ0mwmslFs/H7D3O61lFlaI2+x+7oO+yJvquOphmm8CGDuC784Q/wxBPD71NSApWVCh8ytBg2VGzHdp3dgZ0MrJd943W07H3+6kzs4q3OZxgXmkx15AQiWRA+DIaWvibe6nyanuQehY8MU/iQIbmuvQ0nkTjw/XL0SGKDRgLbhyOK7bvRhe0229L/eQ82lCRRX45855o4fW4HPclW9sa34JKgKFBJyFeYgVqSdCd2E3W76Ey00JdsJ+72oeibWQofInJEYth+G53YQcK7gfewl1lc9oWNBGrtOFrETR8d8R0k3F7e6fwrJcEajis5h4rQZM9rSZoYW3vW0xrbxPbeV2mPb9PEYllA4UNEDpnBho0oNlTEsJdStmPDx05sK0db/+dytDIYkiRMlO7EbsChO7GbsK+YkK+QgBMe8+G3rkkQTXYTdbvoSuyiM95C1O3S6JYsofAhIiOyHduy0YodLtuN7dMRxV5aSaDJwMSKuT1s632l/3KLoTxUz6TCU6kKHz/mX7sv2cXG7hfojO/kve5G2uPb+i+3SDZQ+BCRQVKrw+7/Mdl/Sy3qths7cqUbO2Q2/+ZelSNlcIm6nSRMlM5EC47jpyIxhXiwF5/jx+8E+/d0jqglZN8EYgaXpJ3m3e2kM95CR3wH3Yk99Cbbj/h4ZPQofIjIIFFsyOjDDontxk593oq9nNLZv083NnSoIVsOxDUJdvVtoC22lZ7EHrb1rKcychz1hacTcEL9LSNHdhkmYWIk3D7a4lt5r3s1PYm9NPe9TtTtVPDIQgofIpKWGoUSxwaMLuyllb3YzqTbM1SX5DaDS3dyDyTtRF+d8V34nADV4engMwR8EXwGhgsgqVYRM+Ssh3aba+LE3V664rvY3vMKPcm9tMe3kTD5su5xflH4EDnKGWyLRmrZ+mb29ePoY9+Sdh2ZKlDyStTtwjVJtvSsozfZRshXSHGgipCvgHHhyYR8RQScED4n0N9BtSj9WNfE6U22kzSJ/qnS47THt9OTaKU32U5PspXuxB7a4ttImD6SRhcDs5XCh4jQhu2/sRF4FdvisQP145DRF3O7idFNd3I3O/peJeQroiw4iUJ/OVPNXIoC44n4iu2aMAHfgPCRJEFPspW420ef20nc7WNLzzr2RN+lO7GbzkRLBo9MRkLhQwZxHDjjDJgxY/i1WyIRaGjwti4ZGwYbNF7FdiJtw7Z0aCIw8ULSJOhLtuOaOM29rxP2FxN0IvidABF/6YDZURP9i8Il+y+xJE2cttgWepN7ibu9GTwKGSmFDxnE74f/839g0aLh93Ecu76L5D4DNGGXsU+NbkltFxlrydRIGBz2xjbvN+rFwRmiD8i+CcLsv4xx+6dK1ys2lyh8yJBCISgs1Kq1RwsXtXRIJpn+/1xliKOEb6QP2LZtG5/5zGcYP348BQUFzJw5k7///e/p+40x3HTTTdTW1lJQUMD8+fPZsGHDqBYtIiIiuWtELR979+7l7LPP5mMf+xh//vOfmTBhAhs2bGDcuHHpfe644w7uvfdeHnzwQRoaGrjxxhtZsGABr7/+OpFIZNQPQMbG3r2wefORtXzs2AF9WTWhYClQT37NSlHJYfwNISKSUSMKHz/4wQ+or6/n/vvvT29r2K/XoTGGe+65h+9+97tcdNFFAPzyl7+kurqaxx57jMsvv3yUypaxlEzCr34FzzxzZM8Ti8Gbb45OTaPjfwMzMl3EKKsCFOpFJLeMKHz84Q9/YMGCBXzqU59i1apVTJw4ka9+9at88YtfBGDjxo00Nzczf/789GPKysqYM2cOjY2NQ4aPaDRKNLpvEpiODs0mkGnGwLvv2lvuSq2f6mKnzIphf1FXZbKoMZJaTSW1fqw66ohIdhtR+Hj33Xe57777WLJkCd/5zndYs2YNX/va1wiFQlx55ZU0NzcDUF1dPeBx1dXV6fveb9myZdxyyy2HWb7IcJJACzZ4PImdwSKf7QTewC7tVg1oKJKIZK8RhQ/XdTnjjDP4/ve/D8Bpp53Ga6+9xk9+8hOuvPLKwypg6dKlLFmyJP15R0cH9fX1h/VcIvsk2bew+8vk/8Tgndhpwgww7iD7iohk1ojCR21tLSeeeOKAbTNmzOC///u/AaipqQGgpaWF2tra9D4tLS2ceuqpQz5nOBwmHA6PpAyRAwhi//JPYlsBOoGiAz4iP4Swx+0D6oDxQGFGK8pFxcVw4om2v9LRpLQUApp4QTw0opfb2WefTVNT04Btb731FlOmTAFs59OamhpWrlyZDhsdHR2sXr2ar3zlK6NTscgBhYFa7Es7xtGzIkkIGzpCwGSgAijOaEW5qLQUTjkl01V4T/P5iNdGFD6uv/56PvjBD/L973+fT3/607z00kv87Gc/42c/+xlgVx687rrruO2225g2bVp6qG1dXR0XX3zxWNQv8j4RYCr20kMJcLRNuRzAhq9i7NBiGQnH0S9iES+MKHyceeaZPProoyxdupRbb72VhoYG7rnnHhYuXJje51vf+hbd3d1cffXVtLW18aEPfYgnn3xSc3yIR8qBD2H7PiQ5+qZLdLCXXnxoAmMRyVYjfnf6+Mc/zsc//vFh73cch1tvvZVbb731iAqT4fn9fkpLS0km82myrCPlwy6Ppn4O71dY2A5AIBCguLiYREJr1YJdwygehz17Ml1J9olGHaLRIAAFBQVUVFRgzNEW5IdWUFBAQB1kjpi+gzmooKCAM844Q+FjgCDw/6HZPgebNKkAxymktLSUk046Sb9E9rN7N/zlL5muIvskEi7NzbuALhoaGgYMIDjaOY5DUdHR0Il9bCl85KBUy4e839BzyRztCgsnAIUEAgH9xfY+8Ti0tma6iuyTTCaJRvfiOA4FBQUUFBRkuiTJM/ozUURERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8ZTCh4iIiHhK4UNEREQ8pfAhIiIinlL4EBEREU8pfIiIiIinFD5ERETEUwofIiIi4imFDxEREfGUwoeIiIh4SuFDREREPKXwISIiIp5S+BARERFPKXyIiIiIpxQ+RERExFMKHyIiIuIphQ8RERHxlMKHiIiIeErhQ0RERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8dSIwkcymeTGG2+koaGBgoICjj32WP7jP/4DY0x6H2MMN910E7W1tRQUFDB//nw2bNgw6oWLiIhIbhpR+PjBD37Afffdx49+9CPeeOMNfvCDH3DHHXfwwx/+ML3PHXfcwb333stPfvITVq9eTVFREQsWLKCvr2/UixcREZHcExjJzi+88AIXXXQRF154IQBTp07lN7/5DS+99BJgWz3uuecevvvd73LRRRcB8Mtf/pLq6moee+wxLr/88lEuX0RERHLNiFo+PvjBD7Jy5UreeustAP7xj3/w/PPPc8EFFwCwceNGmpubmT9/fvoxZWVlzJkzh8bGxiGfMxqN0tHRMeAmIiIi+WtELR/f/va36ejoYPr06fj9fpLJJN/73vdYuHAhAM3NzQBUV1cPeFx1dXX6vvdbtmwZt9xyy+HULiIiIjloRC0fv/vd7/j1r3/NQw89xLp163jwwQf5z//8Tx588MHDLmDp0qW0t7enb1u2bDns5xIREZHsN6KWj29+85t8+9vfTvfdmDlzJps2bWLZsmVceeWV1NTUANDS0kJtbW36cS0tLZx66qlDPmc4HCYcDh9m+SIiIpJrRtTy0dPTg8838CF+vx/XdQFoaGigpqaGlStXpu/v6Ohg9erVzJ07dxTKFRERkVw3opaPT3ziE3zve99j8uTJnHTSSbz88svcddddfOELXwDAcRyuu+46brvtNqZNm0ZDQwM33ngjdXV1XHzxxWNRv4iIiOSYEYWPH/7wh9x444189atfZefOndTV1fGlL32Jm266Kb3Pt771Lbq7u7n66qtpa2vjQx/6EE8++SSRSGTUixcREZHc45j9pyfNAh0dHZSVlfHZz36WUCiU6XJEct6ECROoqanBcZxMlyI5IplMsnXrVtrb2zNdiuSQWCzGL3/5S9rb2yktLT3gvlrbRURERDyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8ZTCh4iIiHhK4UNEREQ8pfAhIiIinlL4EBEREU8pfIiIiIinFD5ERETEUwofIiIi4imFDxEREfGUwoeIiIh4SuFDREREPKXwISIiIp5S+BARERFPKXyIiIiIpxQ+RERExFMKHyIiIuIphQ8RERHxlMKHiIiIeErhQ0RERDyl8CEiIiKeCmS6gPczxgAQi8UyXIlIfohGo/T19WW6DMkhrusSjUb1Piwjknq9pH6PH4hjDmUvD23dupX6+vpMlyEiIiKHYcuWLUyaNOmA+2Rd+HBdl6amJk488US2bNlCaWlppksadR0dHdTX1+fl8eXzsUF+H5+OLXfl8/Hp2HKHMYbOzk7q6urw+Q7cqyPrLrv4fD4mTpwIQGlpaV6ckOHk8/Hl87FBfh+fji135fPx6dhyQ1lZ2SHtpw6nIiIi4imFDxEREfFUVoaPcDjMzTffTDgcznQpYyKfjy+fjw3y+/h0bLkrn49Px5afsq7DqYiIiOS3rGz5EBERkfyl8CEiIiKeUvgQERERTyl8iIiIiKcUPkRERMRTWRk+li9fztSpU4lEIsyZM4eXXnop0yWN2LJlyzjzzDMpKSmhqqqKiy++mKampgH7fPSjH8VxnAG3L3/5yxmq+ND9+7//+6C6p0+fnr6/r6+PRYsWMX78eIqLi7n00ktpaWnJYMUjM3Xq1EHH5zgOixYtAnLrvD333HN84hOfoK6uDsdxeOyxxwbcb4zhpptuora2loKCAubPn8+GDRsG7NPa2srChQspLS2lvLycq666iq6uLg+PYngHOr54PM4NN9zAzJkzKSoqoq6ujs9+9rNs3759wHMMdb5vv/12j49ksIOdu8997nOD6j7//PMH7JOt5+5gxzbUz5/jONx5553pfbL1vB3Ke/+hvEdu3ryZCy+8kMLCQqqqqvjmN79JIpHw8lDGVNaFj9/+9rcsWbKEm2++mXXr1jFr1iwWLFjAzp07M13aiKxatYpFixbx4osvsmLFCuLxOOeddx7d3d0D9vviF7/Ijh070rc77rgjQxWPzEknnTSg7ueffz593/XXX88f//hHHnnkEVatWsX27du55JJLMljtyKxZs2bAsa1YsQKAT33qU+l9cuW8dXd3M2vWLJYvXz7k/XfccQf33nsvP/nJT1i9ejVFRUUsWLBgwCq4Cxcu5J///CcrVqzgiSee4LnnnuPqq6/26hAO6EDH19PTw7p167jxxhtZt24dv//972lqauKTn/zkoH1vvfXWAefzmmuu8aL8AzrYuQM4//zzB9T9m9/8ZsD92XruDnZs+x/Tjh07+MUvfoHjOFx66aUD9svG83Yo7/0He49MJpNceOGFxGIxXnjhBR588EEeeOABbrrppkwc0tgwWeass84yixYtSn+eTCZNXV2dWbZsWQarOnI7d+40gFm1alV620c+8hFz7bXXZq6ow3TzzTebWbNmDXlfW1ubCQaD5pFHHklve+ONNwxgGhsbPapwdF177bXm2GOPNa7rGmNy97wB5tFHH01/7rquqampMXfeeWd6W1tbmwmHw+Y3v/mNMcaY119/3QBmzZo16X3+/Oc/G8dxzLZt2zyr/VC8//iG8tJLLxnAbNq0Kb1typQp5u677x7b4o7QUMd25ZVXmosuumjYx+TKuTuU83bRRReZc889d8C2XDhvxgx+7z+U98j/+Z//MT6fzzQ3N6f3ue+++0xpaamJRqPeHsAYyaqWj1gsxtq1a5k/f356m8/nY/78+TQ2NmawsiPX3t4OQEVFxYDtv/71r6msrOTkk09m6dKl9PT0ZKK8EduwYQN1dXUcc8wxLFy4kM2bNwOwdu1a4vH4gHM4ffp0Jk+enJPnMBaL8atf/YovfOELOI6T3p6r521/GzdupLm5ecC5KisrY86cOelz1djYSHl5OWeccUZ6n/nz5+Pz+Vi9erXnNR+p9vZ2HMehvLx8wPbbb7+d8ePHc9ppp3HnnXfmTPP2s88+S1VVFSeccAJf+cpX2LNnT/q+fDl3LS0t/OlPf+Kqq64adF8unLf3v/cfyntkY2MjM2fOpLq6Or3PggUL6Ojo4J///KeH1Y+drFrVdvfu3SSTyQHfcIDq6mrefPPNDFV15FzX5brrruPss8/m5JNPTm//13/9V6ZMmUJdXR2vvPIKN9xwA01NTfz+97/PYLUHN2fOHB544AFOOOEEduzYwS233MKHP/xhXnvtNZqbmwmFQoPe3Kurq2lubs5MwUfgscceo62tjc997nPpbbl63t4vdT6G+nlL3dfc3ExVVdWA+wOBABUVFTl3Pvv6+rjhhhu44oorBqwg+rWvfY3TTz+diooKXnjhBZYuXcqOHTu46667MljtwZ1//vlccsklNDQ08M477/Cd73yHCy64gMbGRvx+f96cuwcffJCSkpJBl25z4bwN9d5/KO+Rzc3NQ/5cpu7LB1kVPvLVokWLeO211wb0iwAGXHudOXMmtbW1zJs3j3feeYdjjz3W6zIP2QUXXJD+9ymnnMKcOXOYMmUKv/vd7ygoKMhgZaPv5z//ORdccAF1dXXpbbl63o5m8XicT3/60xhjuO+++wbct2TJkvS/TznlFEKhEF/60pdYtmxZVq+5cfnll6f/PXPmTE455RSOPfZYnn32WebNm5fBykbXL37xCxYuXEgkEhmwPRfO23Dv/ZJlHU4rKyvx+/2Dev22tLRQU1OToaqOzOLFi3niiSd45plnmDRp0gH3nTNnDgBvv/22F6WNmvLyco4//njefvttampqiMVitLW1DdgnF8/hpk2beOqpp/i3f/u3A+6Xq+ctdT4O9PNWU1MzqLN3IpGgtbU1Z85nKnhs2rSJFStWDGj1GMqcOXNIJBK899573hQ4So455hgqKyvTr8N8OHd//etfaWpqOujPIGTfeRvuvf9Q3iNramqG/LlM3ZcPsip8hEIhZs+ezcqVK9PbXNdl5cqVzJ07N4OVjZwxhsWLF/Poo4/y9NNP09DQcNDHrF+/HoDa2toxrm50dXV18c4771BbW8vs2bMJBoMDzmFTUxObN2/OuXN4//33U1VVxYUXXnjA/XL1vDU0NFBTUzPgXHV0dLB69er0uZo7dy5tbW2sXbs2vc/TTz+N67rp0JXNUsFjw4YNPPXUU4wfP/6gj1m/fj0+n2/QJYtst3XrVvbs2ZN+Heb6uQPb8jh79mxmzZp10H2z5bwd7L3/UN4j586dy6uvvjogPKaC84knnujNgYy1DHd4HeThhx824XDYPPDAA+b11183V199tSkvLx/Q6zcXfOUrXzFlZWXm2WefNTt27Ejfenp6jDHGvP322+bWW281f//7383GjRvN448/bo455hhzzjnnZLjyg/v6179unn32WbNx40bzt7/9zcyfP99UVlaanTt3GmOM+fKXv2wmT55snn76afP3v//dzJ0718ydOzfDVY9MMpk0kydPNjfccMOA7bl23jo7O83LL79sXn75ZQOYu+66y7z88svp0R633367KS8vN48//rh55ZVXzEUXXWQaGhpMb29v+jnOP/98c9ppp5nVq1eb559/3kybNs1cccUVmTqkAQ50fLFYzHzyk580kyZNMuvXrx/wc5gaMfDCCy+Yu+++26xfv96888475le/+pWZMGGC+exnP5vhIzvwsXV2dppvfOMbprGx0WzcuNE89dRT5vTTTzfTpk0zfX196efI1nN3sNelMca0t7ebwsJCc9999w16fDaft4O99xtz8PfIRCJhTj75ZHPeeeeZ9evXmyeffNJMmDDBLF26NBOHNCayLnwYY8wPf/hDM3nyZBMKhcxZZ51lXnzxxUyXNGLAkLf777/fGGPM5s2bzTnnnGMqKipMOBw2xx13nPnmN79p2tvbM1v4IbjssstMbW2tCYVCZuLEieayyy4zb7/9dvr+3t5e89WvftWMGzfOFBYWmn/5l38xO3bsyGDFI/eXv/zFAKapqWnA9lw7b88888yQr8Mrr7zSGGOH2954442murrahMNhM2/evEHHvGfPHnPFFVeY4uJiU1paaj7/+c+bzs7ODBzNYAc6vo0bNw77c/jMM88YY4xZu3atmTNnjikrKzORSMTMmDHDfP/73x/wCzxTDnRsPT095rzzzjMTJkwwwWDQTJkyxXzxi18c9Edatp67g70ujTHmpz/9qSkoKDBtbW2DHp/N5+1g7/3GHNp75HvvvWcuuOACU1BQYCorK83Xv/51E4/HPT6aseMYY8wYNaqIiIiIDJJVfT5EREQk/yl8iIiIiKcUPkRERMRTCh8iIiLiKYUPERER8ZTCh4iIiHhK4UNEREQ8pfAhIiIinlL4EBEREU8pfIiIiIinFD5ERETEU/8/6kMDPbe6OYgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "state_bytes = mdp[\"states\"][state, :mdp[\"state_lengths\"][state]].tobytes()\n",
    "env.unwrapped.render_mode = \"rgb_array\"\n",
    "env.set_state(state_bytes)\n",
    "plt.imshow(env.render())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, we get the same image as in the tabular version.\n",
    "\n",
    "That's it for this quick introduction to the BRIDGE dataset! While all the tabular environments in the dataset have transition and reward matrices, only the smaller ones have screen images and states, since they would be prohibitively large for environments with many states."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "effective_horizon",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  },
  "nbreg": {
   "diff_ignore": [
    "/metadata/language_info/version"
   ]
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}