{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 15wk-1: 강화학습 (4) – LunarLander, A1, A2\n",
        "\n",
        "최규빈  \n",
        "2024-06-10\n",
        "\n",
        "<a href=\"https://colab.research.google.com/github/guebin/DL2024/blob/main/posts/15wk-1.ipynb\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" style=\"text-align: left\"></a>\n",
        "\n",
        "# 1. 강의영상\n",
        "\n",
        "<https://youtu.be/playlist?list=PLQqh36zP38-zesXx2Ad1CQF3V85SBnmcl&si=1N-UhzaPrEwP2sNn>\n",
        "\n",
        "# 2. Imports"
      ],
      "id": "c5d3ff15-f243-4cc3-bf86-c50b355ab337"
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "#!apt-get install swig\n",
        "#!pip install gymnasium[box2d]\n",
        "import gymnasium as gym\n",
        "#--#\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from matplotlib.animation import FuncAnimation\n",
        "import IPython\n",
        "#--#\n",
        "import collections\n",
        "import random\n",
        "#--#\n",
        "import torch"
      ],
      "id": "03fd7323-3dc0-434e-ab1c-97d4c761082b"
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "def show(ims,jump=10):\n",
        "    ims = ims[::jump]\n",
        "    fig = plt.Figure()\n",
        "    ax = fig.subplots()\n",
        "    def update(i):\n",
        "        ax.imshow(ims[i])\n",
        "    ani = FuncAnimation(fig,update,frames=len(ims))\n",
        "    display(IPython.display.HTML(ani.to_jshtml()))"
      ],
      "id": "96f012df-2b65-4c7f-a6cd-4158e908f68c"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 3. 예비학습\n",
        "\n",
        "## A. `collections.deque`\n",
        "\n",
        "`-` `collections.deque` 의 기능"
      ],
      "id": "7376603e-0f86-427b-8f66-278b18e0608c"
    },
    {
      "cell_type": "code",
      "execution_count": 73,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "a = collections.deque([12,21,33], maxlen = 5)\n",
        "a"
      ],
      "id": "8d38a7dc-86e3-434e-97f9-3ccd610b98de"
    },
    {
      "cell_type": "code",
      "execution_count": 74,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "a.append(44)\n",
        "a"
      ],
      "id": "9e58eac0-8b8a-44cf-9892-c45e7fefd346"
    },
    {
      "cell_type": "code",
      "execution_count": 75,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "a.append(55)\n",
        "a"
      ],
      "id": "37c15ea1-4afb-4086-981e-8523c94c83e7"
    },
    {
      "cell_type": "code",
      "execution_count": 76,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "a.append(-66)\n",
        "a"
      ],
      "id": "43601bc6-b3cc-4de0-8c63-fe7c1db02365"
    },
    {
      "cell_type": "code",
      "execution_count": 77,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "a.append(-40)\n",
        "a"
      ],
      "id": "532a2ae4-5c26-4abe-bd52-3ff3ec2f4fd1"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`-` 단점? numpy array 보다는 list 느낌임 (연산에 특화된건 아님)"
      ],
      "id": "dad92f26-f903-4f60-a239-22e403f3e5d3"
    },
    {
      "cell_type": "code",
      "execution_count": 78,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "a + 1"
      ],
      "id": "c44f3d20-f0a4-4a87-a88e-3ba8139dbf32"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`-` 그렇지만 필요하다면 np.array 화 시킬 수 있음."
      ],
      "id": "45ab8372-70f7-4d8c-baa4-ff098f3bac3d"
    },
    {
      "cell_type": "code",
      "execution_count": 79,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "np.array(a) + 1 "
      ],
      "id": "79900a05-c590-454e-af46-e9390687ce44"
    },
    {
      "cell_type": "code",
      "execution_count": 80,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "torch.tensor(a) + 1"
      ],
      "id": "30c4a07f-bda3-4822-be7f-8b7d783183ee"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`-` collection.deque 는 리플레이 버퍼를 구현할때 유용한 자료구조이다.\n",
        "\n",
        "-   (우리가 했던) 기존방식: 모든 데이터를 저장하며 하나의 경험씩 학습함\n",
        "-   리플레이버퍼: 최근 $N$개의 데이터를 저장하여 여러경험을 샘플링하여\n",
        "    학습하는 방식\n",
        "-   리플레이버퍼의 장점: 메모리를 아낄 수 있다, 다양한 종류의 경험을\n",
        "    저장하고 무작위로 재사용하여 학습이 안정적으로 된다, “저장 -\\> 학습\n",
        "    -\\> 저장” 순으로 반드시 실시간으로 학습할 필요가 없어서 병렬처리에\n",
        "    용이하다, 강화학습에서 연속된 경험은 상관관계가 있을 수 있는데\n",
        "    무작위 샘플로 이러한 상관관계를 제거할 수 있음\n",
        "\n",
        "## B. `replay_buffer`"
      ],
      "id": "3c3811e9-fb25-4fff-b1d9-04aa0e69d01a"
    },
    {
      "cell_type": "code",
      "execution_count": 81,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "current_states = collections.deque([torch.tensor([0.23,0.1]),torch.tensor([0.34,0.2])],maxlen=5)\n",
        "actions = collections.deque([torch.tensor(0), torch.tensor(1)],maxlen=5)\n",
        "rewards = collections.deque([torch.tensor(3.43), torch.tensor(0.13)],maxlen=5)\n",
        "next_states = collections.deque([torch.tensor([0.34,0.2]),torch.tensor([0.45,0.3])],maxlen=5)\n",
        "terminations = collections.deque([torch.tensor(False),torch.tensor(False)],maxlen=5)"
      ],
      "id": "0f98b6c6-b5fc-4a3e-91fd-738b4bab320f"
    },
    {
      "cell_type": "code",
      "execution_count": 82,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "current_states, actions,rewards,next_states,terminations"
      ],
      "id": "4a4b672c-0296-4dc0-a4ab-454b04f31778"
    },
    {
      "cell_type": "code",
      "execution_count": 83,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "memory = list(zip(current_states,actions,rewards,next_states,terminations))\n",
        "memory"
      ],
      "id": "50dd479c-3aab-4bff-8d99-1638e91ea9fe"
    },
    {
      "cell_type": "code",
      "execution_count": 84,
      "metadata": {},
      "outputs": [],
      "source": [
        "random.sample(memory,1)"
      ],
      "id": "dc076c53-28f8-4541-a748-b01cbe4f95c7"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 4. LunarLander\n",
        "\n",
        "`-` ref: <https://gymnasium.farama.org/environments/box2d/lunar_lander/>\n",
        "\n",
        "`-` Lunar Lander: 요약\n",
        "\n",
        "**Observation Space (State Space)** – 8개의 변수\n",
        "\n",
        "1.  착륙선의 x 좌표\n",
        "2.  착륙선의 y 좌표\n",
        "3.  착륙선의 x 속도\n",
        "4.  착륙선의 y 속도\n",
        "5.  착륙선의 각도\n",
        "6.  착륙선의 각속도\n",
        "7.  왼쪽 다리가 땅에 닿아있는지 여부 (1 또는 0)\n",
        "8.  오른쪽 다리가 땅에 닿아있는지 여부 (1 또는 0)\n",
        "\n",
        "**Action Space** – 4개의 변수\n",
        "\n",
        "1.  {0 : 아무 행동도 하지 않음}\n",
        "2.  {1 : 왼쪽 엔진 발사 (오른쪽으로 기울임)}\n",
        "3.  {2 : 메인 엔진 발사 (위로 밀어 올림)}\n",
        "4.  {3 : 오른쪽 엔진 발사 (왼쪽으로 기울임)}\n",
        "\n",
        "**Reward**\n",
        "\n",
        "-   **거리 보상**: 착륙 패드에 가까울수록 보상 증가\n",
        "-   **속도 보상**: 속도가 낮을수록 보상 증가\n",
        "-   **각도 보상**: 각도가 수직에 가까울수록 보상 증가\n",
        "-   **착륙 다리 보상**: 다리가 착륙 패드에 닿으면 보상\n",
        "-   **연료 사용 패널티**: 엔진 사용 시 패널티\n",
        "-   **안전한 착륙 보상**: 안정적으로 착륙 시 큰 보상 (+100~+140)\n",
        "-   **충돌 패널티**: 착륙 패드 이외의 장소에 충돌 시 패널티 (-100)\n",
        "\n",
        "`-` 환경생성"
      ],
      "id": "6bb19d3e-2ac6-4972-bd3e-49bf067e8d90"
    },
    {
      "cell_type": "code",
      "execution_count": 85,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "env = gym.make(\"LunarLander-v2\",render_mode = 'rgb_array')\n",
        "env "
      ],
      "id": "33a2796d-86c5-4e97-863f-1abaa6f19b71"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`-` state_space"
      ],
      "id": "0ea32a4e-6ed6-47e2-885e-1a0ebb84d984"
    },
    {
      "cell_type": "code",
      "execution_count": 86,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "env.observation_space"
      ],
      "id": "b05edbde-d1a3-496e-9c10-3738d9618bbc"
    },
    {
      "cell_type": "code",
      "execution_count": 87,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "env.observation_space.sample()"
      ],
      "id": "0ebe9af9-9581-4bfd-80f3-59c21548e6bc"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`-` action_space"
      ],
      "id": "2ae0c13a-7430-4811-bde9-47311d4fd13e"
    },
    {
      "cell_type": "code",
      "execution_count": 88,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "env.action_space"
      ],
      "id": "76a89adf-b4a4-4037-bc0a-8af6c7447ff6"
    },
    {
      "cell_type": "code",
      "execution_count": 89,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "env.action_space.sample()"
      ],
      "id": "991a786c-72a0-4b2b-968d-6edc4e4e676e"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`-` env.reset()"
      ],
      "id": "9b673c86-5df4-4406-bdb4-7eed376f5885"
    },
    {
      "cell_type": "code",
      "execution_count": 90,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "state, _ = env.reset()\n",
        "state "
      ],
      "id": "fff5a9df-9dbf-453e-802f-fba6b9639a34"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`-` env.render()"
      ],
      "id": "096ee729-fa76-4660-a883-4b9dae3746ae"
    },
    {
      "cell_type": "code",
      "execution_count": 91,
      "metadata": {
        "tags": []
      },
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAF7CAYAAAD4/3BBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90\nbGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9h\nAAAPYQGoP6dpAAA1aElEQVR4nO3de3xU9Z3/8fckk5ncJwm5TIaEECCAEC42sRCU+82sCFRbqO5a\nrD7aeoGfFHhYoY8tuNs1FFutrZV2e8FLW9N2FdetiMZFgpS6pQg1AaSoIAESwi0zCYTJ7fv7I2bq\ncE0gYU7I6+nj80jmnG/O+Z4vmHlzzvecsRljjAAAACwkLNQdAAAAOBsBBQAAWA4BBQAAWA4BBQAA\nWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWE5IA8ozzzyj7OxsRUZGKi8vT++8\n804ouwMAACwiZAHld7/7nRYuXKhvf/vb2r59u8aOHavCwkIdOHAgVF0CAAAWYQvVhwWOGjVKn/vc\n57R69erAsuuuu06zZ89WUVFRKLoEAAAswh6KnTY0NGjbtm165JFHgpZPmzZNW7ZsOae93++X3+8P\nvG5padGJEyfUq1cv2Wy2Lu8vAAC4csYY1dbWyuPxKCzs4hdxQhJQjh07pubmZqWlpQUtT0tLU1VV\n1Tnti4qK9Oijj16t7gEAgC5UUVGhjIyMi7YJSUBpc/bZD2PMec+ILF26VIsWLQq89nq96tOnT5f3\nD8Cl3Tr+e4pJTFRm/GiFhzlVU/+JDhz9szb86QnFRqdo1pTvyREeL1dk1/8/a0yzKr079Pa2x1VR\nsV3jblggd/p16pswVmFhDh07vUsb/vp9ffzxn7q8LwAuLC4u7pJtQhJQkpOTFR4efs7Zkurq6nPO\nqkiS0+mU0+m8Wt0D0E7DB9ymmIQEpcePVLSzl5pa6tVoarX7wzflb6hVfIxbjogYOe0xinRc+hfS\nlWoxzYqIiJLN1nrq+G+712pQvymqbzmuXpED5baP1LBBM3T4cJnOnPF1eX8AnF97pmeE5C4eh8Oh\nvLw8lZSUBC0vKSnRmDFjQtElAB1kU5ic0bGKi3IrxpEsSTrVWK09+0t0pHq3jGmRJJlPW181n5n3\n7607pI/3/1l1DVVqbD4te5hTWb3Gqn8/fs8AVhey24wXLVqkX/ziF/rVr36l3bt365vf/KYOHDig\n++67L1RdAtAB/TJu0sghX1Sc06Nwm1NNLfU6UrNTBw/v0OkzJyS1hZOrywTt2egv5WsUZpzy+itk\nTIsSorI0sO9UuVyeEPQOQHuFbA7K3Llzdfz4cf3bv/2bKisrlZubq3Xr1ikrKytUXQLQTnZ7pNJS\nByvMFq5oey9JUq3/sD46WKqDlTs+09Lo6seUFn326Qln/D7t2vu6hl03Qw3Np+QIj1WfpAJlZ31e\nZTtfU3Nz41XuH4D2COkk2QceeEAPPPBAKLsAoMNsmjb6X5WROULREcmyh0eqsbleFSf+T/v2v6um\n5jMh7d3ZcaixqV6fHPo/9cnIV5Q9SY7wGMVFunVd31t0pHqvKqt2hqSfAC6Oz+IB0GE52ZPU3OJX\nnDNdknT09C5VHtmtoyc+DG5oWu+saW5pUFPzGTU1+9XU4ldTS4OaWxo/rSa1mCa1mGa1mBYZE3wG\npMPMuWdtqk/s0ZEju+XzH1azaVC4zSl3wnClp+XKbmcCPmBFIT2DAqD7uXfWf+tE/d+VHvc5hdns\nqm88oRN1H+mvf/vNOW1bTKN89ZU61bhDYbbP/roxn4kQ5wkjtrYvrf9JYZLNJput7fWnXz993XrX\nTusP+Rt9ioiICtpcU/MZ7T/4f0pJHaAaxwH1ispRjCNVIwbepgMHt+rY8Y+vbFAAdDoCCoB2S4zL\n0hlzUlF2l6LsSWoxTao+tUvlu9epqcl/TvvjNfv04itflz3cofAwh+zhToWHO1pfhztlD/902afr\nWpe3tfvH6/CwtrYRCm/bRliEwsIjPv3qUFi4XeFhDvlqjqmiYvs5fdlfuUXDamfruPPvSojso3Cb\nUwlRWerX90Z5fYfV2BjaS1MAghFQALTbxPwlinA6FGaz63TjMTW21OvQke36+MDFHnxmWi/tNPvl\nb6y9an09n227f6vC8ctVfapcnrh8Oe0uZWcV6MChv6qqandI+wYgGHNQALTbtg9+raNH9ivSHq+a\nM/t12LdNH3z4phob6kPdtXY5fHSHqqv36kyjV4dr39Mnx99R2a5XderU8VB3DcBZQvZpxlfC5/PJ\n5XKFuhtAjxQdmaTkhAG6Ifcr8p2p1Dtbf6wz/u7zVNb4mHT9y6zn9d6Hv9buD95U3alj3GoMXGVe\nr1fx8fEXbUNAAXBZwsMckoyaW7rfm3uEPVrNLX61tDSHuitAj9SegMIcFACXpbmlIdRduGyNTadD\n3QUAl8AcFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAA\nYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkE\nFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDmdHlBWrFghm80WVG63O7DeGKMVK1bI4/EoKipKEyZM0M6d\nOzu7GwAAoBvrkjMoQ4cOVWVlZaDKysoC61atWqUnnnhCTz/9tLZu3Sq3262pU6eqtra2K7oCAAC6\noS4JKHa7XW63O1ApKSmSWs+e/PCHP9S3v/1t3XbbbcrNzdVzzz2n06dP67e//W1XdAUAAHRDXRJQ\n9u7dK4/Ho+zsbH35y1/Wxx9/LEnat2+fqqqqNG3atEBbp9Op8ePHa8uWLRfcnt/vl8/nCyoAAHDt\n6vSAMmrUKD3//PN644039POf/1xVVVUaM2aMjh8/rqqqKklSWlpa0M+kpaUF1p1PUVGRXC5XoDIz\nMzu72wAAwEJsxhjTlTs4deqU+vfvr4cfflijR4/WjTfeqMOHDys9PT3Q5mtf+5oqKiq0fv36827D\n7/fL7/cHXvt8PkIKAADdlNfrVXx8/EXbdPltxjExMRo2bJj27t0buJvn7LMl1dXV55xV+Syn06n4\n+PigAgAA164uDyh+v1+7d+9Wenq6srOz5Xa7VVJSEljf0NCg0tJSjRkzpqu7AgAAugl7Z29wyZIl\nuvXWW9WnTx9VV1fru9/9rnw+n+bNmyebzaaFCxfqscceU05OjnJycvTYY48pOjpad955Z2d3BQAA\ndFOdHlAOHjyoO+64Q8eOHVNKSopGjx6td999V1lZWZKkhx9+WPX19XrggQd08uRJjRo1Sm+++abi\n4uI6uysAAKCb6vJJsl3B5/PJ5XKFuhsAAOAyWGKSLAAAQEcRUAAAgOUQUAAAgOUQUAAAgOUQUAAA\ngOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQ\nUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAA\ngOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOV0OKBs2rRJt956qzwe\nj2w2m1555ZWg9cYYrVixQh6PR1FRUZowYYJ27twZ1Mbv92vBggVKTk5WTEyMZs6cqYMHD17RgQAA\ngGtHhwPKqVOnNGLECD399NPnXb9q1So98cQTevrpp7V161a53W5NnTpVtbW1gTYLFy7U2rVrVVxc\nrM2bN6uurk4zZsxQc3Pz5R8JAAC4dpgrIMmsXbs28LqlpcW43W6zcuXKwLIzZ84Yl8tlfvrTnxpj\njKmpqTERERGmuLg40ObQoUMmLCzMrF+/vl379Xq9RhJFURRFUd2wvF7vJd/rO3UOyr59+1RVVaVp\n06YFljmdTo0fP15btmyRJG3btk2NjY1BbTwej3JzcwNtzub3++Xz+YIKAABcuzo1oFRVVUmS0tLS\ngpanpaUF1lVVVcnhcCgxMfGCbc5WVFQkl8sVqMzMzM7sNgAAsJguuYvHZrMFvTbGnLPsbBdrs3Tp\nUnm93kBVVFR0Wl8BAID1dGpAcbvdknTOmZDq6urAWRW3262GhgadPHnygm3O5nQ6FR8fH1QAAODa\n1akBJTs7W263WyUlJYFlDQ0NKi0t1ZgxYyRJeXl5ioiICGpTWVmp8vLyQBsAANCz2Tv6A3V1dfrw\nww8Dr/ft26cdO3YoKSlJffr00cKFC/XYY48pJydHOTk5euyxxxQdHa0777xTkuRyuXTvvfdq8eLF\n6tWrl5KSkrRkyRINGzZMU6ZM6bwjAwAA3Ve77uv9jLfffvu8twzNmzfPGNN6q/Hy5cuN2+02TqfT\njBs3zpSVlQVto76+3syfP98kJSWZqKgoM2PGDHPgwIF294HbjCmKoiiq+1Z7bjO2GWOMuhmfzyeX\nyxXqbgAAgMvg9XovOZ+Uz+IBAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACW\nQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0AB\nAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACW\nQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACW0+GAsmnTJt16663yeDyy2Wx65ZVXgtbffffdstlsQTV6\n9OigNn6/XwsWLFBycrJiYmI0c+ZMHTx48IoOBAAAXDs6HFBOnTqlESNG6Omnn75gm5tvvlmVlZWB\nWrduXdD6hQsXau3atSouLtbmzZtVV1enGTNmqLm5ueNHAAAArjn2jv5AYWGhCgsLL9rG6XTK7Xaf\nd53X69Uvf/lLvfDCC5oyZYok6de//rUyMzP11ltvafr06R3tEgAAuMZ0yRyUjRs3KjU1VQMHDtTX\nvvY1VVdXB9Zt27ZNjY2NmjZtWmCZx+NRbm6utmzZct7t+f1++Xy+oAIAANeuTg8ohYWF+s1vfqMN\nGzboBz/4gbZu3apJkybJ7/dLkqqqquRwOJSYmBj0c2lpaaqqqjrvNouKiuRyuQKVmZnZ2d0GAAAW\n0uFLPJcyd+7cwPe5ubnKz89XVlaWXnvtNd12220X/DljjGw223nXLV26VIsWLQq89vl8hBQAAK5h\nXX6bcXp6urKysrR3715JktvtVkNDg06ePBnUrrq6WmlpaefdhtPpVHx8fFABAIBrV5cHlOPHj6ui\nokLp6emSpLy8PEVERKikpCTQprKyUuXl5RozZkxXdwcAAHQDHb7EU1dXpw8//DDwet++fdqxY4eS\nkpKUlJSkFStW6Pbbb1d6err279+vZcuWKTk5WV/4whckSS6XS/fee68WL16sXr16KSkpSUuWLNGw\nYcMCd/UAAIAeznTQ22+/bSSdU/PmzTOnT58206ZNMykpKSYiIsL06dPHzJs3zxw4cCBoG/X19Wb+\n/PkmKSnJREVFmRkzZpzT5mK8Xu95+0BRFEVRlPXL6/Ve8r3eZowx6mZ8Pp9cLleouwEAAC6D1+u9\n5HxSPosHAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABY\nDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEF\nAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABY\nDgEFAABYDgEFAABYTocCSlFRkW644QbFxcUpNTVVs2fP1p49e4LaGGO0YsUKeTweRUVFacKECdq5\nc2dQG7/frwULFig5OVkxMTGaOXOmDh48eOVHAwAArgkdCiilpaV68MEH9e6776qkpERNTU2aNm2a\nTp06FWizatUqPfHEE3r66ae1detWud1uTZ06VbW1tYE2Cxcu1Nq1a1VcXKzNmzerrq5OM2bMUHNz\nc+cdGQAA6L7MFaiurjaSTGlpqTHGmJaWFuN2u83KlSsDbc6cOWNcLpf56U9/aowxpqamxkRERJji\n4uJAm0OHDpmwsDCzfv36du3X6/UaSRRFURRFdcPyer2XfK+/ojkoXq9XkpSUlCRJ2rdvn6qqqjRt\n2rRAG6fTqfHjx2vLli2SpG3btqmxsTGojcfjUW5ubqDN2fx+v3w+X1ABAIBr12UHFGOMFi1apJtu\nukm5ubmSpKqqKklSWlpaUNu0tLTAuqqqKjkcDiUmJl6wzdmKiorkcrkClZmZebndBgAA3cBlB5T5\n8+fr/fff14svvnjOOpvNFvTaGHPOsrNdrM3SpUvl9XoDVVFRcbndBgAA3cBlBZQFCxbo1Vdf1dtv\nv62MjIzAcrfbLUnnnAmprq4OnFVxu91qaGjQyZMnL9jmbE6nU/Hx8UEFAACuXR0KKMYYzZ8/Xy+/\n/LI2bNig7OzsoPXZ2dlyu90qKSkJLGtoaFBpaanGjBkjScrLy1NERERQm8rKSpWXlwfaAACAHq69\nd+wYY8z9999vXC6X2bhxo6msrAzU6dOnA21WrlxpXC6Xefnll01ZWZm54447THp6uvH5fIE29913\nn8nIyDBvvfWWee+998ykSZPMiBEjTFNTE3fxUBRFUdQ1Xu25i6dDAeVCO1qzZk2gTUtLi1m+fLlx\nu93G6XSacePGmbKysqDt1NfXm/nz55ukpCQTFRVlZsyYYQ4cONDufhBQKIqiKKr7VnsCiu3T4NGt\n+Hw+uVyuUHcDAABcBq/Xe8n5pHwWDwAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwC\nCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAA\nsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwC\nCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsJwOBZSioiLdcMMNiouLU2pqqmbPnq09\ne/YEtbn77rtls9mCavTo0UFt/H6/FixYoOTkZMXExGjmzJk6ePDglR8NAAC4JnQooJSWlurBBx/U\nu+++q5KSEjU1NWnatGk6depUULubb75ZlZWVgVq3bl3Q+oULF2rt2rUqLi7W5s2bVVdXpxkzZqi5\nufnKjwgAAHR/5gpUV1cbSaa0tDSwbN68eWbWrFkX/JmamhoTERFhiouLA8sOHTpkwsLCzPr169u1\nX6/XayRRFHVWLVsm8847MuvWyfzgBzITJsj06iWTlCQTHy/jcIS+jz2lbrml9c/izTdlfvYzmdtv\n/8efhcslExkZ+j5SVKjK6/Ve8r3erivg9XolSUlJSUHLN27cqNTUVCUkJGj8+PH6j//4D6WmpkqS\ntm3bpsbGRk2bNi3Q3uPxKDc3V1u2bNH06dPP2Y/f75ff7w+89vl8V9Jt4Jplt0tRUa2VmiqNHy8Z\nI9XXSwcOSO+8I23fLjU3ty47erS10PnCw//xZ5GUJOXlSY88Ivn90pEj0l/+Ir39ttTSIp05I508\nKR06FOpeA50vIiJCqampSk5Olsvl0qZNm9r1c5cdUIwxWrRokW666Sbl5uYGlhcWFupLX/qSsrKy\ntG/fPv3rv/6rJk2apG3btsnpdKqqqkoOh0OJiYlB20tLS1NVVdV591VUVKRHH330crsK9Gg2mxQd\nLQ0e3FrGSI2N0okT0q5d0s6drYHF65X275fKykLd42uXzSZFRkpZWa31xS9KTU2tY//xx62hpblZ\nqquTDh5sDZNNTaHuNdBxHo9HgwYNUr9+/dS3b18NHTpUgwYNUlpampKTk9u1jcsOKPPnz9f777+v\nzZs3By2fO3du4Pvc3Fzl5+crKytLr732mm677bYLbs8YI5vNdt51S5cu1aJFiwKvfT6fMjMzL7fr\nQI9ms0kOh+R2t9bEia3/ij99uvVf9p980vqmePx4a4ApKWl900Tns9mkiAgpObm1brjhH2e8jh9v\nDS0NDZLPJ/3979Jbb7V+D1jRpEmTlJ+fr+uuu05ZWVnyeDxyu91yuVyBNh25AnJZAWXBggV69dVX\ntWnTJmVkZFy0bXp6urKysrR3715JktvtVkNDg06ePBl0FqW6ulpjxow57zacTqecTufldBXAJdhs\nrZcj4uJaq3//1jfJpqbWN8q5c6WvfjXUvewZbLbWiolprbZ/hzU1tV4Guusu6V/+RTrrvgTgqmm7\nOzcsLEw33HCDxo4dq3Hjxql///5KTExUTEyMoqKiFB4efsX76lBAMcZowYIFWrt2rTZu3Kjs7OxL\n/szx48dVUVGh9PR0SVJeXp4iIiJUUlKiOXPmSJIqKytVXl6uVatWXcYhALgSxrRW27yU2trWS0Bt\n8ySKi0Pdw57DmNavzc2tgaTtz+L4cWnHDukPfyCc4OpyOByKjo5WdHS00tPTNXHiRI0bN07jxo1T\nTExMIKxIuuBVkMvVoYDy4IMP6re//a3++7//W3FxcYE5Iy6XS1FRUaqrq9OKFSt0++23Kz09Xfv3\n79eyZcuUnJysL3zhC4G29957rxYvXqxevXopKSlJS5Ys0bBhwzRlypROPTgA52oLI6dPS1VV0uHD\nrW+CR4+2vglu2BDqHvYcxrReXquvl44d+8fltRMnpN27pfXrWy/xAFdTZmam0tLS1KtXLw0bNkyf\n//znlZ+f366TEp2pQwFl9erVkqQJEyYELV+zZo3uvvtuhYeHq6ysTM8//7xqamoCaet3v/ud4uLi\nAu2ffPJJ2e12zZkzR/X19Zo8ebKeffbZTjklBCCYMa1vckePtk6I/eij1jfBkydb5zWc9axFdKG2\nS2dtY//++61h0edrnaC8fXuoe4iexmazKTMzUzk5ORo4cKDcbrdGjhypAQMGKCsrSzExMSHrW4cv\n8VxMVFSU3njjjUtuJzIyUj/+8Y/14x//uCO7B9AObRNeP/5YKi1tfSNsuzOkqqr1X+e4OoxpvVRz\n6JC0ZYu0des/zpgcPSpVVoa6h+iJEhISlJeXp7y8POXn5yszM1PJyclKSUlRXFxc4JJNqF3Rc1AA\nWEtGxvf18MO/VHn5bjU2tgaVxsZQ96pnSkycqxdfjNALL/xajY2tQeXMmVD3Cj1JWFhYoIYPH65J\nkyZpwoQJGj58uKKiohQZGanIyEjLBJKzEVCAa4jdnqSTJx06dizUPUFYWLROnXLwIDxcNZGRkYqN\njVVsbKx69+6tG2+8UTfddJNGjx6tXr16BU1i7ewJrV2BgAIAQDfkdDqVlpYmj8ej3r17a/DgwRo5\ncqRGjhyp7Ozsbj+vk4ACAEA30bt3bw0ZMkSDBw9WTk6O+vXrp+zs7JBPaO0KBBQAACzM6XRqzJgx\n+upXv6ohQ4YEHtERGxtr2fkjnYGAAgCAxYSHh8vhcGjmzJm6//77df311ys6Olp2e8952+45RwoA\ngMUlJCSoT58+mjZtmr7xjW+oX79+gcfL9zQEFAAAQqxPnz7Kz8/XlClTdPPNN1/1p7ZaEQEFAIAQ\nCA8P19ChQ3XLLbdo7Nixys/PV3Jyco88W3I+BBQAAK6ym266Sffcc49uvPFG9e7d+5q7A6czEFAA\nAOhiERERio+P17hx4/TNb35TI0eOVFRUVI+a9NpRjAwAAF3E5XIpKytLU6dO1V133aURI0aEukvd\nBgEFAIBOZLPZlJGRodGjR2vChAmBSa/MLekYAgoAAJ1k6NChmjVrliZOnKjhw4crNTU11F3qtggo\nAABcAbvdrpEjR+r+++/X2LFjlZ6ertjY2FB3q9sjoAAA0EERERGKi4vT2LFjNX/+fI0dO1Z2u11h\nYWFcyukkBBQAANrJ5XKpf//+Gjt2rL7yla9oxIgR3f5Tg62KgAIAwCVkZGSooKBAEydO1JQpU5ST\nkxPqLl3zCCgAAJyHzWZT//79NWfOHE2aNElDhw5VWloal3CuEgIKAACfERYWppEjR+q+++5TYWGh\nEhMTFR0dTTC5yggoAIAez+FwKCEhQfn5+fp//+//ady4cXI6nQoLCwt113osAgoAoMdKSEhQTk6O\nbrzxRn35y1/WqFGjQt0lfIqAAgDocVJTUzVx4kRNnDhR48aN06BBgzhbYjEEFABAj9GvXz/NmTNH\nt9xyiwYMGKCUlBRuE7YoAgoAoNtom6h69teLrbPZbMrOztZDDz2kwsJCpaSkKCoqikmvFkdAAQB0\nufDwcIWHhyssLOyiXy+0LiYmRnFxcYGKjY1VfHx84PvPrjvfssjIyPOGGlgXAQUAcFF2u10Oh0MR\nERGX/Hq+ZU6nU9HR0ZesmJgYRUdHKyoqKvB922vmh/Q8BBQAgBwOh3r16qW0tLRApaamKjU1VdHR\n0XI6nXI4HHI6nedU2/LIyMjztrHbeatBx/G3BgB6kKioKGVlZSkrK0t9+/YNfO/xeBQXF3fOWQ3O\nXiBUCCgA0I3ZbDaFhYWdU8nJyRowYID69++vAQMGaMCAAcrJyVF6enpgrkd4eLjsdntgngdzM2Al\nHQooq1ev1urVq7V//35J0tChQ/Wd73xHhYWFkiRjjB599FH953/+p06ePKlRo0bpJz/5iYYOHRrY\nht/v15IlS/Tiiy+qvr5ekydP1jPPPKOMjIwOd97lcsnr9Xb45wCgO7HZbHI4HIqMjFRUVJQiIyMD\n5fF4lJ2dHai+ffsqOztbCQkJF9wW0B10KKBkZGRo5cqVGjBggCTpueee06xZs7R9+3YNHTpUq1at\n0hNPPKFnn31WAwcO1He/+11NnTpVe/bsUVxcnCRp4cKF+p//+R8VFxerV69eWrx4sWbMmKFt27Z1\n+F7073znO3rjjTe0ZcsW1dXVdehnAcBqbDabEhMTlZCQEFRJSUlKS0uT2+1Wenq6PB5P4PvIyMhQ\ndxvoEjZjjLmSDSQlJenxxx/XPffcI4/Ho4ULF+pb3/qWpNazJWlpafre976nb3zjG/J6vUpJSdEL\nL7yguXPnSpIOHz6szMxMrVu3TtOnT2/XPn0+n1wul2pqalRTU6MNGzboxRdfVElJyZUcCtDt/epX\nv9JTTz2lv/3tb6HuSo/31a9+VQ6HQz/72c/OWRcZGRkIGG63O+j7hISEQEhJTExUYmKi4uLiFBER\nEYKjADpX2/u31+tVfHz8RdtedkBpbm7WH/7wB82bN0/bt29XZGSk+vfvr/fee0/XX399oN2sWbOU\nkJCg5557Ths2bNDkyZN14sQJJSYmBtqMGDFCs2fP1qOPPnreffn9fvn9/qADzMzMDBxgc3OzDh06\npE2bNumJJ57Q9u3bL+eQ0EEOh0NTp07VQw89pGPHjmn37t3avXu3PvjgA33wwQdqamoKdRd7nOzs\nbFVVVam+vj7UXenxkpOTlZSUpOjo6HMuv6SlpSkmJibokk1UVJScTieXYHBN60hA6fAk2bKyMhUU\nFOjMmTOKjY3V2rVrNWTIEG3ZskWSlJaWFtQ+LS1Nn3zyiSSpqqpKDocjKJy0tamqqrrgPouKii4Y\nXqTWBwBlZmbqjjvu0PTp01VcXKwf/vCHOnjwoBoaGjp6iLiI8PBwxcbG6qabbtLDDz+sG264QZGR\nkWppaVFLS4uam5sDtXv3bpWVlamsrEzl5eUqKytTQ0ODmpqaAtXc3BzqQ+oW2h5Y1Tah0W63y+Vy\nacCAAbruuusCNWjQIMXGxoa6u/iUzWYLTED9bNlsNoIIcAkdDiiDBg3Sjh07VFNTo5deeknz5s1T\naWlpYP3Z/9MZYy75P+Kl2ixdulSLFi0KvG47g/JZbb8IUlJStGDBAn3xi1/U6tWr9V//9V/65JNP\ndPr06Y4cJs7icDjUu3dvjR49Wl//+tc1YcKEoPVtdwR89jT0qFGjgj4ZtKWlRR9++KH27NkTVLW1\ntaqrq1NdXZ1qa2t16tSpq3VYltP2xMzPVnx8vDwej/r376/+/fsrJydH/fr1k9vt5vkSAK5ZHf7t\n5nA4ApNk8/PztXXrVj311FOBeSdVVVVKT08PtK+urg6cVXG73WpoaNDJkyeDzqJUV1drzJgxF9xn\n28N+OiI9PV0rVqzQl770Jf3+97/Xa6+9pvLycjU2NnZoOz2dzWbT8OHDNWnSJN16660qKCi47El5\nYWFhGjhwoAYOHKhbb71VUmtoOXTokA4cOKCDBw+qoqJCBw4cUHV1daCOHDmiY8eOdeZhWYLL5VJy\ncnJQpaSkqHfv3vJ4PPJ4POrdu7fS0tIUHR0d6u4CwFV1xf/8MsbI7/crOztbbrdbJSUlgTkoDQ0N\nKi0t1fe+9z1JUl5eniIiIlRSUqI5c+ZIkiorK1VeXq5Vq1ZdaVfOERYWpmHDhmnAgAGaPXu21q9f\nrzVr1uijjz7q9H1di66//nrdcccdmjBhgoYMGaKYmJhO30dYWJgyMzODzoi1tLToxIkTOnHihI4f\nP64TJ07o6NGj2r9/v/bt2xf4WlFR0en96QqRkZGBY+zTp0/ga3Jy8nknQwIAOhhQli1bpsLCQmVm\nZqq2tlbFxcXauHGj1q9fL5vNpoULF+qxxx5TTk6OcnJy9Nhjjyk6Olp33nmnpNZ/Md57771avHix\nevXqpaSkJC1ZskTDhg3TlClTuuQApdYnJ+bl5em6667Tbbfdpueff16//OUvdfTo0S7bZ3fWv39/\nfetb39LkyZPVu3dvORyOq3q9vO0hU8nJyYFlxhjV19cHlc/n0wcffKCdO3dq165d2rlzpz766CNd\n4Y1pl81msykzM1MDBw7UoEGDlJOTo0GDBikjI0PR0dGB51a0PRKcj3gHgAvrUEA5cuSI7rrrLlVW\nVsrlcmn48OFav369pk6dKkl6+OGHVV9frwceeCDwoLY333wz6F+FTz75pOx2u+bMmRN4UNuzzz57\nVX5ZR0dHa/Dgwfr3f/93feUrX9Hjjz+u1157TceOHVNLS0uX79+qbDabYmJilJmZqUceeURz586V\n3W631JMlbTZb4NHbbYwxuv7669XS0iJjjFpaWlRfX6+dO3cGJua+//77+vDDD9XQ0KDGxkY1Njaq\noaGhQ5Nz2+bWtJXdbldSUpIGDBigIUOGaPDgwRo8eLAGDRqkmJiYwJM92yZCMiESADruip+DEgod\nuU3pUrZs2aIf/ehHevfdd1VRUdGjgorNZlNaWppGjhypL33pS7rzzjuvyYc+nTlzRnv37g3U3//+\nd33yySfy+Xzyer3y+Xzy+Xzy+/3nfGS7y+VS7969lZ2drX79+qlfv37Kzs5WSkoKZ0AAoIOuynNQ\nQqkzA4rU+pyVt956S6+88or++Mc/XvSW52tFRkaGpkyZounTp2vSpElKTU0NdZeuKr/fr8OHD+vg\nwYOBr3V1dec8OCslJeWaDG0AEAoElMt04sQJ/e1vf9Nvf/tbrVmz5pp8Rkfv3r01Z84c3XLLLRox\nYkTQPA8AALoSAeUK1dTUaO/evVq+fLneeOONbn/Zp23+xgMPPKC77rpLffv2VUxMDB+hDgC4qggo\nncAYo+bmZm3YsEErVqzQrl27ut0nJ0dGRqpXr16644479PDDDyspKclSE18BAD0LAaWTNTQ06De/\n+Y1eeOEFvf/++zp+/HiX7/NKxMbGavjw4ZoyZYruueceZWVlhbpLAAAQULqCMUbV1dX64x//qLVr\n12rDhg2W+0A2l8ulG2+8Uf/0T/+kqVOnauDAgaHuEgAAAQSULtTU1KRDhw7pz3/+s5555hm98847\nV3X/5xMZGanCwkLdeeedys/PV58+fZhfAgCwnC79NOOezm63KysrS71799bkyZP1+uuva9WqVfrg\ngw+u6l0/NptNdrtdhYWFWrJkiYYPH67Y2FiezQEAuCZwBuUKtA3d0aNHtWbNGj3//PPat29fl176\nCQsLk9vt1o033qhFixYpPz9f4eHhTHwFAFgel3hC5KOPPtKzzz6r119/Xbt27erUoGK32zVo0CAV\nFBTorrvuUkFBgSIiIjpt+wAAdDUCSgg1NTWprKxM69at0+9//3uVl5df0XNUIiIiNHz4cM2aNUuT\nJk3SmDFjOFsCAOiWCCgWcPr0ae3fv1+vvPKKfvSjH+nIkSMd3saIESP09a9/XZMmTVLfvn155DoA\noFsjoFhIQ0ODqqqq9OSTT+rXv/61Tpw4ccEzKjabTQ6HQ0OHDtWCBQs0a9YsxcbGcikHAHBNIKBY\nTNsQl5WV6fvf/742bdqkAwcO6LNDn5KSoiFDhuiee+7RrFmzFB8fz6UcAMA1hYBicf/7v/+rF154\nQaWlpWpsbNTo0aM1e/ZszZgxQwkJCaHuHgAAXYLnoFjc5MmTVVBQoD/96U9qbGxUQUGBEhISOGMC\nAMCnCCghEh0dralTp4a6GwAAWBLPQwcAAJZDQAEAAJZDQAEAAJZDQAEAAJZDQAEAAJZDQAEAAJZD\nQAEAAJZDQAEAAJZDQAEAAJZDQAEAAJZDQAEAAJZDQAEAAJZDQAEAAJbToYCyevVqDR8+XPHx8YqP\nj1dBQYFef/31wPq7775bNpstqEaPHh20Db/frwULFig5OVkxMTGaOXOmDh482DlHAwAArgkdCigZ\nGRlauXKl/vrXv+qvf/2rJk2apFmzZmnnzp2BNjfffLMqKysDtW7duqBtLFy4UGvXrlVxcbE2b96s\nuro6zZgxQ83NzZ1zRAAAoNuzGWPMlWwgKSlJjz/+uO69917dfffdqqmp0SuvvHLetl6vVykpKXrh\nhRc0d+5cSdLhw4eVmZmpdevWafr06e3ap8/nk8vlktfrVXx8/JV0HwAAXCUdef++7Dkozc3NKi4u\n1qlTp1RQUBBYvnHjRqWmpmrgwIH62te+purq6sC6bdu2qbGxUdOmTQss83g8ys3N1ZYtWy64L7/f\nL5/PF1QAAODa1eGAUlZWptjYWDmdTt13331au3athgwZIkkqLCzUb37zG23YsEE/+MEPtHXrVk2a\nNEl+v1+SVFVVJYfDocTExKBtpqWlqaqq6oL7LCoqksvlClRmZmZHuw0AALoRe0d/YNCgQdqxY4dq\namr00ksvad68eSotLdWQIUMCl20kKTc3V/n5+crKytJrr72m22677YLbNMbIZrNdcP3SpUu1aNGi\nwGufz0dIAQDgGtbhgOJwODRgwABJUn5+vrZu3aqnnnpKP/vZz85pm56erqysLO3du1eS5Ha71dDQ\noJMnTwadRamurtaYMWMuuE+n0ymn09nRrgIAgG7qip+DYowJXMI52/Hjx1VRUaH09HRJUl5eniIi\nIlRSUhJoU1lZqfLy8osGFAAA0LN06AzKsmXLVFhYqMzMTNXW1qq4uFgbN27U+vXrVVdXpxUrVuj2\n229Xenq69u/fr2XLlik5OVlf+MIXJEkul0v33nuvFi9erF69eikpKUlLlizRsGHDNGXKlC45QAAA\n0P10KKAcOXJEd911lyorK+VyuTR8+HCtX79eU6dOVX19vcrKyvT888+rpqZG6enpmjhxon73u98p\nLi4usI0nn3xSdrtdc+bMUX19vSZPnqxnn31W4eHhnX5wAACge7ri56CEAs9BAQCg+7kqz0EBAADo\nKgQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQU\nAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABg\nOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQU\nAABgOQQUAABgOQQUAABgOQQUAABgOfZQd+ByGGMkST6fL8Q9AQAA7dX2vt32Pn4x3TKg1NbWSpIy\nMzND3BMAANBRtbW1crlcF21jM+2JMRbT0tKiPXv2aMiQIaqoqFB8fHyou9Rt+Xw+ZWZmMo6dgLHs\nPIxl52AcOw9j2TmMMaqtrZXH41FY2MVnmXTLMyhhYWHq3bu3JCk+Pp6/LJ2Acew8jGXnYSw7B+PY\neRjLK3epMydtmCQLAAAsh4ACAAAsp9sGFKfTqeXLl8vpdIa6K90a49h5GMvOw1h2Dsax8zCWV1+3\nnCQLAACubd32DAoAALh2EVAAAIDlEFAAAIDlEFAAAIDldMuA8swzzyg7O1uRkZHKy8vTO++8E+ou\nWc6mTZt06623yuPxyGaz6ZVXXglab4zRihUr5PF4FBUVpQkTJmjnzp1Bbfx+vxYsWKDk5GTFxMRo\n5syZOnjw4FU8itArKirSDTfcoLi4OKWmpmr27Nnas2dPUBvGsn1Wr16t4cOHBx50VVBQoNdffz2w\nnnG8PEVFRbLZbFq4cGFgGWPZPitWrJDNZgsqt9sdWM84hpjpZoqLi01ERIT5+c9/bnbt2mUeeugh\nExMTYz755JNQd81S1q1bZ7797W+bl156yUgya9euDVq/cuVKExcXZ1566SVTVlZm5s6da9LT043P\n5wu0ue+++0zv3r1NSUmJee+998zEiRPNiBEjTFNT01U+mtCZPn26WbNmjSkvLzc7duwwt9xyi+nT\np4+pq6sLtGEs2+fVV181r732mtmzZ4/Zs2ePWbZsmYmIiDDl5eXGGMbxcvzlL38xffv2NcOHDzcP\nPfRQYDlj2T7Lly83Q4cONZWVlYGqrq4OrGccQ6vbBZTPf/7z5r777gtaNnjwYPPII4+EqEfWd3ZA\naWlpMW6326xcuTKw7MyZM8blcpmf/vSnxhhjampqTEREhCkuLg60OXTokAkLCzPr16+/an23murq\naiPJlJaWGmMYyyuVmJhofvGLXzCOl6G2ttbk5OSYkpISM378+EBAYSzbb/ny5WbEiBHnXcc4hl63\nusTT0NCgbdu2adq0aUHLp02bpi1btoSoV93Pvn37VFVVFTSOTqdT48ePD4zjtm3b1NjYGNTG4/Eo\nNze3R4+11+uVJCUlJUliLC9Xc3OziouLderUKRUUFDCOl+HBBx/ULbfcoilTpgQtZyw7Zu/evfJ4\nPMrOztaXv/xlffzxx5IYRyvoVh8WeOzYMTU3NystLS1oeVpamqqqqkLUq+6nbazON46ffPJJoI3D\n4VBiYuI5bXrqWBtjtGjRIt10003Kzc2VxFh2VFlZmQoKCnTmzBnFxsZq7dq1GjJkSOCXOePYPsXF\nxXrvvfe0devWc9bxd7L9Ro0apeeff14DBw7UkSNH9N3vfldjxozRzp07GUcL6FYBpY3NZgt6bYw5\nZxku7XLGsSeP9fz58/X+++9r8+bN56xjLNtn0KBB2rFjh2pqavTSSy9p3rx5Ki0tDaxnHC+toqJC\nDz30kN58801FRkZesB1jeWmFhYWB74cNG6aCggL1799fzz33nEaPHi2JcQylbnWJJzk5WeHh4eck\n0+rq6nNSLi6sbZb6xcbR7XaroaFBJ0+evGCbnmTBggV69dVX9fbbbysjIyOwnLHsGIfDoQEDBig/\nP19FRUUaMWKEnnrqKcaxA7Zt26bq6mrl5eXJbrfLbrertLRUP/rRj2S32wNjwVh2XExMjIYNG6a9\ne/fyd9ICulVAcTgcysvLU0lJSdDykpISjRkzJkS96n6ys7PldruDxrGhoUGlpaWBcczLy1NERERQ\nm8rKSpWXl/eosTbGaP78+Xr55Ze1YcMGZWdnB61nLK+MMUZ+v59x7IDJkyerrKxMO3bsCFR+fr7+\n+Z//WTt27FC/fv0Yy8vk9/u1e/dupaen83fSCkIxM/dKtN1m/Mtf/tLs2rXLLFy40MTExJj9+/eH\numuWUltba7Zv3262b99uJJknnnjCbN++PXA79sqVK43L5TIvv/yyKSsrM3fcccd5b5/LyMgwb731\nlnnvvffMpEmTetztc/fff79xuVxm48aNQbcinj59OtCGsWyfpUuXmk2bNpl9+/aZ999/3yxbtsyE\nhYWZN9980xjDOF6Jz97FYwxj2V6LFy82GzduNB9//LF59913zYwZM0xcXFzg/YRxDK1uF1CMMeYn\nP/mJycrKMg6Hw3zuc58L3PKJf3j77beNpHNq3rx5xpjWW+iWL19u3G63cTqdZty4caasrCxoG/X1\n9Wb+/PkmKSnJREVFmRkzZpgDBw6E4GhC53xjKMmsWbMm0IaxbJ977rkn8P9tSkqKmTx5ciCcGMM4\nXomzAwpj2T5tzzWJiIgwHo/H3HbbbWbnzp2B9YxjaNmMMSY0524AAADOr1vNQQEAAD0DAQUAAFgO\nAQUAAFgOAQUAAFgOAQUAAFgOAQUAAFgOAQUAAFgOAQUAAFgOAQUAAFgOAQUAAFgOAQUAAFgOAQUA\nAFjO/wcIaFm37RUddQAAAABJRU5ErkJggg==\n"
          }
        }
      ],
      "source": [
        "plt.imshow(env.render())"
      ],
      "id": "d97c051d-a9be-4caf-ad95-963358cee7eb"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`-` env.step"
      ],
      "id": "d9d4111a-e045-4c90-90f2-8094173872f7"
    },
    {
      "cell_type": "code",
      "execution_count": 92,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "env.step??"
      ],
      "id": "210ce01d-0a07-48c1-85d6-707f33e7d878"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "-   리턴되는 값은 (다음상태, 보상, 종료여부, 중단여부, 추가정보) 임.."
      ],
      "id": "13e7a80c-7136-4a46-9b4e-2e9a5c5591e2"
    },
    {
      "cell_type": "code",
      "execution_count": 93,
      "metadata": {
        "tags": []
      },
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAF7CAYAAAD4/3BBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90\nbGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9h\nAAAPYQGoP6dpAAA1aElEQVR4nO3de3xU9Z3/8fckk5ncJwm5TIaEECCAEC42sRCU+82sCFRbqO5a\nrD7aeoGfFHhYoY8tuNs1FFutrZV2e8FLW9N2FdetiMZFgpS6pQg1AaSoIAESwi0zCYTJ7fv7I2bq\ncE0gYU7I6+nj80jmnG/O+Z4vmHlzzvecsRljjAAAACwkLNQdAAAAOBsBBQAAWA4BBQAAWA4BBQAA\nWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWA4BBQAAWE5IA8ozzzyj7OxsRUZGKi8vT++8\n804ouwMAACwiZAHld7/7nRYuXKhvf/vb2r59u8aOHavCwkIdOHAgVF0CAAAWYQvVhwWOGjVKn/vc\n57R69erAsuuuu06zZ89WUVFRKLoEAAAswh6KnTY0NGjbtm165JFHgpZPmzZNW7ZsOae93++X3+8P\nvG5padGJEyfUq1cv2Wy2Lu8vAAC4csYY1dbWyuPxKCzs4hdxQhJQjh07pubmZqWlpQUtT0tLU1VV\n1Tnti4qK9Oijj16t7gEAgC5UUVGhjIyMi7YJSUBpc/bZD2PMec+ILF26VIsWLQq89nq96tOnT5f3\nD8Cl3Tr+e4pJTFRm/GiFhzlVU/+JDhz9szb86QnFRqdo1pTvyREeL1dk1/8/a0yzKr079Pa2x1VR\nsV3jblggd/p16pswVmFhDh07vUsb/vp9ffzxn7q8LwAuLC4u7pJtQhJQkpOTFR4efs7Zkurq6nPO\nqkiS0+mU0+m8Wt0D0E7DB9ymmIQEpcePVLSzl5pa6tVoarX7wzflb6hVfIxbjogYOe0xinRc+hfS\nlWoxzYqIiJLN1nrq+G+712pQvymqbzmuXpED5baP1LBBM3T4cJnOnPF1eX8AnF97pmeE5C4eh8Oh\nvLw8lZSUBC0vKSnRmDFjQtElAB1kU5ic0bGKi3IrxpEsSTrVWK09+0t0pHq3jGmRJJlPW181n5n3\n7607pI/3/1l1DVVqbD4te5hTWb3Gqn8/fs8AVhey24wXLVqkX/ziF/rVr36l3bt365vf/KYOHDig\n++67L1RdAtAB/TJu0sghX1Sc06Nwm1NNLfU6UrNTBw/v0OkzJyS1hZOrywTt2egv5WsUZpzy+itk\nTIsSorI0sO9UuVyeEPQOQHuFbA7K3Llzdfz4cf3bv/2bKisrlZubq3Xr1ikrKytUXQLQTnZ7pNJS\nByvMFq5oey9JUq3/sD46WKqDlTs+09Lo6seUFn326Qln/D7t2vu6hl03Qw3Np+QIj1WfpAJlZ31e\nZTtfU3Nz41XuH4D2COkk2QceeEAPPPBAKLsAoMNsmjb6X5WROULREcmyh0eqsbleFSf+T/v2v6um\n5jMh7d3ZcaixqV6fHPo/9cnIV5Q9SY7wGMVFunVd31t0pHqvKqt2hqSfAC6Oz+IB0GE52ZPU3OJX\nnDNdknT09C5VHtmtoyc+DG5oWu+saW5pUFPzGTU1+9XU4ldTS4OaWxo/rSa1mCa1mGa1mBYZE3wG\npMPMuWdtqk/s0ZEju+XzH1azaVC4zSl3wnClp+XKbmcCPmBFIT2DAqD7uXfWf+tE/d+VHvc5hdns\nqm88oRN1H+mvf/vNOW1bTKN89ZU61bhDYbbP/roxn4kQ5wkjtrYvrf9JYZLNJput7fWnXz993XrX\nTusP+Rt9ioiICtpcU/MZ7T/4f0pJHaAaxwH1ispRjCNVIwbepgMHt+rY8Y+vbFAAdDoCCoB2S4zL\n0hlzUlF2l6LsSWoxTao+tUvlu9epqcl/TvvjNfv04itflz3cofAwh+zhToWHO1pfhztlD/902afr\nWpe3tfvH6/CwtrYRCm/bRliEwsIjPv3qUFi4XeFhDvlqjqmiYvs5fdlfuUXDamfruPPvSojso3Cb\nUwlRWerX90Z5fYfV2BjaS1MAghFQALTbxPwlinA6FGaz63TjMTW21OvQke36+MDFHnxmWi/tNPvl\nb6y9an09n227f6vC8ctVfapcnrh8Oe0uZWcV6MChv6qqandI+wYgGHNQALTbtg9+raNH9ivSHq+a\nM/t12LdNH3z4phob6kPdtXY5fHSHqqv36kyjV4dr39Mnx99R2a5XderU8VB3DcBZQvZpxlfC5/PJ\n5XKFuhtAjxQdmaTkhAG6Ifcr8p2p1Dtbf6wz/u7zVNb4mHT9y6zn9d6Hv9buD95U3alj3GoMXGVe\nr1fx8fEXbUNAAXBZwsMckoyaW7rfm3uEPVrNLX61tDSHuitAj9SegMIcFACXpbmlIdRduGyNTadD\n3QUAl8AcFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAA\nYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkE\nFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDmdHlBWrFghm80WVG63O7DeGKMVK1bI4/EoKipKEyZM0M6d\nOzu7GwAAoBvrkjMoQ4cOVWVlZaDKysoC61atWqUnnnhCTz/9tLZu3Sq3262pU6eqtra2K7oCAAC6\noS4JKHa7XW63O1ApKSmSWs+e/PCHP9S3v/1t3XbbbcrNzdVzzz2n06dP67e//W1XdAUAAHRDXRJQ\n9u7dK4/Ho+zsbH35y1/Wxx9/LEnat2+fqqqqNG3atEBbp9Op8ePHa8uWLRfcnt/vl8/nCyoAAHDt\n6vSAMmrUKD3//PN644039POf/1xVVVUaM2aMjh8/rqqqKklSWlpa0M+kpaUF1p1PUVGRXC5XoDIz\nMzu72wAAwEJsxhjTlTs4deqU+vfvr4cfflijR4/WjTfeqMOHDys9PT3Q5mtf+5oqKiq0fv36827D\n7/fL7/cHXvt8PkIKAADdlNfrVXx8/EXbdPltxjExMRo2bJj27t0buJvn7LMl1dXV55xV+Syn06n4\n+PigAgAA164uDyh+v1+7d+9Wenq6srOz5Xa7VVJSEljf0NCg0tJSjRkzpqu7AgAAugl7Z29wyZIl\nuvXWW9WnTx9VV1fru9/9rnw+n+bNmyebzaaFCxfqscceU05OjnJycvTYY48pOjpad955Z2d3BQAA\ndFOdHlAOHjyoO+64Q8eOHVNKSopGjx6td999V1lZWZKkhx9+WPX19XrggQd08uRJjRo1Sm+++abi\n4uI6uysAAKCb6vJJsl3B5/PJ5XKFuhsAAOAyWGKSLAAAQEcRUAAAgOUQUAAAgOUQUAAAgOUQUAAA\ngOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQ\nUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAA\ngOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOUQUAAAgOV0OKBs2rRJt956qzwe\nj2w2m1555ZWg9cYYrVixQh6PR1FRUZowYYJ27twZ1Mbv92vBggVKTk5WTEyMZs6cqYMHD17RgQAA\ngGtHhwPKqVOnNGLECD399NPnXb9q1So98cQTevrpp7V161a53W5NnTpVtbW1gTYLFy7U2rVrVVxc\nrM2bN6uurk4zZsxQc3Pz5R8JAAC4dpgrIMmsXbs28LqlpcW43W6zcuXKwLIzZ84Yl8tlfvrTnxpj\njKmpqTERERGmuLg40ObQoUMmLCzMrF+/vl379Xq9RhJFURRFUd2wvF7vJd/rO3UOyr59+1RVVaVp\n06YFljmdTo0fP15btmyRJG3btk2NjY1BbTwej3JzcwNtzub3++Xz+YIKAABcuzo1oFRVVUmS0tLS\ngpanpaUF1lVVVcnhcCgxMfGCbc5WVFQkl8sVqMzMzM7sNgAAsJguuYvHZrMFvTbGnLPsbBdrs3Tp\nUnm93kBVVFR0Wl8BAID1dGpAcbvdknTOmZDq6urAWRW3262GhgadPHnygm3O5nQ6FR8fH1QAAODa\n1akBJTs7W263WyUlJYFlDQ0NKi0t1ZgxYyRJeXl5ioiICGpTWVmp8vLyQBsAANCz2Tv6A3V1dfrw\nww8Dr/ft26cdO3YoKSlJffr00cKFC/XYY48pJydHOTk5euyxxxQdHa0777xTkuRyuXTvvfdq8eLF\n6tWrl5KSkrRkyRINGzZMU6ZM6bwjAwAA3Ve77uv9jLfffvu8twzNmzfPGNN6q/Hy5cuN2+02TqfT\njBs3zpSVlQVto76+3syfP98kJSWZqKgoM2PGDHPgwIF294HbjCmKoiiq+1Z7bjO2GWOMuhmfzyeX\nyxXqbgAAgMvg9XovOZ+Uz+IBAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACW\nQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0AB\nAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACW\nQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACW0+GAsmnTJt16663yeDyy2Wx65ZVXgtbffffdstlsQTV6\n9OigNn6/XwsWLFBycrJiYmI0c+ZMHTx48IoOBAAAXDs6HFBOnTqlESNG6Omnn75gm5tvvlmVlZWB\nWrduXdD6hQsXau3atSouLtbmzZtVV1enGTNmqLm5ueNHAAAArjn2jv5AYWGhCgsLL9rG6XTK7Xaf\nd53X69Uvf/lLvfDCC5oyZYok6de//rUyMzP11ltvafr06R3tEgAAuMZ0yRyUjRs3KjU1VQMHDtTX\nvvY1VVdXB9Zt27ZNjY2NmjZtWmCZx+NRbm6utmzZct7t+f1++Xy+oAIAANeuTg8ohYWF+s1vfqMN\nGzboBz/4gbZu3apJkybJ7/dLkqqqquRwOJSYmBj0c2lpaaqqqjrvNouKiuRyuQKVmZnZ2d0GAAAW\n0uFLPJcyd+7cwPe5ubnKz89XVlaWXnvtNd12220X/DljjGw223nXLV26VIsWLQq89vl8hBQAAK5h\nXX6bcXp6urKysrR3715JktvtVkNDg06ePBnUrrq6WmlpaefdhtPpVHx8fFABAIBrV5cHlOPHj6ui\nokLp6emSpLy8PEVERKikpCTQprKyUuXl5RozZkxXdwcAAHQDHb7EU1dXpw8//DDwet++fdqxY4eS\nkpKUlJSkFStW6Pbbb1d6err279+vZcuWKTk5WV/4whckSS6XS/fee68WL16sXr16KSkpSUuWLNGw\nYcMCd/UAAIAeznTQ22+/bSSdU/PmzTOnT58206ZNMykpKSYiIsL06dPHzJs3zxw4cCBoG/X19Wb+\n/PkmKSnJREVFmRkzZpzT5mK8Xu95+0BRFEVRlPXL6/Ve8r3eZowx6mZ8Pp9cLleouwEAAC6D1+u9\n5HxSPosHAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABY\nDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEF\nAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABYDgEFAABY\nDgEFAABYDgEFAABYTocCSlFRkW644QbFxcUpNTVVs2fP1p49e4LaGGO0YsUKeTweRUVFacKECdq5\nc2dQG7/frwULFig5OVkxMTGaOXOmDh48eOVHAwAArgkdCiilpaV68MEH9e6776qkpERNTU2aNm2a\nTp06FWizatUqPfHEE3r66ae1detWud1uTZ06VbW1tYE2Cxcu1Nq1a1VcXKzNmzerrq5OM2bMUHNz\nc+cdGQAA6L7MFaiurjaSTGlpqTHGmJaWFuN2u83KlSsDbc6cOWNcLpf56U9/aowxpqamxkRERJji\n4uJAm0OHDpmwsDCzfv36du3X6/UaSRRFURRFdcPyer2XfK+/ojkoXq9XkpSUlCRJ2rdvn6qqqjRt\n2rRAG6fTqfHjx2vLli2SpG3btqmxsTGojcfjUW5ubqDN2fx+v3w+X1ABAIBr12UHFGOMFi1apJtu\nukm5ubmSpKqqKklSWlpaUNu0tLTAuqqqKjkcDiUmJl6wzdmKiorkcrkClZmZebndBgAA3cBlB5T5\n8+fr/fff14svvnjOOpvNFvTaGHPOsrNdrM3SpUvl9XoDVVFRcbndBgAA3cBlBZQFCxbo1Vdf1dtv\nv62MjIzAcrfbLUnnnAmprq4OnFVxu91qaGjQyZMnL9jmbE6nU/Hx8UEFAACuXR0KKMYYzZ8/Xy+/\n/LI2bNig7OzsoPXZ2dlyu90qKSkJLGtoaFBpaanGjBkjScrLy1NERERQm8rKSpWXlwfaAACAHq69\nd+wYY8z9999vXC6X2bhxo6msrAzU6dOnA21WrlxpXC6Xefnll01ZWZm54447THp6uvH5fIE29913\nn8nIyDBvvfWWee+998ykSZPMiBEjTFNTE3fxUBRFUdQ1Xu25i6dDAeVCO1qzZk2gTUtLi1m+fLlx\nu93G6XSacePGmbKysqDt1NfXm/nz55ukpCQTFRVlZsyYYQ4cONDufhBQKIqiKKr7VnsCiu3T4NGt\n+Hw+uVyuUHcDAABcBq/Xe8n5pHwWDwAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwC\nCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAA\nsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwC\nCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsJwOBZSioiLdcMMNiouLU2pqqmbPnq09\ne/YEtbn77rtls9mCavTo0UFt/H6/FixYoOTkZMXExGjmzJk6ePDglR8NAAC4JnQooJSWlurBBx/U\nu+++q5KSEjU1NWnatGk6depUULubb75ZlZWVgVq3bl3Q+oULF2rt2rUqLi7W5s2bVVdXpxkzZqi5\nufnKjwgAAHR/5gpUV1cbSaa0tDSwbN68eWbWrFkX/JmamhoTERFhiouLA8sOHTpkwsLCzPr169u1\nX6/XayRRFHVWLVsm8847MuvWyfzgBzITJsj06iWTlCQTHy/jcIS+jz2lbrml9c/izTdlfvYzmdtv\n/8efhcslExkZ+j5SVKjK6/Ve8r3erivg9XolSUlJSUHLN27cqNTUVCUkJGj8+PH6j//4D6WmpkqS\ntm3bpsbGRk2bNi3Q3uPxKDc3V1u2bNH06dPP2Y/f75ff7w+89vl8V9Jt4Jplt0tRUa2VmiqNHy8Z\nI9XXSwcOSO+8I23fLjU3ty47erS10PnCw//xZ5GUJOXlSY88Ivn90pEj0l/+Ir39ttTSIp05I508\nKR06FOpeA50vIiJCqampSk5Olsvl0qZNm9r1c5cdUIwxWrRokW666Sbl5uYGlhcWFupLX/qSsrKy\ntG/fPv3rv/6rJk2apG3btsnpdKqqqkoOh0OJiYlB20tLS1NVVdV591VUVKRHH330crsK9Gg2mxQd\nLQ0e3FrGSI2N0okT0q5d0s6drYHF65X275fKykLd42uXzSZFRkpZWa31xS9KTU2tY//xx62hpblZ\nqquTDh5sDZNNTaHuNdBxHo9HgwYNUr9+/dS3b18NHTpUgwYNUlpampKTk9u1jcsOKPPnz9f777+v\nzZs3By2fO3du4Pvc3Fzl5+crKytLr732mm677bYLbs8YI5vNdt51S5cu1aJFiwKvfT6fMjMzL7fr\nQI9ms0kOh+R2t9bEia3/ij99uvVf9p980vqmePx4a4ApKWl900Tns9mkiAgpObm1brjhH2e8jh9v\nDS0NDZLPJ/3979Jbb7V+D1jRpEmTlJ+fr+uuu05ZWVnyeDxyu91yuVyBNh25AnJZAWXBggV69dVX\ntWnTJmVkZFy0bXp6urKysrR3715JktvtVkNDg06ePBl0FqW6ulpjxow57zacTqecTufldBXAJdhs\nrZcj4uJaq3//1jfJpqbWN8q5c6WvfjXUvewZbLbWiolprbZ/hzU1tV4Guusu6V/+RTrrvgTgqmm7\nOzcsLEw33HCDxo4dq3Hjxql///5KTExUTEyMoqKiFB4efsX76lBAMcZowYIFWrt2rTZu3Kjs7OxL\n/szx48dVUVGh9PR0SVJeXp4iIiJUUlKiOXPmSJIqKytVXl6uVatWXcYhALgSxrRW27yU2trWS0Bt\n8ySKi0Pdw57DmNavzc2tgaTtz+L4cWnHDukPfyCc4OpyOByKjo5WdHS00tPTNXHiRI0bN07jxo1T\nTExMIKxIuuBVkMvVoYDy4IMP6re//a3++7//W3FxcYE5Iy6XS1FRUaqrq9OKFSt0++23Kz09Xfv3\n79eyZcuUnJysL3zhC4G29957rxYvXqxevXopKSlJS5Ys0bBhwzRlypROPTgA52oLI6dPS1VV0uHD\nrW+CR4+2vglu2BDqHvYcxrReXquvl44d+8fltRMnpN27pfXrWy/xAFdTZmam0tLS1KtXLw0bNkyf\n//znlZ+f366TEp2pQwFl9erVkqQJEyYELV+zZo3uvvtuhYeHq6ysTM8//7xqamoCaet3v/ud4uLi\nAu2ffPJJ2e12zZkzR/X19Zo8ebKeffbZTjklBCCYMa1vckePtk6I/eij1jfBkydb5zWc9axFdKG2\nS2dtY//++61h0edrnaC8fXuoe4iexmazKTMzUzk5ORo4cKDcbrdGjhypAQMGKCsrSzExMSHrW4cv\n8VxMVFSU3njjjUtuJzIyUj/+8Y/14x//uCO7B9AObRNeP/5YKi1tfSNsuzOkqqr1X+e4OoxpvVRz\n6JC0ZYu0des/zpgcPSpVVoa6h+iJEhISlJeXp7y8POXn5yszM1PJyclKSUlRXFxc4JJNqF3Rc1AA\nWEtGxvf18MO/VHn5bjU2tgaVxsZQ96pnSkycqxdfjNALL/xajY2tQeXMmVD3Cj1JWFhYoIYPH65J\nkyZpwoQJGj58uKKiohQZGanIyEjLBJKzEVCAa4jdnqSTJx06dizUPUFYWLROnXLwIDxcNZGRkYqN\njVVsbKx69+6tG2+8UTfddJNGjx6tXr16BU1i7ewJrV2BgAIAQDfkdDqVlpYmj8ej3r17a/DgwRo5\ncqRGjhyp7Ozsbj+vk4ACAEA30bt3bw0ZMkSDBw9WTk6O+vXrp+zs7JBPaO0KBBQAACzM6XRqzJgx\n+upXv6ohQ4YEHtERGxtr2fkjnYGAAgCAxYSHh8vhcGjmzJm6//77df311ys6Olp2e8952+45RwoA\ngMUlJCSoT58+mjZtmr7xjW+oX79+gcfL9zQEFAAAQqxPnz7Kz8/XlClTdPPNN1/1p7ZaEQEFAIAQ\nCA8P19ChQ3XLLbdo7Nixys/PV3Jyco88W3I+BBQAAK6ym266Sffcc49uvPFG9e7d+5q7A6czEFAA\nAOhiERERio+P17hx4/TNb35TI0eOVFRUVI+a9NpRjAwAAF3E5XIpKytLU6dO1V133aURI0aEukvd\nBgEFAIBOZLPZlJGRodGjR2vChAmBSa/MLekYAgoAAJ1k6NChmjVrliZOnKjhw4crNTU11F3qtggo\nAABcAbvdrpEjR+r+++/X2LFjlZ6ertjY2FB3q9sjoAAA0EERERGKi4vT2LFjNX/+fI0dO1Z2u11h\nYWFcyukkBBQAANrJ5XKpf//+Gjt2rL7yla9oxIgR3f5Tg62KgAIAwCVkZGSooKBAEydO1JQpU5ST\nkxPqLl3zCCgAAJyHzWZT//79NWfOHE2aNElDhw5VWloal3CuEgIKAACfERYWppEjR+q+++5TYWGh\nEhMTFR0dTTC5yggoAIAez+FwKCEhQfn5+fp//+//ady4cXI6nQoLCwt113osAgoAoMdKSEhQTk6O\nbrzxRn35y1/WqFGjQt0lfIqAAgDocVJTUzVx4kRNnDhR48aN06BBgzhbYjEEFABAj9GvXz/NmTNH\nt9xyiwYMGKCUlBRuE7YoAgoAoNtom6h69teLrbPZbMrOztZDDz2kwsJCpaSkKCoqikmvFkdAAQB0\nufDwcIWHhyssLOyiXy+0LiYmRnFxcYGKjY1VfHx84PvPrjvfssjIyPOGGlgXAQUAcFF2u10Oh0MR\nERGX/Hq+ZU6nU9HR0ZesmJgYRUdHKyoqKvB922vmh/Q8BBQAgBwOh3r16qW0tLRApaamKjU1VdHR\n0XI6nXI4HHI6nedU2/LIyMjztrHbeatBx/G3BgB6kKioKGVlZSkrK0t9+/YNfO/xeBQXF3fOWQ3O\nXiBUCCgA0I3ZbDaFhYWdU8nJyRowYID69++vAQMGaMCAAcrJyVF6enpgrkd4eLjsdntgngdzM2Al\nHQooq1ev1urVq7V//35J0tChQ/Wd73xHhYWFkiRjjB599FH953/+p06ePKlRo0bpJz/5iYYOHRrY\nht/v15IlS/Tiiy+qvr5ekydP1jPPPKOMjIwOd97lcsnr9Xb45wCgO7HZbHI4HIqMjFRUVJQiIyMD\n5fF4lJ2dHai+ffsqOztbCQkJF9wW0B10KKBkZGRo5cqVGjBggCTpueee06xZs7R9+3YNHTpUq1at\n0hNPPKFnn31WAwcO1He/+11NnTpVe/bsUVxcnCRp4cKF+p//+R8VFxerV69eWrx4sWbMmKFt27Z1\n+F7073znO3rjjTe0ZcsW1dXVdehnAcBqbDabEhMTlZCQEFRJSUlKS0uT2+1Wenq6PB5P4PvIyMhQ\ndxvoEjZjjLmSDSQlJenxxx/XPffcI4/Ho4ULF+pb3/qWpNazJWlpafre976nb3zjG/J6vUpJSdEL\nL7yguXPnSpIOHz6szMxMrVu3TtOnT2/XPn0+n1wul2pqalRTU6MNGzboxRdfVElJyZUcCtDt/epX\nv9JTTz2lv/3tb6HuSo/31a9+VQ6HQz/72c/OWRcZGRkIGG63O+j7hISEQEhJTExUYmKi4uLiFBER\nEYKjADpX2/u31+tVfHz8RdtedkBpbm7WH/7wB82bN0/bt29XZGSk+vfvr/fee0/XX399oN2sWbOU\nkJCg5557Ths2bNDkyZN14sQJJSYmBtqMGDFCs2fP1qOPPnreffn9fvn9/qADzMzMDBxgc3OzDh06\npE2bNumJJ57Q9u3bL+eQ0EEOh0NTp07VQw89pGPHjmn37t3avXu3PvjgA33wwQdqamoKdRd7nOzs\nbFVVVam+vj7UXenxkpOTlZSUpOjo6HMuv6SlpSkmJibokk1UVJScTieXYHBN60hA6fAk2bKyMhUU\nFOjMmTOKjY3V2rVrNWTIEG3ZskWSlJaWFtQ+LS1Nn3zyiSSpqqpKDocjKJy0tamqqrrgPouKii4Y\nXqTWBwBlZmbqjjvu0PTp01VcXKwf/vCHOnjwoBoaGjp6iLiI8PBwxcbG6qabbtLDDz+sG264QZGR\nkWppaVFLS4uam5sDtXv3bpWVlamsrEzl5eUqKytTQ0ODmpqaAtXc3BzqQ+oW2h5Y1Tah0W63y+Vy\nacCAAbruuusCNWjQIMXGxoa6u/iUzWYLTED9bNlsNoIIcAkdDiiDBg3Sjh07VFNTo5deeknz5s1T\naWlpYP3Z/9MZYy75P+Kl2ixdulSLFi0KvG47g/JZbb8IUlJStGDBAn3xi1/U6tWr9V//9V/65JNP\ndPr06Y4cJs7icDjUu3dvjR49Wl//+tc1YcKEoPVtdwR89jT0qFGjgj4ZtKWlRR9++KH27NkTVLW1\ntaqrq1NdXZ1qa2t16tSpq3VYltP2xMzPVnx8vDwej/r376/+/fsrJydH/fr1k9vt5vkSAK5ZHf7t\n5nA4ApNk8/PztXXrVj311FOBeSdVVVVKT08PtK+urg6cVXG73WpoaNDJkyeDzqJUV1drzJgxF9xn\n28N+OiI9PV0rVqzQl770Jf3+97/Xa6+9pvLycjU2NnZoOz2dzWbT8OHDNWnSJN16660qKCi47El5\nYWFhGjhwoAYOHKhbb71VUmtoOXTokA4cOKCDBw+qoqJCBw4cUHV1daCOHDmiY8eOdeZhWYLL5VJy\ncnJQpaSkqHfv3vJ4PPJ4POrdu7fS0tIUHR0d6u4CwFV1xf/8MsbI7/crOztbbrdbJSUlgTkoDQ0N\nKi0t1fe+9z1JUl5eniIiIlRSUqI5c+ZIkiorK1VeXq5Vq1ZdaVfOERYWpmHDhmnAgAGaPXu21q9f\nrzVr1uijjz7q9H1di66//nrdcccdmjBhgoYMGaKYmJhO30dYWJgyMzODzoi1tLToxIkTOnHihI4f\nP64TJ07o6NGj2r9/v/bt2xf4WlFR0en96QqRkZGBY+zTp0/ga3Jy8nknQwIAOhhQli1bpsLCQmVm\nZqq2tlbFxcXauHGj1q9fL5vNpoULF+qxxx5TTk6OcnJy9Nhjjyk6Olp33nmnpNZ/Md57771avHix\nevXqpaSkJC1ZskTDhg3TlClTuuQApdYnJ+bl5em6667Tbbfdpueff16//OUvdfTo0S7bZ3fWv39/\nfetb39LkyZPVu3dvORyOq3q9vO0hU8nJyYFlxhjV19cHlc/n0wcffKCdO3dq165d2rlzpz766CNd\n4Y1pl81msykzM1MDBw7UoEGDlJOTo0GDBikjI0PR0dGB51a0PRKcj3gHgAvrUEA5cuSI7rrrLlVW\nVsrlcmn48OFav369pk6dKkl6+OGHVV9frwceeCDwoLY333wz6F+FTz75pOx2u+bMmRN4UNuzzz57\nVX5ZR0dHa/Dgwfr3f/93feUrX9Hjjz+u1157TceOHVNLS0uX79+qbDabYmJilJmZqUceeURz586V\n3W631JMlbTZb4NHbbYwxuv7669XS0iJjjFpaWlRfX6+dO3cGJua+//77+vDDD9XQ0KDGxkY1Njaq\noaGhQ5Nz2+bWtJXdbldSUpIGDBigIUOGaPDgwRo8eLAGDRqkmJiYwJM92yZCMiESADruip+DEgod\nuU3pUrZs2aIf/ehHevfdd1VRUdGjgorNZlNaWppGjhypL33pS7rzzjuvyYc+nTlzRnv37g3U3//+\nd33yySfy+Xzyer3y+Xzy+Xzy+/3nfGS7y+VS7969lZ2drX79+qlfv37Kzs5WSkoKZ0AAoIOuynNQ\nQqkzA4rU+pyVt956S6+88or++Mc/XvSW52tFRkaGpkyZounTp2vSpElKTU0NdZeuKr/fr8OHD+vg\nwYOBr3V1dec8OCslJeWaDG0AEAoElMt04sQJ/e1vf9Nvf/tbrVmz5pp8Rkfv3r01Z84c3XLLLRox\nYkTQPA8AALoSAeUK1dTUaO/evVq+fLneeOONbn/Zp23+xgMPPKC77rpLffv2VUxMDB+hDgC4qggo\nncAYo+bmZm3YsEErVqzQrl27ut0nJ0dGRqpXr16644479PDDDyspKclSE18BAD0LAaWTNTQ06De/\n+Y1eeOEFvf/++zp+/HiX7/NKxMbGavjw4ZoyZYruueceZWVlhbpLAAAQULqCMUbV1dX64x//qLVr\n12rDhg2W+0A2l8ulG2+8Uf/0T/+kqVOnauDAgaHuEgAAAQSULtTU1KRDhw7pz3/+s5555hm98847\nV3X/5xMZGanCwkLdeeedys/PV58+fZhfAgCwnC79NOOezm63KysrS71799bkyZP1+uuva9WqVfrg\ngw+u6l0/NptNdrtdhYWFWrJkiYYPH67Y2FiezQEAuCZwBuUKtA3d0aNHtWbNGj3//PPat29fl176\nCQsLk9vt1o033qhFixYpPz9f4eHhTHwFAFgel3hC5KOPPtKzzz6r119/Xbt27erUoGK32zVo0CAV\nFBTorrvuUkFBgSIiIjpt+wAAdDUCSgg1NTWprKxM69at0+9//3uVl5df0XNUIiIiNHz4cM2aNUuT\nJk3SmDFjOFsCAOiWCCgWcPr0ae3fv1+vvPKKfvSjH+nIkSMd3saIESP09a9/XZMmTVLfvn155DoA\noFsjoFhIQ0ODqqqq9OSTT+rXv/61Tpw4ccEzKjabTQ6HQ0OHDtWCBQs0a9YsxcbGcikHAHBNIKBY\nTNsQl5WV6fvf/742bdqkAwcO6LNDn5KSoiFDhuiee+7RrFmzFB8fz6UcAMA1hYBicf/7v/+rF154\nQaWlpWpsbNTo0aM1e/ZszZgxQwkJCaHuHgAAXYLnoFjc5MmTVVBQoD/96U9qbGxUQUGBEhISOGMC\nAMCnCCghEh0dralTp4a6GwAAWBLPQwcAAJZDQAEAAJZDQAEAAJZDQAEAAJZDQAEAAJZDQAEAAJZD\nQAEAAJZDQAEAAJZDQAEAAJZDQAEAAJZDQAEAAJZDQAEAAJZDQAEAAJbToYCyevVqDR8+XPHx8YqP\nj1dBQYFef/31wPq7775bNpstqEaPHh20Db/frwULFig5OVkxMTGaOXOmDh482DlHAwAArgkdCigZ\nGRlauXKl/vrXv+qvf/2rJk2apFmzZmnnzp2BNjfffLMqKysDtW7duqBtLFy4UGvXrlVxcbE2b96s\nuro6zZgxQ83NzZ1zRAAAoNuzGWPMlWwgKSlJjz/+uO69917dfffdqqmp0SuvvHLetl6vVykpKXrh\nhRc0d+5cSdLhw4eVmZmpdevWafr06e3ap8/nk8vlktfrVXx8/JV0HwAAXCUdef++7Dkozc3NKi4u\n1qlTp1RQUBBYvnHjRqWmpmrgwIH62te+purq6sC6bdu2qbGxUdOmTQss83g8ys3N1ZYtWy64L7/f\nL5/PF1QAAODa1eGAUlZWptjYWDmdTt13331au3athgwZIkkqLCzUb37zG23YsEE/+MEPtHXrVk2a\nNEl+v1+SVFVVJYfDocTExKBtpqWlqaqq6oL7LCoqksvlClRmZmZHuw0AALoRe0d/YNCgQdqxY4dq\namr00ksvad68eSotLdWQIUMCl20kKTc3V/n5+crKytJrr72m22677YLbNMbIZrNdcP3SpUu1aNGi\nwGufz0dIAQDgGtbhgOJwODRgwABJUn5+vrZu3aqnnnpKP/vZz85pm56erqysLO3du1eS5Ha71dDQ\noJMnTwadRamurtaYMWMuuE+n0ymn09nRrgIAgG7qip+DYowJXMI52/Hjx1VRUaH09HRJUl5eniIi\nIlRSUhJoU1lZqfLy8osGFAAA0LN06AzKsmXLVFhYqMzMTNXW1qq4uFgbN27U+vXrVVdXpxUrVuj2\n229Xenq69u/fr2XLlik5OVlf+MIXJEkul0v33nuvFi9erF69eikpKUlLlizRsGHDNGXKlC45QAAA\n0P10KKAcOXJEd911lyorK+VyuTR8+HCtX79eU6dOVX19vcrKyvT888+rpqZG6enpmjhxon73u98p\nLi4usI0nn3xSdrtdc+bMUX19vSZPnqxnn31W4eHhnX5wAACge7ri56CEAs9BAQCg+7kqz0EBAADo\nKgQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQU\nAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABg\nOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQUAABgOQQU\nAABgOQQUAABgOQQUAABgOQQUAABgOfZQd+ByGGMkST6fL8Q9AQAA7dX2vt32Pn4x3TKg1NbWSpIy\nMzND3BMAANBRtbW1crlcF21jM+2JMRbT0tKiPXv2aMiQIaqoqFB8fHyou9Rt+Xw+ZWZmMo6dgLHs\nPIxl52AcOw9j2TmMMaqtrZXH41FY2MVnmXTLMyhhYWHq3bu3JCk+Pp6/LJ2Acew8jGXnYSw7B+PY\neRjLK3epMydtmCQLAAAsh4ACAAAsp9sGFKfTqeXLl8vpdIa6K90a49h5GMvOw1h2Dsax8zCWV1+3\nnCQLAACubd32DAoAALh2EVAAAIDlEFAAAIDlEFAAAIDldMuA8swzzyg7O1uRkZHKy8vTO++8E+ou\nWc6mTZt06623yuPxyGaz6ZVXXglab4zRihUr5PF4FBUVpQkTJmjnzp1Bbfx+vxYsWKDk5GTFxMRo\n5syZOnjw4FU8itArKirSDTfcoLi4OKWmpmr27Nnas2dPUBvGsn1Wr16t4cOHBx50VVBQoNdffz2w\nnnG8PEVFRbLZbFq4cGFgGWPZPitWrJDNZgsqt9sdWM84hpjpZoqLi01ERIT5+c9/bnbt2mUeeugh\nExMTYz755JNQd81S1q1bZ7797W+bl156yUgya9euDVq/cuVKExcXZ1566SVTVlZm5s6da9LT043P\n5wu0ue+++0zv3r1NSUmJee+998zEiRPNiBEjTFNT01U+mtCZPn26WbNmjSkvLzc7duwwt9xyi+nT\np4+pq6sLtGEs2+fVV181r732mtmzZ4/Zs2ePWbZsmYmIiDDl5eXGGMbxcvzlL38xffv2NcOHDzcP\nPfRQYDlj2T7Lly83Q4cONZWVlYGqrq4OrGccQ6vbBZTPf/7z5r777gtaNnjwYPPII4+EqEfWd3ZA\naWlpMW6326xcuTKw7MyZM8blcpmf/vSnxhhjampqTEREhCkuLg60OXTokAkLCzPr16+/an23murq\naiPJlJaWGmMYyyuVmJhofvGLXzCOl6G2ttbk5OSYkpISM378+EBAYSzbb/ny5WbEiBHnXcc4hl63\nusTT0NCgbdu2adq0aUHLp02bpi1btoSoV93Pvn37VFVVFTSOTqdT48ePD4zjtm3b1NjYGNTG4/Eo\nNze3R4+11+uVJCUlJUliLC9Xc3OziouLderUKRUUFDCOl+HBBx/ULbfcoilTpgQtZyw7Zu/evfJ4\nPMrOztaXv/xlffzxx5IYRyvoVh8WeOzYMTU3NystLS1oeVpamqqqqkLUq+6nbazON46ffPJJoI3D\n4VBiYuI5bXrqWBtjtGjRIt10003Kzc2VxFh2VFlZmQoKCnTmzBnFxsZq7dq1GjJkSOCXOePYPsXF\nxXrvvfe0devWc9bxd7L9Ro0apeeff14DBw7UkSNH9N3vfldjxozRzp07GUcL6FYBpY3NZgt6bYw5\nZxku7XLGsSeP9fz58/X+++9r8+bN56xjLNtn0KBB2rFjh2pqavTSSy9p3rx5Ki0tDaxnHC+toqJC\nDz30kN58801FRkZesB1jeWmFhYWB74cNG6aCggL1799fzz33nEaPHi2JcQylbnWJJzk5WeHh4eck\n0+rq6nNSLi6sbZb6xcbR7XaroaFBJ0+evGCbnmTBggV69dVX9fbbbysjIyOwnLHsGIfDoQEDBig/\nP19FRUUaMWKEnnrqKcaxA7Zt26bq6mrl5eXJbrfLbrertLRUP/rRj2S32wNjwVh2XExMjIYNG6a9\ne/fyd9ICulVAcTgcysvLU0lJSdDykpISjRkzJkS96n6ys7PldruDxrGhoUGlpaWBcczLy1NERERQ\nm8rKSpWXl/eosTbGaP78+Xr55Ze1YcMGZWdnB61nLK+MMUZ+v59x7IDJkyerrKxMO3bsCFR+fr7+\n+Z//WTt27FC/fv0Yy8vk9/u1e/dupaen83fSCkIxM/dKtN1m/Mtf/tLs2rXLLFy40MTExJj9+/eH\numuWUltba7Zv3262b99uJJknnnjCbN++PXA79sqVK43L5TIvv/yyKSsrM3fcccd5b5/LyMgwb731\nlnnvvffMpEmTetztc/fff79xuVxm48aNQbcinj59OtCGsWyfpUuXmk2bNpl9+/aZ999/3yxbtsyE\nhYWZN9980xjDOF6Jz97FYwxj2V6LFy82GzduNB9//LF59913zYwZM0xcXFzg/YRxDK1uF1CMMeYn\nP/mJycrKMg6Hw3zuc58L3PKJf3j77beNpHNq3rx5xpjWW+iWL19u3G63cTqdZty4caasrCxoG/X1\n9Wb+/PkmKSnJREVFmRkzZpgDBw6E4GhC53xjKMmsWbMm0IaxbJ977rkn8P9tSkqKmTx5ciCcGMM4\nXomzAwpj2T5tzzWJiIgwHo/H3HbbbWbnzp2B9YxjaNmMMSY0524AAADOr1vNQQEAAD0DAQUAAFgO\nAQUAAFgOAQUAAFgOAQUAAFgOAQUAAFgOAQUAAFgOAQUAAFgOAQUAAFgOAQUAAFgOAQUAAFgOAQUA\nAFjO/wcIaFm37RUddQAAAABJRU5ErkJggg==\n"
          }
        }
      ],
      "source": [
        "plt.imshow(env.render())"
      ],
      "id": "3100a95b-a606-470d-988d-952321e928f9"
    },
    {
      "cell_type": "code",
      "execution_count": 94,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "next_state, reward, terminated, _, _ = env.step(0)\n",
        "next_state, reward, terminated"
      ],
      "id": "371ecae3-a634-4f3f-ac9e-887cbc8294a5"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`-` play"
      ],
      "id": "b6a883ac-f93b-436b-88eb-606ee854a310"
    },
    {
      "cell_type": "code",
      "execution_count": 95,
      "metadata": {
        "tags": []
      },
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAF7CAYAAAD4/3BBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90\nbGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9h\nAAAPYQGoP6dpAAAzD0lEQVR4nO3de3xU9Z3/8fckmUzuQyYhMxkSQuQqJlwMCGStICjKFtFqV6qt\nheKjW6rwaBb42UIvYNcapK2uWyr25r02bqu4dqVoXCHKsraUSwmoiBruCRFMZhIIk9v390c2pw7X\nBBLmJLyej8fnAXPONzOf+QrOm3O+54zDGGMEAABgI1GRbgAAAOBkBBQAAGA7BBQAAGA7BBQAAGA7\nBBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7EQ0ojz32mHJzcxUXF6eCggK9/fbb\nkWwHAADYRMQCygsvvKCioiJ997vf1datW/W5z31O06ZN0759+yLVEgAAsAlHpL4scNy4cbryyiu1\natUqa9vll1+uW265RcXFxZFoCQAA2ERMJF60sbFRmzdv1ne+852w7VOnTtXGjRtPGR8KhRQKhazH\nra2t+vTTT5WWliaHw9Ht/QIAgAtnjFFdXZ38fr+ios5+EiciAeXIkSNqaWmR1+sN2+71elVVVXXK\n+OLiYt1///0Xqz0AANCN9u/fr6ysrLOOiUhAaXfy0Q9jzGmPiCxevFgLFiywHgcCAfXv37/b+wNw\nerExifrc2HvVv99Y9U0YpiiHU4frd2jj9se1a/cbkqSoqBjd+vmfKDd1sqKjXN3flDHaevBp/ff6\nh9XcHFLeZTdrVP5tinUmKCMpX62tjfr4yDr99//+WDU1+7u/HwBnlJycfM4xEQko6enpio6OPuVo\nSXV19SlHVSTJ5XLJ5boI/4MD0CFOZ5wuy/kHpbiylOBKU2PzMVUF/6Z9+/9qjXE4HHI64+WKTVbM\nRQgoxhg5nXHWP3I+2P+Ghl12o6L6OGQczUqIS9OAjM/p8qHbtPGdJ7q9HwBn1pHlGRG5iic2NlYF\nBQUqLS0N215aWqrCwsJItASgwxy64x+fUKualRTrlTFGlfVb9f7u/1bDidpIN2dpbDqmsr8+oj5x\nA3Tk+PsyxijR2VeDs66T1zsk0u0BOIeIXWa8YMEC/frXv9YTTzyh9957T//yL/+iffv2ae7cuZFq\nCUAHpKZkq8EcUXr85XI4onSs6RMd+GSzKvb8b6RbO8Untbv1t/dWK8oRo/rGw3I4otQv5SpdPnSq\noqNjI90egLOI2BqUmTNn6ujRo/rhD3+oyspK5eXlac2aNcrJyYlUSwA64PrPLVFSXKbinX3U0tqo\n6rod2rr995JOvWOBUauaW0/ImNaL05wxkj576Nho87vPKSdrrOqjKpXgTJMzOk6DvFNUkfVn7d27\n6eL0BaDTIrpI9p577tE999wTyRYAdMKwy26UoluUHj9UkhQMHdKuPa+rNnDwtOObWhtUfWynohxd\ne7D2tDdvMlKLWuQ46bWOh2q0/f2XNCr/NjXEfqpEZ4b6Jg/XoJxr9Mknu3X8eG2X9gaga0Q0oADo\nWTzp/ZTpHqWYqDg1tTTok+D7qtj3v2puDp0ytqWlWevf+nc5HNHd0Mnp7y/Z0tKopqYTYdtaW5u0\n5+A76pc5UnGxKYqP8Sg2OlGX+SZpd/o67du39YzPByByCCgAOmTIZdcq21+guJg+MmrVscZqlX/4\nko5+uucMP2HOsu/iqq0/oH0HNisuKVFuV3/FxfRRSrxfecNuUm3tQQWDhyPdIoCTEFAAdIjDOBVj\nknQg+GeluLJ0uGanKqveVUtLU6Rb6wCjXXtL1d9foMr4bUqK9amqZrv27f+rQqFjkW4OwGkQUAB0\nyK6K1xUK1evKYXfIlVan8g9f1pGjFZFuq8Mam+pV/sEfNb7PV7X/0BZ9XPE/2nvoz5FuC8AZEFAA\ndNieQxv1aaBCWd4C7dn/F/W0tRv7qv4s19+SVH30PQXqD0W6HQBnEbFvM74QwWBQbrc70m0AAIDz\nEAgElJKSctYxEbtRGwAAwJkQUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAA\ngO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0Q\nUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO10eUBZtmyZHA5HWPl8Pmu/\nMUbLli2T3+9XfHy8Jk2apJ07d3Z1GwAAoAfrliMoV1xxhSorK60qLy+39q1YsUIPP/ywVq5cqU2b\nNsnn8+n6669XXV1dd7QCAAB6oG4JKDExMfL5fFb17dtXUtvRk3/7t3/Td7/7Xd16663Ky8vT008/\nrePHj+v555/vjlYAAEAP1C0BZffu3fL7/crNzdWXvvQlffzxx5KkiooKVVVVaerUqdZYl8uliRMn\nauPGjWd8vlAopGAwGFYAAKD36vKAMm7cOD3zzDN67bXX9Ktf/UpVVVUqLCzU0aNHVVVVJUnyer1h\nP+P1eq19p1NcXCy3221VdnZ2V7cNAABsxGGMMd35AseOHdPAgQN13333afz48fqHf/gHHTp0SJmZ\nmdaYr3/969q/f7/Wrl172ucIhUIKhULW42AwSEgBAKCHCgQCSklJOeuYbr/MODExUfn5+dq9e7d1\nNc/JR0uqq6tPOaryWS6XSykpKWEFAAB6r24PKKFQSO+9954yMzOVm5srn8+n0tJSa39jY6PKyspU\nWFjY3a0AAIAeIqarn3DRokW66aab1L9/f1VXV+uBBx5QMBjUrFmz5HA4VFRUpAcffFCDBw/W4MGD\n9eCDDyohIUF33nlnV7cCAAB6qC4PKAcOHNAdd9yhI0eOqG/fvho/frzeeecd5eTkSJLuu+8+NTQ0\n6J577lFNTY3GjRun119/XcnJyV3dCgAA6KG6fZFsdwgGg3K73ZFuAwAAnAdbLJIFAADoLAIKAACw\nHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIK\nAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACw\nHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIKAACwHQIK\nAACwnU4HlLfeeks33XST/H6/HA6HXn755bD9xhgtW7ZMfr9f8fHxmjRpknbu3Bk2JhQKaf78+UpP\nT1diYqJmzJihAwcOXNAbAQAAvUenA8qxY8c0cuRIrVy58rT7V6xYoYcfflgrV67Upk2b5PP5dP31\n16uurs4aU1RUpNWrV6ukpEQbNmxQfX29pk+frpaWlvN/JwAAoPcwF0CSWb16tfW4tbXV+Hw+s3z5\ncmvbiRMnjNvtNo8//rgxxpja2lrjdDpNSUmJNebgwYMmKirKrF27tkOvGwgEjCSKoiiKonpgBQKB\nc37Wd+kalIqKClVVVWnq1KnWNpfLpYkTJ2rjxo2SpM2bN6upqSlsjN/vV15enjXmZKFQSMFgMKwA\nAEDv1aUBpaqqSpLk9XrDtnu9XmtfVVWVYmNjlZqaesYxJysuLpbb7bYqOzu7K9sGAAA20y1X8Tgc\njrDHxphTtp3sbGMWL16sQCBg1f79+7usVwAAYD9dGlB8Pp8knXIkpLq62jqq4vP51NjYqJqamjOO\nOZnL5VJKSkpYAQCA3qtLA0pubq58Pp9KS0utbY2NjSorK1NhYaEkqaCgQE6nM2xMZWWlduzYYY0B\nAACXtpjO/kB9fb0+/PBD63FFRYW2bdsmj8ej/v37q6ioSA8++KAGDx6swYMH68EHH1RCQoLuvPNO\nSZLb7dbdd9+thQsXKi0tTR6PR4sWLVJ+fr6uu+66rntnAACg5+rQdb2fsW7dutNeMjRr1ixjTNul\nxkuXLjU+n8+4XC5zzTXXmPLy8rDnaGhoMPPmzTMej8fEx8eb6dOnm3379nW4By4zpiiKoqieWx25\nzNhhjDHqYYLBoNxud6TbAAAA5yEQCJxzPSnfxQMAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyH\ngAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIA\nAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyH\ngAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyHgAIAAGyn0wHlrbfe0k033SS/3y+Hw6GXX345\nbP/s2bPlcDjCavz48WFjQqGQ5s+fr/T0dCUmJmrGjBk6cODABb0RAADQe3Q6oBw7dkwjR47UypUr\nzzjmxhtvVGVlpVVr1qwJ219UVKTVq1erpKREGzZsUH19vaZPn66WlpbOvwMAANDrxHT2B6ZNm6Zp\n06addYzL5ZLP5zvtvkAgoN/85jd69tlndd1110mSnnvuOWVnZ+uNN97QDTfc0NmWAABAL9Mta1DW\nr1+vjIwMDRkyRF//+tdVXV1t7du8ebOampo0depUa5vf71deXp42btx42ucLhUIKBoNhBQAAeq8u\nDyjTpk3Tb3/7W7355pv66U9/qk2bNmny5MkKhUKSpKqqKsXGxio1NTXs57xer6qqqk77nMXFxXK7\n3VZlZ2d3ddsAAMBGOn2K51xmzpxp/T4vL09jxoxRTk6OXn31Vd16661n/DljjBwOx2n3LV68WAsW\nLLAeB4NBQgoAAL1Yt19mnJmZqZycHO3evVuS5PP51NjYqJqamrBx1dXV8nq9p30Ol8ullJSUsAIA\nAL1XtweUo0ePav/+/crMzJQkFRQUyOl0qrS01BpTWVmpHTt2qLCwsLvbAQAAPUCnT/HU19frww8/\ntB5XVFRo27Zt8ng88ng8WrZsmW677TZlZmZqz549WrJkidLT0/WFL3xBkuR2u3X33Xdr4cKFSktL\nk8fj0aJFi5Sfn29d1QMAAC5xppPWrVtnJJ1Ss2bNMsePHzdTp041ffv2NU6n0/Tv39/MmjXL7Nu3\nL+w5GhoazLx584zH4zHx8fFm+vTpp4w5m0AgcNoeKIqiKIqyfwUCgXN+1juMMUY9TDAYlNvtjnQb\nAADgPAQCgXOuJ+W7eAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAA\ngO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0Q\nUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAAgO0QUAAA\ngO0QUAAAgO0QUAAAgO0QUAAAgO10KqAUFxdr7NixSk5OVkZGhm655Rbt2rUrbIwxRsuWLZPf71d8\nfLwmTZqknTt3ho0JhUKaP3++0tPTlZiYqBkzZujAgQMX/m4AAECv0KmAUlZWpnvvvVfvvPOOSktL\n1dzcrKlTp+rYsWPWmBUrVujhhx/WypUrtWnTJvl8Pl1//fWqq6uzxhQVFWn16tUqKSnRhg0bVF9f\nr+nTp6ulpaXr3hkAAOi5zAWorq42kkxZWZkxxpjW1lbj8/nM8uXLrTEnTpwwbrfbPP7448YYY2pr\na43T6TQlJSXWmIMHD5qoqCizdu3aDr1uIBAwkiiKoiiK6oEVCATO+Vl/QWtQAoGAJMnj8UiSKioq\nVFVVpalTp1pjXC6XJk6cqI0bN0qSNm/erKamprAxfr9feXl51piThUIhBYPBsAIAAL3XeQcUY4wW\nLFigq6++Wnl5eZKkqqoqSZLX6w0b6/V6rX1VVVWKjY1VamrqGcecrLi4WG6326rs7OzzbRsAAPQA\n5x1Q5s2bp+3bt+t3v/vdKfscDkfYY2PMKdtOdrYxixcvViAQsGr//v3n2zYAAOgBziugzJ8/X6+8\n8orWrVunrKwsa7vP55OkU46EVFdXW0dVfD6fGhsbVVNTc8YxJ3O5XEpJSQkrAADQe3UqoBhjNG/e\nPL300kt68803lZubG7Y/NzdXPp9PpaWl1rbGxkaVlZWpsLBQklRQUCCn0xk2prKyUjt27LDGAACA\nS1xHr9gxxphvfvObxu12m/Xr15vKykqrjh8/bo1Zvny5cbvd5qWXXjLl5eXmjjvuMJmZmSYYDFpj\n5s6da7Kysswbb7xhtmzZYiZPnmxGjhxpmpubuYqHoiiKonp5deQqnk4FlDO90JNPPmmNaW1tNUuX\nLjU+n8+4XC5zzTXXmPLy8rDnaWhoMPPmzTMej8fEx8eb6dOnm3379nW4DwIKRVEURfXc6khAcfxf\n8OhRgsGg3G53pNsAAADnIRAInHM9Kd/FAwAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAA\nbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeA\nAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAA\nbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbIeAAgAAbKdTAaW4uFhjx45VcnKyMjIydMst\nt2jXrl1hY2bPni2HwxFW48ePDxsTCoU0f/58paenKzExUTNmzNCBAwcu/N0AAIBeoVMBpaysTPfe\ne6/eeecdlZaWqrm5WVOnTtWxY8fCxt14442qrKy0as2aNWH7i4qKtHr1apWUlGjDhg2qr6/X9OnT\n1dLScuHvCAAA9HzmAlRXVxtJpqyszNo2a9Ysc/PNN5/xZ2pra43T6TQlJSXWtoMHD5qoqCizdu3a\nDr1uIBAwkiiKOqmWLJF5+22ZNWtkfvpTmUmTZNLSZDwemZQUmdjYyPd4qdTnP9/23+L112V+8QuZ\n2277+38Lt1smLi7yPVJUpCoQCJzzsz5GFyAQCEiSPB5P2Pb169crIyNDffr00cSJE/WjH/1IGRkZ\nkqTNmzerqalJU6dOtcb7/X7l5eVp48aNuuGGG055nVAopFAoZD0OBoMX0jbQa8XESPHxbZWRIU2c\nKBkjNTRI+/ZJb78tbd0qtbS0bfvkk7ZC14uO/vt/C49HKiiQvvMdKRSSDh+W/vIXad06qbVVOnFC\nqqmRDh6MdNeAfZx3QDHGaMGCBbr66quVl5dnbZ82bZr+6Z/+STk5OaqoqND3v/99TZ48WZs3b5bL\n5VJVVZViY2OVmpoa9nxer1dVVVWnfa3i4mLdf//959sqcElzOKSEBGnYsLYyRmpqkj79VHr3XWnn\nzrbAEghIe/ZI5eWR7rj3cjikuDgpJ6etvvhFqbm5be4//rgttLS0SPX10oEDbWGyuTnSXQORcd4B\nZd68edq+fbs2bNgQtn3mzJnW7/Py8jRmzBjl5OTo1Vdf1a233nrG5zPGyOFwnHbf4sWLtWDBAutx\nMBhUdnb2+bYOXNIcDik2VvL52uraa9v+FX/8eNu/7PfubftQPHq0LcCUlrZ9aKLrORyS0ymlp7fV\n2LF/P+J19GhbaGlslIJB6YMPpDfeaPs9cCk4r4Ayf/58vfLKK3rrrbeUlZV11rGZmZnKycnR7t27\nJUk+n0+NjY2qqakJO4pSXV2twsLC0z6Hy+WSy+U6n1YBnIPD0XY6Ijm5rQYObPuQbG5u+6CcOVP6\n2tci3eWlweFoq8TEtmr/d1hzc9tpoLvukr7yFemk6xKAXqlTAcUYo/nz52v16tVav369cnNzz/kz\nR48e1f79+5WZmSlJKigokNPpVGlpqW6//XZJUmVlpXbs2KEVK1acx1sAcCGMaav2dSl1dW2ngNrX\nSZSURLrDS4cxbb+2tLQFkvb/FkePStu2Sb//PeEEl45OBZR7771Xzz//vP7zP/9TycnJ1poRt9ut\n+Ph41dfXa9myZbrtttuUmZmpPXv2aMmSJUpPT9cXvvAFa+zdd9+thQsXKi0tTR6PR4sWLVJ+fr6u\nu+66rn+HAMK0h5Hjx6WqKunQobYPwU8+afsQfPPNSHd46TCm7fRaQ4N05MjfT699+qn03nvS2rVt\np3iAS1GnAsqqVaskSZMmTQrb/uSTT2r27NmKjo5WeXm5nnnmGdXW1iozM1PXXnutXnjhBSUnJ1vj\nH3nkEcXExOj2229XQ0ODpkyZoqeeekrR0dEX/o4AhDGm7UPuk0/aFsR+9FHbh2BNTdu6hpPutYhu\n1H7qrH3ut29vC4vBYNsC5a1bI90hYB+dPsVzNvHx8XrttdfO+TxxcXH62c9+pp/97GedeXkAHdC+\n4PXjj6WysrYPwvYrQ6qq2v51jovDmLZTNQcPShs3Sps2/f2IySefSJWVke4QsK8Lug8KAHvJyvqJ\n7rvvN9qx4z01NbUFlaamSHd1aUpNnanf/c6pZ599Tk1NbUHlxIlIdwX0HAQUoBeJifGopiZWR45E\nuhNERSXo2LFYboQHnCe+zRgAANgOAQUAANgOAQUAANgOAQUAANgOAQUAANgOAQUAANgOAQUAANgO\nAQUAANgOAQUAANgOAQUAANgOAQUAANgOAQUAANgOAQUAANgOAQUAANgOAQUAANgOAQUAANhOTKQb\nAACgK8XExKhfv34aMmSIhgwZoqFDh8rj8Wjv3r3au3ev9uzZoz179ujAgQM6fvx4pNvFGRBQAAA9\njsPhsCo1NVUjRozQqFGjNHr0aF1xxRVKT0+Xy+VSXFyc4uLiFBMTo1AopMbGRjU2NioUCqmhoUEH\nDhzQrl279P777+v999/Xrl27VFlZqaamJkmSMUbGmAi/20sTAQUAYFtRUVFW0HC5XEpOTtbAgQM1\natQoXXnllRo1apRycnIUFRUVFlocDscpzxUfH6/4+HjrsTFGgwYN0sSJE60QYoxRMBjUhx9+aAWW\n9l8DgYAVcNqrubn5os3FpYaAAgCwhaioKCUlJcnj8ViVmZlpnaYZNmyYcnNzlZSU1CWv1x5iTg4z\nHo9HV111la666qqw7YcPH1ZFRYV1imjv3r06ePCgamtrFQwGFQgEFAwGFQwGCS5dgIACAIiIpKQk\n+f1+9e/fX9nZ2crOzlZWVpb69eunrKwsZWVlqU+fPoqKssf1HF6vV16vV+PHj7e2NTY2qrq6WocP\nH1Z1dbX1+8OHD6uqqiqsPv300wh23/MQUAAA3S4mJkb9+/fX8OHDdfnll2vYsGHq37+/UlNT1adP\nH/Xp00cpKSlyOp2RbrVTYmNjrTDVzhijhoYG1dXVhVVNTY0qKiqs+uijj7Rv3z7V19dH8B3YFwEF\nAHDBHA6HtQ4kKipKffv21ciRI3XllVdq9OjRysvLU58+fRQbG2tVdHT0adeK9HQOh0MJCQlKSEiQ\n1+u1tre2tqq5uVlNTU1WnThxQocOHbIW6b733nt6//33deDAATU2NlqLdFtbWy+5xboEFABAp0RH\nRyshIcFadJqUlKRhw4Zp1KhRGjFihEaOHCm/339KAOmNYaQzoqKirHDWzhijrKwsa71Lewipr69X\nRUWFdu3apQ8++MD69cMPP7xkThURUAAAZ5Wamqr09HSr+vXrp8GDB2vQoEEaPHiwLrvsMrlcrki3\n2SOdHNraH6ekpGjkyJEaOXJk2P7nnntOK1eu1J///OeL1mOkEFAAAIqNjVVaWpq1ENTr9SojI0Ne\nr1eZmZlhlZKSYpuFq5eaL3/5yxoxYoSefPJJPf/886quro50S92GgAIAl5D4+Hjl5OQoJydHAwYM\nsH7v9/uVnJxsrZ1or/j4eMKIjTgcDuXn5+uHP/yhxo8fr6VLl2rXrl2RbqtbEFAAoAdrX5R6cqWn\np2vQoEEaOHCgBg0aZJ2OyczMVHR0tFUxMTGKjo62FrjC/hwOh5KTk3Xbbbfpmmuu0f/7f/9Pf/jD\nHxQKhSLdWpfqVEBZtWqVVq1apT179kiSrrjiCv3gBz/QtGnTJLUt7rn//vv1y1/+UjU1NRo3bpx+\n/vOf64orrrCeIxQKadGiRfrd736nhoYGTZkyRY899ljYJVoAgL9zOByKjY1VXFyc4uPjrdu3x8XF\nye/3Kzc316oBAwYoNzdXffr0OeNzoXeIiYmRz+fTM888o0mTJmn58uXat2+fdZv+nq5TASUrK0vL\nly/XoEGDJElPP/20br75Zm3dulVXXHGFVqxYoYcfflhPPfWUhgwZogceeEDXX3+9du3apeTkZElS\nUVGR/vjHP6qkpERpaWlauHChpk+frs2bNys6Orrr3yEA9BDt3yvTfl+Q9vJ4PPJ6vfL5fMrMzJTf\n77d+HxcXF+m2EUHtt/WfM2eOxowZo4cfflivvPKKAoFApFu7YA5zgRdWezwe/fjHP9acOXPk9/tV\nVFSkb3/725LajpZ4vV499NBD+sY3vqFAIKC+ffvq2Wef1cyZMyVJhw4dUnZ2ttasWaMbbrihQ68Z\nDAbldrsvpG1cgKioKOtw8eDBgzVw4MAed3Ol02ltbVVTU5P1RWInf+fGZ79k7GyPzza+paWlW9/D\nE088oUcffVR/+9vfuvV1cG5f+9rXFBsbq1/84hen7IuLi7MChs/nC/t9nz59rJCSmpqq1NRUJScn\n94q/Y+h+VVVV+uMf/6if/OQn+uCDDyLdzhkFAgGlpKScdcx5B5SWlhb9/ve/16xZs7R161bFxcVp\n4MCB2rJli0aPHm2Nu/nmm9WnTx89/fTTevPNNzVlyhR9+umnSk1NtcaMHDlSt9xyi+6///7TvlYo\nFAo7txYMBpWdnX0+beM8xMXFafTo0dY3hY4YMUJut1tJSUlKTk5WcnJyr1hE134zpM9WS0vLKdvO\nZ3v7vubmZp04ceKUamho6PD2M409ceKEfD6fDh06pIaGhkhP5yUvPT1dHo9HCQkJp5x+8Xq9SkxM\nDDtlEx8fL5fLxSkYXLCmpiZ98MEH+td//Ve98MILkW7ntDoSUDq9SLa8vFwTJkzQiRMnlJSUpNWr\nV2v48OHauHGjJIXdNa/98d69eyW1JbvY2NiwcNI+pqqq6oyvWVxcfMbwggvncDjkdDrldDoVExOj\ntLQ0jRs3TmPHjtW4ceOUn58ftqiuN9/9sf39dZfPfmPquX7tzNjT/YrIa/8zdfIC1jN92y7QFZxO\np4YPH65f/vKXmjRpkn70ox/p0KFDam1tjXRrndLpgDJ06FBt27ZNtbW1evHFFzVr1iyVlZVZ+0/+\nS2eMOedfxHONWbx4sRYsWGA95gjKhYmLi1NKSorcbrfcbreysrI0atQo66ZAubm5kW6x1zrTt6cC\nQFdyOBxKSUnR3LlzNXLkSD3wwANav369jh8/HunWOqzTASU2NtZaJDtmzBht2rRJjz76qLXupKqq\nSpmZmdb46upq66iKz+dTY2Ojampqwo6iVFdXq7Cw8Iyv6XK5uEvhBfB4POrXr59VOTk51qWHl112\nmdLT0yPdIgCgm0yYMEFPPvmkfvnLX+q5557rMfdNueD7oBhjFAqFlJubK5/Pp9LSUmsNSmNjo8rK\nyvTQQw9JkgoKCuR0OlVaWqrbb79dklRZWakdO3ZoxYoVF9oK/s+AAQN0+eWXa8iQIRo6dKiysrKs\nu0JmZGQoISEh0i0CAC6ijIwMLV68WOPHj9djjz2m//qv/7L95cidCihLlizRtGnTlJ2drbq6OpWU\nlGj9+vVau3atHA6HioqK9OCDD1pXdzz44INKSEjQnXfeKUlyu926++67tXDhQqWlpcnj8WjRokXK\nz8/Xdddd1y1vsLeLiorS6NGjdeWVV1rl8XiUmJioxMREJSQkKCaG+/EBwKUuOjpa1157rfLz8zV+\n/HgtX75cNTU1kW7rjDr1yXX48GHdddddqqyslNvt1ogRI7R27Vpdf/31kqT77rtPDQ0Nuueee6wb\ntb3++uvWPVAk6ZFHHlFMTIxuv/1260ZtTz31FPdAOYP2GzS5XC7FxsYqKSlJY8eOtaqgoECxsbGn\nLMADAOBk0dHR8nq9WrhwoSZOnKg5c+boo48+suVdaC/4PiiR0Jvvg9J+lVN7eb1e5efnKz8/X3l5\neRo2bBgBBADQJY4fP65ly5bp+eef18GDBy/a63brfVAiqTcFlJSUFGVnZ6t///5WDRgwwLpfwmcX\nHAMA0NWOHz+u1157TatWrVJZWZkaGxu7/TUJKDaWkpKiOXPm6MYbb1RaWprS09OVlpYWdjoMAICL\n5aOPPtJzzz2nH/3oR92+gJaAYkMOh0MTJkzQ8uXLNWrUKAIJAMA2jh07pp07d+qb3/ymtm7d2m03\nfuxIQOn59yfvIZxOpwYNGqRHH31UpaWluvrqqwknAABbSUhI0NixY/XKK6/onnvuUVpaWsTWPXIE\n5SLIzc3VtGnTNH/+fA0bNizS7QAAcE5NTU164YUX9POf/1xbtmzp0rUpnOKJMLfbrenTp+srX/mK\nrr32Wu6GCwDocbZv367f/va3euKJJ3TkyJEueU4CSgSNHTtWixYt0sSJE5WRkcGlwQCAHskYo/r6\ner366qtauXKl/vKXv1zwIloCykXmcDjUp08fLV68WHPmzJHb7eYurgCAXqG5uVkHDhzQ97//ff3h\nD39QKBQ670W0BJSLxOFwyOPxaPLkySouLtZll13GERMAQK/THhmeeuop/epXv9KmTZvU3Nzc6ech\noFwEiYmJ+tznPqfZs2drxowZio+Pj3RLAAB0q9bWVu3YsUMPPfSQ3nrrLVVWVqqlpaXDP09A6WZj\nxozR7NmzNX36dOXk5ES6HQAALqrq6mq99tprWrFihXbu3NnhUz4ElG6SlpamuXPn6s4779SQIUNY\nZwIAuGSFQiF98MEHWrFihf7nf/5HFRUV5/wZAkoXc7lcKiws1EMPPaQRI0YoNjaWtSYAAKjts/k/\n/uM/9L3vfU9Hjx4969oUAkoXiY+P1+DBg7Vw4ULdddddkkQwAQDgM9rjxJYtW/TUU0/p+eef16ef\nfnrasR0JKJybOIvo6GgNGTJEt9xyi77xjW+of//+BBMAAE6j/fOxoKBAfr9f6enpKikp0dGjR/XJ\nJ590/vk4gnJ6SUlJmjVrlr70pS+poKCAq3MAAOgEY4zWrVunF154Qb/+9a/V2tpq7eMIynmaNGmS\n7r//fuXn5ys1NTXS7QAA0OM4HA5NmjRJl19+uYYOHao//elPevvttxUKhTr28xxBaRMdHa2cnBwt\nXLhQX/7yl5WSksLpHAAAukBTU5M2bdqkWbNm6cMPP2SRbEdlZ2drypQp+va3v823DQMA0E3aP785\nxXMOHo9HU6ZM0ezZs/WP//iPkW4HAAD8n0syoERHR2vcuHH6xje+oRtuuEEZGRmRbgkAAHzGJRdQ\nEhMT9b3vfU9f+cpX5PV65XQ6I90SAAA4ySURUBwOh5KSkjRt2jT95Cc/kd/vV3R0dKTbAgAAZ9Dr\nA0p8fLwKCwv1z//8z7rpppu4nwkAAD1Arw4oY8aM0cyZM/XFL35RAwYMiHQ7AACgg3plQMnMzNSc\nOXN02223afTo0ZFuBwAAdFKvCihOp1OTJk3SD37wA40ePVoJCQmRbgkAAJyHXhFQ4uLilJmZqfvv\nv1933nmnoqKiuAssAAA9WI8OKNHR0Ro0aJBuvvlmLViwQBkZGQQTAAB6gR4dUL761a/qq1/9qsaN\nG8fVOQAA9CJRnRm8atUqjRgxQikpKUpJSdGECRP0pz/9ydo/e/ZsORyOsBo/fnzYc4RCIc2fP1/p\n6elKTEzUjBkzdODAgfNq/vvf/74mTZpEOAEAoJfpVEDJysrS8uXL9de//lV//etfNXnyZN18883a\nuXOnNebGG29UZWWlVWvWrAl7jqKiIq1evVolJSXasGGD6uvrNX36dLW0tHS6+bS0tE7/DAAAsL8L\n/jZjj8ejH//4x7r77rs1e/Zs1dbW6uWXXz7t2EAgoL59++rZZ5/VzJkzJUmHDh1Sdna21qxZoxtu\nuKFDr9mZb0MEAAD20JnP704dQfmslpYWlZSU6NixY5owYYK1ff369crIyNCQIUP09a9/XdXV1da+\nzZs3q6mpSVOnTrW2+f1+5eXlaePGjWd8rVAopGAwGFYAAKD36nRAKS8vV1JSklwul+bOnavVq1dr\n+PDhkqRp06bpt7/9rd5880399Kc/1aZNmzR58mSFQiFJUlVVlWJjY5Wamhr2nF6vV1VVVWd8zeLi\nYrndbquys7M72zYAAOhBOn0Vz9ChQ7Vt2zbV1tbqxRdf1KxZs1RWVqbhw4dbp20kKS8vT2PGjFFO\nTo5effVV3XrrrWd8TmPMWS8PXrx4sRYsWGA9DgaDhBQAAHqxTgeU2NhYDRo0SFLbd91s2rRJjz76\nqH7xi1+cMjYzM1M5OTnavXu3JMnn86mxsVE1NTVhR1Gqq6tVWFh4xtd0uVxyuVydbRUAAPRQ570G\npZ0xxjqFc7KjR49q//79yszMlCQVFBTI6XSqtLTUGlNZWakdO3acNaAAAIBLS6eOoCxZskTTpk1T\ndna26urqVFJSovXr12vt2rWqr6/XsmXLdNtttykzM1N79uzRkiVLlJ6eri984QuSJLfbrbvvvlsL\nFy5UWlqaPB6PFi1apPz8fF133XXd8gYBAEDP06mAcvjwYd11112qrKyU2+3WiBEjtHbtWl1//fVq\naGhQeXm5nnnmGdXW1iozM1PXXnutXnjhBSUnJ1vP8cgjjygmJka33367GhoaNGXKFD311FOKjo7u\n8jcHAAB6pgu+D0okcB8UAAB6notyHxQAAIDuQkABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2\nQ0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0AB\nAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2\nQ0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2Q0ABAAC2ExPpBs6HMUaS\nFAwGI9wJAADoqPbP7fbP8bPpkQGlrq5OkpSdnR3hTgAAQGfV1dXJ7XafdYzDdCTG2Exra6t27dql\n4cOHa//+/UpJSYl0Sz1WMBhUdnY289gFmMuuw1x2Deax6zCXXcMYo7q6Ovn9fkVFnX2VSY88ghIV\nFaV+/fpJklJSUvjD0gWYx67DXHYd5rJrMI9dh7m8cOc6ctKORbIAAMB2CCgAAMB2emxAcblcWrp0\nqVwuV6Rb6dGYx67DXHYd5rJrMI9dh7m8+HrkIlkAANC79dgjKAAAoPcioAAAANshoAAAANshoAAA\nANvpkQHlscceU25uruLi4lRQUKC333470i3ZzltvvaWbbrpJfr9fDodDL7/8cth+Y4yWLVsmv9+v\n+Ph4TZo0STt37gwbEwqFNH/+fKWnpysxMVEzZszQgQMHLuK7iLzi4mKNHTtWycnJysjI0C233KJd\nu3aFjWEuO2bVqlUaMWKEdaOrCRMm6E9/+pO1n3k8P8XFxXI4HCoqKrK2MZcds2zZMjkcjrDy+XzW\nfuYxwkwPU1JSYpxOp/nVr35l3n33XfOtb33LJCYmmr1790a6NVtZs2aN+e53v2tefPFFI8msXr06\nbP/y5ctNcnKyefHFF015ebmZOXOmyczMNMFg0Bozd+5c069fP1NaWmq2bNlirr32WjNy5EjT3Nx8\nkd9N5Nxwww3mySefNDt27DDbtm0zn//8503//v1NfX29NYa57JhXXnnFvPrqq2bXrl1m165dZsmS\nJcbpdJodO3YYY5jH8/GXv/zFDBgwwIwYMcJ861vfsrYzlx2zdOlSc8UVV5jKykqrqqurrf3MY2T1\nuIBy1VVXmblz54ZtGzZsmPnOd74ToY7s7+SA0traanw+n1m+fLm17cSJE8btdpvHH3/cGGNMbW2t\ncTqdpqSkxBpz8OBBExUVZdauXXvRereb6upqI8mUlZUZY5jLC5Wammp+/etfM4/noa6uzgwePNiU\nlpaaiRMnWgGFuey4pUuXmpEjR552H/MYeT3qFE9jY6M2b96sqVOnhm2fOnWqNm7cGKGuep6KigpV\nVVWFzaPL5dLEiROtedy8ebOamprCxvj9fuXl5V3Scx0IBCRJHo9HEnN5vlpaWlRSUqJjx45pwoQJ\nzON5uPfee/X5z39e1113Xdh25rJzdu/eLb/fr9zcXH3pS1/Sxx9/LIl5tIMe9WWBR44cUUtLi7xe\nb9h2r9erqqqqCHXV87TP1enmce/evdaY2NhYpaamnjLmUp1rY4wWLFigq6++Wnl5eZKYy84qLy/X\nhAkTdOLECSUlJWn16tUaPny49T9z5rFjSkpKtGXLFm3atOmUffyZ7Lhx48bpmWee0ZAhQ3T48GE9\n8MADKiws1M6dO5lHG+hRAaWdw+EIe2yMOWUbzu185vFSnut58+Zp+/bt2rBhwyn7mMuOGTp0qLZt\n26ba2lq9+OKLmjVrlsrKyqz9zOO57d+/X9/61rf0+uuvKy4u7ozjmMtzmzZtmvX7/Px8TZgwQQMH\nDtTTTz+t8ePHS2IeI6lHneJJT09XdHT0Kcm0urr6lJSLM2tfpX62efT5fGpsbFRNTc0Zx1xK5s+f\nr1deeUXr1q1TVlaWtZ257JzY2FgNGjRIY8aMUXFxsUaOHKlHH32UeeyEzZs3q7q6WgUFBYqJiVFM\nTIzKysr07//+74qJibHmgrnsvMTEROXn52v37t38mbSBHhVQYmNjVVBQoNLS0rDtpaWlKiwsjFBX\nPU9ubq58Pl/YPDY2NqqsrMyax4KCAjmdzrAxlZWV2rFjxyU118YYzZs3Ty+99JLefPNN5ebmhu1n\nLi+MMUahUIh57IQpU6aovLxc27Zts2rMmDH68pe/rG3btumyyy5jLs9TKBTSe++9p8zMTP5M2kEk\nVuZeiPbLjH/zm9+Yd9991xQVFZnExESzZ8+eSLdmK3V1dWbr1q1m69atRpJ5+OGHzdatW63LsZcv\nX27cbrd56aWXTHl5ubnjjjtOe/lcVlaWeeONN8yWLVvM5MmTL7nL5775zW8at9tt1q9fH3Yp4vHj\nx60xzGXHLF682Lz11lumoqLCbN++3SxZssRERUWZ119/3RjDPF6Iz17FYwxz2VELFy4069evNx9/\n/LF55513zPTp001ycrL1ecI8RlaPCyjGGPPzn//c5OTkmNjYWHPllVdal3zi79atW2cknVKzZs0y\nxrRdQrd06VLj8/mMy+Uy11xzjSkvLw97joaGBjNv3jzj8XhMfHy8mT59utm3b18E3k3knG4OJZkn\nn3zSGsNcdsycOXOsv7d9+/Y1U6ZMscKJMczjhTg5oDCXHdN+XxOn02n8fr+59dZbzc6dO639zGNk\nOYwxJjLHbgAAAE6vR61BAQAAlwYCCgAAsB0CCgAAsB0CCgAAsB0CCgAAsB0CCgAAsB0CCgAAsB0C\nCgAAsB0CCgAAsB0CCgAAsB0CCgAAsB0CCgAAsJ3/D7wecIz/MGotAAAAAElFTkSuQmCC\n"
          }
        }
      ],
      "source": [
        "env.reset()\n",
        "plt.imshow(env.render())"
      ],
      "id": "714f450e-d0e3-4cb1-9421-50b7d2ce8c92"
    },
    {
      "cell_type": "code",
      "execution_count": 96,
      "metadata": {
        "tags": []
      },
      "outputs": [
        {
          "output_type": "display_data",
          "metadata": {},
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAF7CAYAAAD4/3BBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90\nbGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9h\nAAAPYQGoP6dpAAAzFUlEQVR4nO3de3RU5b3/8c/kNrkPmYRkMiSEyE0hATFQLj8ryFXaiKittNoW\niqunqLCaA/QCtgV7rEHaYj1FaW3rvTacU8XSI0VjlSDlWJFLCaiIyi2QEIFkJoEwCcnz+yMnuw7X\nBBJmJ7xfa30Xmb2fzHznCTCf7Hn2HocxxggAAMBGwkLdAAAAwOkIKAAAwHYIKAAAwHYIKAAAwHYI\nKAAAwHYIKAAAwHYIKAAAwHYIKAAAwHYIKAAAwHYIKAAAwHZCGlAef/xxZWdnKzo6Wnl5eXrrrbdC\n2Q4AALCJkAWUlStXqqCgQPfff7+2bt2qz3/+85o8ebL2798fqpYAAIBNOEL1YYHDhw/XddddpxUr\nVljbrrnmGk2dOlWFhYWhaAkAANhERCgetL6+Xps3b9YPfvCDoO0TJ07Uxo0bzxgfCAQUCASs201N\nTTp27JiSk5PlcDg6vF8AAHDpjDGqqamR1+tVWNj538QJSUA5cuSIGhsblZaWFrQ9LS1NFRUVZ4wv\nLCzUAw88cLnaAwAAHejAgQPKyMg475iQBJQWpx/9MMac9YjIggULNHfuXOu2z+dTz549O7w/AOfm\nTcvVhOvvV0pMP0WGx8p3skzvfPCEtmx/UcY0KiwsUl/K/6UyEkcqKiKu4xsyRqXlK/XGW8tUV+dT\nzlW3aFDOrYpxJqp73EA1NgX08ad/0982/kw+36GO7wfAOSUkJFxwTEgCSkpKisLDw884WlJZWXnG\nURVJcjqdcjqdl6s9ABcQ7XRpxJC7lRiTrviYVElSxadb9Mm+/5UxjZKafwGJioyRMypezoj4Du/J\nGKPIyGjrl5wPD7yu/ldNUnhUmBTWpLioFPVK/bz6992sd959vsP7AXBurVmeEZKzeKKiopSXl6fi\n4uKg7cXFxRo1alQoWgLQBhERUYpPSJYruqccjjAdq/tYnx77WFXVB84y+nKuw//Xf3r1Dce1/t1f\nyhXVU0frdskYo4Qoj/pljVdKSvZl7AnAxQjZacZz587V7373Oz355JN6//339e///u/av3+/Zs2a\nFaqWALRCsqu3vvyFx5QUna1wR5Qam+p1sPodvf3u06Fu7f/8K6R8Wr1bpbv+LGOMTjR8KocjTBmu\n4erfb7zCwyND2COACwnZGpRp06bp6NGj+slPfqLy8nLl5ORozZo1ysrKClVLAFph6thlCpij6hbd\nSw6HQ58ef1979r2jU6cCZ4w1Mmoyp9TY1HCWe2q/IyvGui+jzwYUyWjze8+rZ8ZQ1YSVKzoiSc7w\nBPVJH6d96e+orOyf7dYDgPYV0kWy9957r+69995QtgCgDQb3/ZL8jQfUy/V5ORxhOnnKpyP+3fpg\n92tnGW1U31ijw8e3K9wRdVn6O9VUp/Dw4Mc6EajW9g9W6bpBX9bJxmrFRqTIk5irq3peryNH9ujk\nSf9l6Q1A24Q0oADoXFJT+yox1qvYyGQZ0yTfyX3a9t6fdDJQe8bYxsZT+tv6R+RQW69VdPHXNmoy\np3TixLHgbU0N2nfwbXnTcxUdmaSYiCQ5IxLVL2OCPtm/gaMogE0RUAC0Sm6/qUpO7Sl3dG9JDp08\nVa3Kqg9UVVVmnbkTzOjYsX2Xu82zqq49qANl2xQTH69u0VmKCo9Xt9ieGtjvi6quPqja2iOhbhHA\nafg0YwCtUheoVl1djQ7XlsoX2C/fyf3a8dFfdLTqk1C31gpGH+4r1gmfXxW12+QPlGnvkQ36tPpD\nNTWdLVwBCDWOoABolY/2rVNt7RH16TVax9x7ZdSoqmNnO63YnuobavXe7rWKTUzQJwdLtGfvJu0/\nuCnUbQE4h5B9WOCl8Pv9crlcoW4DuGKlp+QqLMKhQ4d3yJimULfTJlf1vF7Hqveq2l8W6laAK5bP\n51NiYuJ5xxBQAADAZdWagMIaFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAA\nYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsE\nFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDsEFAAAYDvtHlAWL14sh8MR\nVB6Px9pvjNHixYvl9XoVExOjMWPGaOfOne3dBgAA6MQ65AjKwIEDVV5eblVpaam1b+nSpVq2bJmW\nL1+uTZs2yePxaMKECaqpqemIVgAAQCfUIQElIiJCHo/Hqu7du0tqPnryy1/+Uvfff79uu+025eTk\n6JlnntGJEyf0wgsvdEQrAACgE+qQgLJ79255vV5lZ2frK1/5ij755BNJ0p49e1RRUaGJEydaY51O\np0aPHq2NGzee8/4CgYD8fn9QAQCArqvdA8rw4cP17LPP6tVXX9Vvf/tbVVRUaNSoUTp69KgqKiok\nSWlpaUHfk5aWZu07m8LCQrlcLqsyMzPbu20AAGAjDmOM6cgHOH78uHr37q3vfe97GjFihP7f//t/\nOnTokNLT060x3/rWt3TgwAGtXbv2rPcRCAQUCASs236/n5ACAEAn5fP5lJiYeN4xHX6acVxcnHJz\nc7V7927rbJ7Tj5ZUVlaecVTls5xOpxITE4MKAAB0XR0eUAKBgN5//32lp6crOztbHo9HxcXF1v76\n+nqVlJRo1KhRHd0KAADoJCLa+w7nz5+vm2++WT179lRlZaUefPBB+f1+TZ8+XQ6HQwUFBXrooYfU\nt29f9e3bVw899JBiY2N15513tncrAACgk2r3gFJWVqavfvWrOnLkiLp3764RI0bo7bffVlZWliTp\ne9/7nurq6nTvvfeqqqpKw4cP12uvvaaEhIT2bgUAAHRSHb5ItiP4/X65XK5QtwEAAC6CLRbJAgAA\ntBUBBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4B\nBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA\n2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4B\nBQAA2A4BBQAA2E6bA8r69et18803y+v1yuFw6OWXXw7ab4zR4sWL5fV6FRMTozFjxmjnzp1BYwKB\ngObMmaOUlBTFxcVpypQpKisru6QnAgAAuo42B5Tjx49r8ODBWr58+Vn3L126VMuWLdPy5cu1adMm\neTweTZgwQTU1NdaYgoICrVq1SkVFRdqwYYNqa2uVn5+vxsbGi38mAACg6zCXQJJZtWqVdbupqcl4\nPB6zZMkSa9vJkyeNy+Uyv/71r40xxlRXV5vIyEhTVFRkjTl48KAJCwsza9eubdXj+nw+I4miKIqi\nqE5YPp/vgq/17boGZc+ePaqoqNDEiROtbU6nU6NHj9bGjRslSZs3b1ZDQ0PQGK/Xq5ycHGvM6QKB\ngPx+f1ABAICuq10DSkVFhSQpLS0taHtaWpq1r6KiQlFRUUpKSjrnmNMVFhbK5XJZlZmZ2Z5tAwAA\nm+mQs3gcDkfQbWPMGdtOd74xCxYskM/ns+rAgQPt1isAALCfdg0oHo9Hks44ElJZWWkdVfF4PKqv\nr1dVVdU5x5zO6XQqMTExqAAAQNfVrgElOztbHo9HxcXF1rb6+nqVlJRo1KhRkqS8vDxFRkYGjSkv\nL9eOHTusMQAA4MoW0dZvqK2t1UcffWTd3rNnj7Zt2ya3262ePXuqoKBADz30kPr27au+ffvqoYce\nUmxsrO68805Jksvl0t1336158+YpOTlZbrdb8+fPV25ursaPH99+zwwAAHRerTqv9zPefPPNs54y\nNH36dGNM86nGixYtMh6PxzidTnPDDTeY0tLSoPuoq6szs2fPNm6328TExJj8/Hyzf//+VvfAacYU\nRVEU1XmrNacZO4wxRp2M3++Xy+UKdRsAAOAi+Hy+C64n5bN4AACA7RBQAACA7RBQAACA7RBQAACA\n7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQ\nAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA\n7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7RBQAACA7bQ5oKxfv14333yzvF6v\nHA6HXn755aD9M2bMkMPhCKoRI0YEjQkEApozZ45SUlIUFxenKVOmqKys7JKeCAAA6DraHFCOHz+u\nwYMHa/ny5eccc9NNN6m8vNyqNWvWBO0vKCjQqlWrVFRUpA0bNqi2tlb5+flqbGxs+zMAAABdTkRb\nv2Hy5MmaPHnyecc4nU55PJ6z7vP5fPr973+v5557TuPHj5ckPf/888rMzNTrr7+uSZMmtbUlAADQ\nxXTIGpR169YpNTVV/fr107e+9S1VVlZa+zZv3qyGhgZNnDjR2ub1epWTk6ONGzee9f4CgYD8fn9Q\nAQCArqvdA8rkyZP1hz/8QW+88YZ+8YtfaNOmTRo7dqwCgYAkqaKiQlFRUUpKSgr6vrS0NFVUVJz1\nPgsLC+VyuazKzMxs77YBAICNtPktnguZNm2a9XVOTo6GDh2qrKwsvfLKK7rtttvO+X3GGDkcjrPu\nW7BggebOnWvd9vv9hBQAALqwDj/NOD09XVlZWdq9e7ckyePxqL6+XlVVVUHjKisrlZaWdtb7cDqd\nSkxMDCoAANB1dXhAOXr0qA4cOKD09HRJUl5eniIjI1VcXGyNKS8v144dOzRq1KiObgcAAHQCbX6L\np7a2Vh999JF1e8+ePdq2bZvcbrfcbrcWL16s22+/Xenp6dq7d68WLlyolJQU3XrrrZIkl8ulu+++\nW/PmzVNycrLcbrfmz5+v3Nxc66weAABwhTNt9OabbxpJZ9T06dPNiRMnzMSJE0337t1NZGSk6dmz\np5k+fbrZv39/0H3U1dWZ2bNnG7fbbWJiYkx+fv4ZY87H5/OdtQeKoiiKouxfPp/vgq/1DmOMUSfj\n9/vlcrlC3QYAALgIPp/vgutJ+SweAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQU\nAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABg\nOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgOwQU\nAABgOwQUAABgOwQUAABgOwQUAABgOwQUAABgO20KKIWFhRo2bJgSEhKUmpqqqVOnateuXUFjjDFa\nvHixvF6vYmJiNGbMGO3cuTNoTCAQ0Jw5c5SSkqK4uDhNmTJFZWVll/5sAABAl9CmgFJSUqL77rtP\nb7/9toqLi3Xq1ClNnDhRx48ft8YsXbpUy5Yt0/Lly7Vp0yZ5PB5NmDBBNTU11piCggKtWrVKRUVF\n2rBhg2pra5Wfn6/Gxsb2e2YAAKDzMpegsrLSSDIlJSXGGGOampqMx+MxS5YsscacPHnSuFwu8+tf\n/9oYY0x1dbWJjIw0RUVF1piDBw+asLAws3bt2lY9rs/nM5IoiqIoiuqE5fP5Lvhaf0lrUHw+nyTJ\n7XZLkvbs2aOKigpNnDjRGuN0OjV69Ght3LhRkrR582Y1NDQEjfF6vcrJybHGnC4QCMjv9wcVAADo\nui46oBhjNHfuXF1//fXKycmRJFVUVEiS0tLSgsampaVZ+yoqKhQVFaWkpKRzjjldYWGhXC6XVZmZ\nmRfbNgAA6AQuOqDMnj1b27dv1x//+Mcz9jkcjqDbxpgztp3ufGMWLFggn89n1YEDBy62bQAA0Alc\nVECZM2eOVq9erTfffFMZGRnWdo/HI0lnHAmprKy0jqp4PB7V19erqqrqnGNO53Q6lZiYGFQAAKDr\nalNAMcZo9uzZeumll/TGG28oOzs7aH92drY8Ho+Ki4utbfX19SopKdGoUaMkSXl5eYqMjAwaU15e\nrh07dlhjAADAFa61Z+wYY8w999xjXC6XWbdunSkvL7fqxIkT1pglS5YYl8tlXnrpJVNaWmq++tWv\nmvT0dOP3+60xs2bNMhkZGeb11183W7ZsMWPHjjWDBw82p06d4iweiqIoiuri1ZqzeNoUUM71QE89\n9ZQ1pqmpySxatMh4PB7jdDrNDTfcYEpLS4Pup66uzsyePdu43W4TExNj8vPzzf79+1vdBwGFoiiK\nojpvtSagOP4veHQqfr9fLpcr1G0AAICL4PP5LrielM/iAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNA\nAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAA\ntkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNA\nAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAtkNAAQAAttOmgFJYWKhhw4Yp\nISFBqampmjp1qnbt2hU0ZsaMGXI4HEE1YsSIoDGBQEBz5sxRSkqK4uLiNGXKFJWVlV36swEAAF1C\nmwJKSUmJ7rvvPr399tsqLi7WqVOnNHHiRB0/fjxo3E033aTy8nKr1qxZE7S/oKBAq1atUlFRkTZs\n2KDa2lrl5+ersbHx0p8RAADo/MwlqKysNJJMSUmJtW369OnmlltuOef3VFdXm8jISFNUVGRtO3jw\noAkLCzNr165t1eP6fD4jiaKo02rhQpkNG2T+9jeZJ5+U+cIXZJKTZdxumW7dZJzO0Pd4pdQXv9j8\ns1i3TmblSplvfrP5Z5GcLJOUJBMbG/oeKSpU5fP5LvhaH6FL4PP5JElutzto+7p165Samqpu3bpp\n9OjR+ulPf6rU1FRJ0ubNm9XQ0KCJEyda471er3JycrRx40ZNmjTpjMcJBAIKBALWbb/ffyltA11W\nRIQUHd1cgwY1lzHSqVNSZaW0bp309783b6uvlz79VDp0KNRdd03h4c0/B0mKj5fuu0+6916psVHy\n+aRNm6TVq5t/Fg0NUnW1tHdvKDsG7OWiA4oxRnPnztX111+vnJwca/vkyZP15S9/WVlZWdqzZ49+\n9KMfaezYsdq8ebOcTqcqKioUFRWlpKSkoPtLS0tTRUXFWR+rsLBQDzzwwMW2ClzRHA4pMlLq0UO6\n6y7pzjulpiaptlZ6/33p3Xebb9fVSfv2Se+8E+qOuy6HozlEJidLN90kTZrUHFBOnGie+3Xrmn8W\nJ09K5eXS//5vc3gBrkQXHVBmz56t7du3a8OGDUHbp02bZn2dk5OjoUOHKisrS6+88opuu+22c96f\nMUYOh+Os+xYsWKC5c+dat/1+vzIzMy+2deCK5nA0/3bvckkjRkjDhwcfUfnww+YXyePHm79+8cXm\n3/rR/hyO5oqPlwYOlAYMaN7e0CAdOybt2tX8dV1d89GV1aulqqqQtgxcNhcVUObMmaPVq1dr/fr1\nysjIOO/Y9PR0ZWVlaffu3ZIkj8ej+vp6VVVVBR1Fqays1KhRo856H06nU06n82JaBXABLS+S0dFS\nZqbU8k+6sbH5N/nJk6VvfjO0PV4pWn5Hi4qSPB4pLa35dlOTFAhIt97afBTstPMSgC6pTWfxGGM0\ne/ZsvfTSS3rjjTeUnZ19we85evSoDhw4oPT0dElSXl6eIiMjVVxcbI0pLy/Xjh07zhlQAHQcY5qr\nsbH5bZ9Dh6Sysubf3v/8Z+mee0Ld4ZXjsz+LEyekigrp4EHpo4+k115rXsNCOMGVok1HUO677z69\n8MIL+vOf/6yEhARrzYjL5VJMTIxqa2u1ePFi3X777UpPT9fevXu1cOFCpaSk6NZbb7XG3n333Zo3\nb56Sk5Pldrs1f/585ebmavz48e3/DAEEaXkRDASaXwD3729+QaypkbZvbw4luDxafhYtb+l89u21\n3bulP/2p+ecEXInaFFBWrFghSRozZkzQ9qeeekozZsxQeHi4SktL9eyzz6q6ulrp6em68cYbtXLl\nSiUkJFjjH3nkEUVEROiOO+5QXV2dxo0bp6efflrh4eGX/owABDGm+UWvuloqLZXeey94jck//xnq\nDq8cLT+L48ebj4r84x//WqB84EDzGVYAmrUpoBhjzrs/JiZGr7766gXvJzo6Wr/61a/0q1/9qi0P\nD6AVWn4jP3RIKi5uPipiTPOLYEWFdPhwqDu8crS8XXP0aHP4aDlLp75eOnKk+egVgLO7pOugALCX\njIyf67vf/b127nxfp041/6ZeXx/qrq5MSUnT9Mc/Ruq5555XY2NzQKyrC3VXQOdBQAG6kIgIt6qq\novTpp6HuBGFhsTp+nJ8FcLH4NGMAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQA\nAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7\nEaFuAACA9hQREaEePXqoX79+6tevn/r37y+32619+/Zp37592rt3r/bu3auysjKdOHEi1O3iHAgo\nAIBOx+FwWJWUlKRBgwbp2muv1ZAhQzRw4EClpKTI6XQqOjpa0dHRioiIUCAQUH19verr6xUIBFRX\nV6eysjLt2rVLH3zwgT744APt2rVL5eXlamhokCQZY2SMCfGzvTIRUAAAthUWFmYFDafTqYSEBPXu\n3VvXXnutrrvuOl177bXKyspSWFhYUGhxOBxn3FdMTIxiYmKs28YY9enTR6NHj7ZCiDFGfr9fH330\nkRVYWv70+XxWwGmpU6dOXba5uNIQUAAAthAWFqb4+Hi53W6r0tPTrbdprr76amVnZys+Pr5dHq8l\nxJweZtxutz73uc/pc5/7XND2w4cPa8+ePdZbRPv27dPBgwdVXV0tv98vn88nv98vv99PcGkHBBQA\nQEjEx8fL6/WqZ8+eyszMVGZmpjIyMtSjRw9lZGQoIyND3bp1U1iYPc7nSEtLU1pamkaMGGFtq6+v\nV2VlpQ4fPqzKykrr68OHD6uioiKojh07FsLuOx8CCgCgw0VERKhnz54aMGCArrnmGl199dXq2bOn\nkpKS1K1bN3Xr1k2JiYmKjIwMdattEhUVZYWpFsYY1dXVqaamJqiqqqq0Z88eqz7++GPt379ftbW1\nIXwG9kVAAQBcMofDYa0DCQsLU/fu3TV48GBdd911GjJkiHJyctStWzdFRUVZFR4efta1Ip2dw+FQ\nbGysYmNjlZaWZm1vamrSqVOn1NDQYNXJkyd16NAha5Hu+++/rw8++EBlZWWqr6+3Fuk2NTVdcYt1\nCSgAgDYJDw9XbGysteg0Pj5eV199ta699loNGjRIgwcPltfrPSOAdMUw0hZhYWFWOGthjFFGRoa1\n3qUlhNTW1mrPnj3atWuXPvzwQ+vPjz766Ip5q4iAAgA4r6SkJKWkpFjVo0cP9e3bV3369FHfvn11\n1VVXyel0hrrNTun00NZyOzExUYMHD9bgwYOD9j///PNavny5/vGPf1y2HkOFgAIAUFRUlJKTk62F\noGlpaUpNTVVaWprS09ODKjEx0TYLV680d911lwYNGqSnnnpKL7zwgiorK0PdUochoADAFSQmJkZZ\nWVnKyspSr169rK+9Xq8SEhKstRMtFRMTQxixEYfDodzcXP3kJz/RiBEjtGjRIu3atSvUbXUIAgoA\ndGIti1JPr5SUFPXp00e9e/dWnz59rLdj0tPTFR4eblVERITCw8OtBa6wP4fDoYSEBN1+++264YYb\n9N3vfld/+tOfFAgEQt1au2pTQFmxYoVWrFihvXv3SpIGDhyoH//4x5o8ebKk5sU9DzzwgJ544glV\nVVVp+PDheuyxxzRw4EDrPgKBgObPn68//vGPqqur07hx4/T4448HnaIFAPgXh8OhqKgoRUdHKyYm\nxrp8e3R0tLxer7Kzs63q1auXsrOz1a1bt3PeF7qGiIgIeTwePfvssxozZoyWLFmi/fv3W5fp7+za\nFFAyMjK0ZMkS9enTR5L0zDPP6JZbbtHWrVs1cOBALV26VMuWLdPTTz+tfv366cEHH9SECRO0a9cu\nJSQkSJIKCgr0l7/8RUVFRUpOTta8efOUn5+vzZs3Kzw8vP2fIQB0Ei2fK9NyXZCWcrvdSktLk8fj\nUXp6urxer/V1dHR0qNtGCLVc1n/mzJkaOnSoli1bptWrV8vn84W6tUvmMJd4YrXb7dbPfvYzzZw5\nU16vVwUFBfr+978vqfloSVpamh5++GF9+9vfls/nU/fu3fXcc89p2rRpkqRDhw4pMzNTa9as0aRJ\nk1r1mH6/Xy6X61LaxiUICwuzDhf37dtXvXv37nQXVzqbpqYmNTQ0WB8kdvpnbnz2Q8bOd/t84xsb\nGzv0OTz55JN69NFH9c9//rNDHwcX9s1vflNRUVH6zW9+c8a+6OhoK2B4PJ6gr7t162aFlKSkJCUl\nJSkhIaFL/BtDx6uoqNBf/vIX/fznP9eHH34Y6nbOyefzKTEx8bxjLjqgNDY26r//+781ffp0bd26\nVdHR0erdu7e2bNmiIUOGWONuueUWdevWTc8884zeeOMNjRs3TseOHVNSUpI1ZvDgwZo6daoeeOCB\nsz5WIBAIem/N7/crMzPzYtrGRYiOjtaQIUOsTwodNGiQXC6X4uPjlZCQoISEhC6xiK7lYkifrcbG\nxjO2Xcz2ln2nTp3SyZMnz6i6urpWbz/X2JMnT8rj8ejQoUOqq6sL9XRe8VJSUuR2uxUbG3vG2y9p\naWmKi4sLessmJiZGTqeTt2BwyRoaGvThhx/qP/7jP7Ry5cpQt3NWrQkobV4kW1paqpEjR+rkyZOK\nj4/XqlWrNGDAAG3cuFGSgq6a13J73759kpqTXVRUVFA4aRlTUVFxzscsLCw8Z3jBpXM4HIqMjFRk\nZKQiIiKUnJys4cOHa9iwYRo+fLhyc3ODFtV15as/tjy/jvLZT0y90J9tGXu2PxF6LX+nTl/Aeq5P\n2wXaQ2RkpAYMGKAnnnhCY8aM0U9/+lMdOnRITU1NoW6tTdocUPr3769t27apurpaL774oqZPn66S\nkhJr/+n/6IwxF/yHeKExCxYs0Ny5c63bHEG5NNHR0UpMTJTL5ZLL5VJGRoauvfZa66JA2dnZoW6x\nyzrXp6cCQHtyOBxKTEzUrFmzNHjwYD344INat26dTpw4EerWWq3NASUqKspaJDt06FBt2rRJjz76\nqLXupKKiQunp6db4yspK66iKx+NRfX29qqqqgo6iVFZWatSoUed8TKfTyVUKL4Hb7VaPHj2sysrK\nsk49vOqqq5SSkhLqFgEAHWTkyJF66qmn9MQTT+j555/vNNdNueTroBhjFAgElJ2dLY/Ho+LiYmsN\nSn19vUpKSvTwww9LkvLy8hQZGani4mLdcccdkqTy8nLt2LFDS5cuvdRW8H969eqla665Rv369VP/\n/v2VkZFhXRUyNTVVsbGxoW4RAHAZpaamasGCBRoxYoQef/xx/c///I/tT0duU0BZuHChJk+erMzM\nTNXU1KioqEjr1q3T2rVr5XA4VFBQoIceesg6u+Ohhx5SbGys7rzzTkmSy+XS3XffrXnz5ik5OVlu\nt1vz589Xbm6uxo8f3yFPsKsLCwvTkCFDdN1111nldrsVFxenuLg4xcbGKiKC6/EBwJUuPDxcN954\no3JzczVixAgtWbJEVVVVoW7rnNr0ynX48GF9/etfV3l5uVwulwYNGqS1a9dqwoQJkqTvfe97qqur\n07333mtdqO21116zroEiSY888ogiIiJ0xx13WBdqe/rpp7kGyjm0XKDJ6XQqKipK8fHxGjZsmFV5\neXmKioo6YwEeAACnCw8PV1pamubNm6fRo0dr5syZ+vjjj215FdpLvg5KKHTl66C0nOXUUmlpacrN\nzVVubq5ycnJ09dVXE0AAAO3ixIkTWrx4sV544QUdPHjwsj1uh14HJZS6UkBJTExUZmamevbsaVWv\nXr2s6yV8dsExAADt7cSJE3r11Ve1YsUKlZSUqL6+vsMfk4BiY4mJiZo5c6ZuuukmJScnKyUlRcnJ\nyUFvhwEAcLl8/PHHev755/XTn/60wxfQElBsyOFwaOTIkVqyZImuvfZaAgkAwDaOHz+unTt36p57\n7tHWrVs77MKPrQkonf/65J1EZGSk+vTpo0cffVTFxcW6/vrrCScAAFuJjY3VsGHDtHr1at17771K\nTk4O2bpHjqBcBtnZ2Zo8ebLmzJmjq6++OtTtAABwQQ0NDVq5cqUee+wxbdmypV3XpvAWT4i5XC7l\n5+fra1/7mm688UauhgsA6HS2b9+uP/zhD3ryySd15MiRdrlPAkoIDRs2TPPnz9fo0aOVmprKqcEA\ngE7JGKPa2lq98sorWr58ud55551LXkRLQLnMHA6HunXrpgULFmjmzJlyuVxcxRUA0CWcOnVKZWVl\n+tGPfqQ//elPCgQCF72IloBymTgcDrndbo0dO1aFhYW66qqrOGICAOhyWiLD008/rd/+9rfatGmT\nTp061eb7IaBcBnFxcfr85z+vGTNmaMqUKYqJiQl1SwAAdKimpibt2LFDDz/8sNavX6/y8nI1Nja2\n+vsJKB1s6NChmjFjhvLz85WVlRXqdgAAuKwqKyv16quvaunSpdq5c2er3/IhoHSQ5ORkzZo1S3fe\neaf69evHOhMAwBUrEAjoww8/1NKlS/X3v/9de/bsueD3EFDamdPp1KhRo/Twww9r0KBBioqKYq0J\nAABqfm3+r//6L/3whz/U0aNHz7s2hYDSTmJiYtS3b1/NmzdPX//61yWJYAIAwGe0xIktW7bo6aef\n1gsvvKBjx46ddWxrAgrvTZxHeHi4+vXrp6lTp+rb3/62evbsSTABAOAsWl4f8/Ly5PV6lZKSoqKi\nIh09elSffvpp2++PIyhnFx8fr+nTp+srX/mK8vLyODsHAIA2MMbozTff1MqVK/W73/1OTU1N1j6O\noFykMWPG6IEHHlBubq6SkpJC3Q4AAJ2Ow+HQmDFjdM0116h///7661//qrfeekuBQKB1388RlGbh\n4eHKysrSvHnzdNdddykxMZG3cwAAaAcNDQ3atGmTpk+fro8++ohFsq2VmZmpcePG6fvf/z6fNgwA\nQAdpef3mLZ4LcLvdGjdunGbMmKEvfOELoW4HAAD8nysyoISHh2v48OH69re/rUmTJik1NTXULQEA\ngM+44gJKXFycfvjDH+prX/ua0tLSFBkZGeqWAADAaa6IgOJwOBQfH6/Jkyfr5z//ubxer8LDw0Pd\nFgAAOIcuH1BiYmI0atQo/du//ZtuvvlmrmcCAEAn0KUDytChQzVt2jR96UtfUq9evULdDgAAaKUu\nGVDS09M1c+ZM3X777RoyZEio2wEAAG3UpQJKZGSkxowZox//+McaMmSIYmNjQ90SAAC4CF0ioERH\nRys9PV0PPPCA7rzzToWFhXEVWAAAOrFOHVDCw8PVp08f3XLLLZo7d65SU1MJJgAAdAGdOqB84xvf\n0De+8Q0NHz6cs3MAAOhCwtoyeMWKFRo0aJASExOVmJiokSNH6q9//au1f8aMGXI4HEE1YsSIoPsI\nBAKaM2eOUlJSFBcXpylTpqisrOyimv/Rj36kMWPGEE4AAOhi2hRQMjIytGTJEr377rt69913NXbs\nWN1yyy3auXOnNeamm25SeXm5VWvWrAm6j4KCAq1atUpFRUXasGGDamtrlZ+fr8bGxjY3n5yc3Obv\nAQAA9nfJn2bsdrv1s5/9THfffbdmzJih6upqvfzyy2cd6/P51L17dz333HOaNm2aJOnQoUPKzMzU\nmjVrNGnSpFY9Zls+DREAANhDW16/23QE5bMaGxtVVFSk48ePa+TIkdb2devWKTU1Vf369dO3vvUt\nVVZWWvs2b96shoYGTZw40drm9XqVk5OjjRs3nvOxAoGA/H5/UAEAgK6rzQGltLRU8fHxcjqdmjVr\nllatWqUBAwZIkiZPnqw//OEPeuONN/SLX/xCmzZt0tixYxUIBCRJFRUVioqKUlJSUtB9pqWlqaKi\n4pyPWVhYKJfLZVVmZmZb2wYAAJ1Im8/i6d+/v7Zt26bq6mq9+OKLmj59ukpKSjRgwADrbRtJysnJ\n0dChQ5WVlaVXXnlFt9122znv0xhz3tODFyxYoLlz51q3/X4/IQUAgC6szQElKipKffr0kdT8WTeb\nNm3So48+qt/85jdnjE1PT1dWVpZ2794tSfJ4PKqvr1dVVVXQUZTKykqNGjXqnI/pdDrldDrb2ioA\nAOikLnoNSgtjjPUWzumOHj2qAwcOKD09XZKUl5enyMhIFRcXW2PKy8u1Y8eO8wYUAABwZWnTEZSF\nCxdq8uTJyszMVE1NjYqKirRu3TqtXbtWtbW1Wrx4sW6//Xalp6dr7969WrhwoVJSUnTrrbdKklwu\nl+6++27NmzdPycnJcrvdmj9/vnJzczV+/PgOeYIAAKDzaVNAOXz4sL7+9a+rvLxcLpdLgwYN0tq1\nazVhwgTV1dWptLRUzz77rKqrq5Wenq4bb7xRK1euVEJCgnUfjzzyiCIiInTHHXeorq5O48aN09NP\nP63w8PB2f3IAAKBzuuTroIQC10EBAKDzuSzXQQEAAOgoBBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7\nBBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQA\nAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7\nBBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7BBQAAGA7EaFu\n4GIYYyRJfr8/xJ0AAIDWanndbnkdP59OGVBqamokSZmZmSHuBAAAtFVNTY1cLtd5xzhMa2KMzTQ1\nNWnXrl0aMGCADhw4oMTExFC31Gn5/X5lZmYyj+2AuWw/zGX7YB7bD3PZPowxqqmpkdfrVVjY+VeZ\ndMojKGFhYerRo4ckKTExkb8s7YB5bD/MZfthLtsH89h+mMtLd6EjJy1YJAsAAGyHgAIAAGyn0wYU\np9OpRYsWyel0hrqVTo15bD/MZfthLtsH89h+mMvLr1MukgUAAF1bpz2CAgAAui4CCgAAsB0CCgAA\nsB0CCgAAsJ1OGVAef/xxZWdnKzo6Wnl5eXrrrbdC3ZLtrF+/XjfffLO8Xq8cDodefvnloP3GGC1e\nvFher1cxMTEaM2aMdu7cGTQmEAhozpw5SklJUVxcnKZMmaKysrLL+CxCr7CwUMOGDVNCQoJSU1M1\ndepU7dq1K2gMc9k6K1as0KBBg6wLXY0cOVJ//etfrf3M48UpLCyUw+FQQUGBtY25bJ3FixfL4XAE\nlcfjsfYzjyFmOpmioiITGRlpfvvb35r33nvPfOc73zFxcXFm3759oW7NVtasWWPuv/9+8+KLLxpJ\nZtWqVUH7lyxZYhISEsyLL75oSktLzbRp00x6errx+/3WmFmzZpkePXqY4uJis2XLFnPjjTeawYMH\nm1OnTl3mZxM6kyZNMk899ZTZsWOH2bZtm/niF79oevbsaWpra60xzGXrrF692rzyyitm165dZteu\nXWbhwoUmMjLS7NixwxjDPF6Md955x/Tq1csMGjTIfOc737G2M5ets2jRIjNw4EBTXl5uVWVlpbWf\neQytThdQPve5z5lZs2YFbbv66qvND37wgxB1ZH+nB5Smpibj8XjMkiVLrG0nT540LpfL/PrXvzbG\nGFNdXW0iIyNNUVGRNebgwYMmLCzMrF279rL1bjeVlZVGkikpKTHGMJeXKikpyfzud79jHi9CTU2N\n6du3rykuLjajR4+2Agpz2XqLFi0ygwcPPus+5jH0OtVbPPX19dq8ebMmTpwYtH3ixInauHFjiLrq\nfPbs2aOKioqgeXQ6nRo9erQ1j5s3b1ZDQ0PQGK/Xq5ycnCt6rn0+nyTJ7XZLYi4vVmNjo4qKinT8\n+HGNHDmSebwI9913n774xS9q/PjxQduZy7bZvXu3vF6vsrOz9ZWvfEWffPKJJObRDjrVhwUeOXJE\njY2NSktLC9qelpamioqKEHXV+bTM1dnmcd++fdaYqKgoJSUlnTHmSp1rY4zmzp2r66+/Xjk5OZKY\ny7YqLS3VyJEjdfLkScXHx2vVqlUaMGCA9Z8589g6RUVF2rJlizZt2nTGPv5Ott7w4cP17LPPql+/\nfjp8+LAefPBBjRo1Sjt37mQebaBTBZQWDocj6LYx5oxtuLCLmccrea5nz56t7du3a8OGDWfsYy5b\np3///tq2bZuqq6v14osvavr06SopKbH2M48XduDAAX3nO9/Ra6+9pujo6HOOYy4vbPLkydbXubm5\nGjlypHr37q1nnnlGI0aMkMQ8hlKneosnJSVF4eHhZyTTysrKM1Iuzq1llfr55tHj8ai+vl5VVVXn\nHHMlmTNnjlavXq0333xTGRkZ1nbmsm2ioqLUp08fDR06VIWFhRo8eLAeffRR5rENNm/erMrKSuXl\n5SkiIkIREREqKSnRf/7nfyoiIsKaC+ay7eLi4pSbm6vdu3fzd9IGOlVAiYqKUl5enoqLi4O2FxcX\na9SoUSHqqvPJzs6Wx+MJmsf6+nqVlJRY85iXl6fIyMigMeXl5dqxY8cVNdfGGM2ePVsvvfSS3njj\nDWVnZwftZy4vjTFGgUCAeWyDcePGqbS0VNu2bbNq6NChuuuuu7Rt2zZdddVVzOVFCgQCev/995We\nns7fSTsIxcrcS9FymvHvf/97895775mCggITFxdn9u7dG+rWbKWmpsZs3brVbN261Ugyy5YtM1u3\nbrVOx16yZIlxuVzmpZdeMqWlpearX/3qWU+fy8jIMK+//rrZsmWLGTt27BV3+tw999xjXC6XWbdu\nXdCpiCdOnLDGMJets2DBArN+/XqzZ88es337drNw4UITFhZmXnvtNWMM83gpPnsWjzHMZWvNmzfP\nrFu3znzyySfm7bffNvn5+SYhIcF6PWEeQ6vTBRRjjHnsscdMVlaWiYqKMtddd511yif+5c033zSS\nzqjp06cbY5pPoVu0aJHxeDzG6XSaG264wZSWlgbdR11dnZk9e7Zxu90mJibG5Ofnm/3794fg2YTO\n2eZQknnqqaesMcxl68ycOdP6d9u9e3czbtw4K5wYwzxeitMDCnPZOi3XNYmMjDRer9fcdtttZufO\nndZ+5jG0HMYYE5pjNwAAAGfXqdagAACAKwMBBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4B\nBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2A4BBQAA2M7/B1JFgE2575gzAAAAAElFTkSuQmCC\n"
          }
        }
      ],
      "source": [
        "env.step(1)\n",
        "plt.imshow(env.render())"
      ],
      "id": "f70e4b61-174b-47d7-a8c9-0897987a7a52"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "-   0 : 아무행동도 하지 않음\n",
        "-   1 : 왼쪽\n",
        "-   2 : 위\n",
        "-   3 : 오른쪽\n",
        "\n",
        "# 5. 시각화"
      ],
      "id": "dfe98041-24ef-4428-82e6-1ce47335828e"
    },
    {
      "cell_type": "code",
      "execution_count": 97,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "current_state, _ = env.reset()\n",
        "ims = [] \n",
        "for t in range(500): \n",
        "    action = env.action_space.sample()\n",
        "    next_state, reward, terminated, _, _ = env.step(action)\n",
        "    ims.append(env.render()) \n",
        "    current_state = next_state \n",
        "    if terminated: break "
      ],
      "id": "6f6c3d2e-36a4-4591-8d5e-198baf62a1b9"
    },
    {
      "cell_type": "code",
      "execution_count": 98,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "show(ims) "
      ],
      "id": "aad2ec81-b1cc-48d3-98a2-73ff7cf2a070"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 6. `AgentRandom`"
      ],
      "id": "99914b12-b03f-4f4d-93d6-3660091be130"
    },
    {
      "cell_type": "code",
      "execution_count": 99,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "env.step(0)"
      ],
      "id": "1dd35ce1-414d-4274-bd6c-fb64c19d8fa7"
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "class AgentRandom: \n",
        "    def __init__(self,env):\n",
        "        #--# define spaces \n",
        "        self.action_space = env.action_space\n",
        "        self.state_space = env.observation_space\n",
        "        #--# replay buffer \n",
        "        self.current_state =  None    ## 길이가 8인 np.array        \n",
        "        self.action = None            ## int, 0,1,2,3 중 하나\n",
        "        self.reward = None            ## float  \n",
        "        self.next_state =  None       ## np.array\n",
        "        self.terminated = None        ## bool \n",
        "        #-#\n",
        "        self.buffer_size = 5000 \n",
        "        self.current_states = collections.deque(maxlen=self.buffer_size) # 원소는 텐서         \n",
        "        self.actions = collections.deque(maxlen=self.buffer_size) # 원소는 텐서 \n",
        "        self.rewards = collections.deque(maxlen=self.buffer_size) # 원소는 텐서\n",
        "        self.next_states = collections.deque(maxlen=self.buffer_size) # 원소는 텐서 \n",
        "        self.terminations = collections.deque(maxlen=self.buffer_size) # 원소는 텐서 \n",
        "        #--# other information \n",
        "        self.n_episodes = 0         \n",
        "        self.n_experiences = 0\n",
        "        self.playtimes = [] \n",
        "        self.score = 0                \n",
        "        self.scores = []\n",
        "    def act(self):\n",
        "        self.action = self.action_space.sample()\n",
        "    def learn(self):\n",
        "        pass \n",
        "    def save_experience(self):\n",
        "        self.current_states.append(torch.tensor(self.current_state))\n",
        "        self.actions.append(torch.tensor(self.action))\n",
        "        self.rewards.append(torch.tensor(self.reward))\n",
        "        self.next_states.append(torch.tensor(self.next_state))\n",
        "        self.terminations.append(torch.tensor(self.terminated))           \n",
        "        #--#\n",
        "        self.n_experiences = self.n_experiences + 1 \n",
        "        self.score = self.score + self.reward"
      ],
      "id": "65170a8c-ebac-4fba-8e62-706392685423"
    },
    {
      "cell_type": "code",
      "execution_count": 101,
      "metadata": {
        "tags": []
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "에피소드:100    경험(t):9227  점수(에피소드):-187.87    게임시간(에피소드):92.27\n",
            "에피소드:200    경험(t):19217 점수(에피소드):-180.42    게임시간(에피소드):99.90\n",
            "에피소드:300    경험(t):28415 점수(에피소드):-195.30    게임시간(에피소드):91.98\n",
            "에피소드:400    경험(t):37470 점수(에피소드):-179.93    게임시간(에피소드):90.55\n",
            "에피소드:500    경험(t):46602 점수(에피소드):-195.32    게임시간(에피소드):91.32"
          ]
        }
      ],
      "source": [
        "env = gym.make(\"LunarLander-v2\",render_mode = 'rgb_array')\n",
        "agent = AgentRandom(env)\n",
        "#--#\n",
        "for _ in range(500): \n",
        "    agent.current_state,_ = env.reset()\n",
        "    agent.score = 0\n",
        "    for t in range(1,1001):\n",
        "        ## step1: 행동 \n",
        "        agent.act()\n",
        "        ## step2: 보상\n",
        "        agent.next_state, agent.reward, agent.terminated, _, _ = env.step(agent.action)\n",
        "        ## step3: 저장 & 학습 \n",
        "        agent.save_experience()\n",
        "        agent.learn()\n",
        "        ## step4: \n",
        "        agent.current_state = agent.next_state \n",
        "        if agent.terminated: break \n",
        "    agent.scores.append(agent.score) \n",
        "    agent.playtimes.append(t) \n",
        "    agent.n_episodes = agent.n_episodes + 1 \n",
        "    #---#\n",
        "    logfreq = 100\n",
        "    if (agent.n_episodes % logfreq) == 0: \n",
        "        print(\n",
        "            f\"에피소드:{agent.n_episodes}\\t\"\n",
        "            f\"경험(t):{agent.n_experiences}\\t\"            \n",
        "            f\"점수(에피소드):{np.mean(agent.scores[-logfreq:]):.2f}\\t\"\n",
        "            f\"게임시간(에피소드):{np.mean(agent.playtimes[-logfreq:]):.2f}\"\n",
        "        )    "
      ],
      "id": "af82faec-b149-45f4-926a-01d2e4d25698"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 7. `AgentExplorer`\n",
        "\n",
        "## A. `q_net`\n",
        "\n",
        "`-` 전략: 4x4에서 `q_table`에 대응하는 정보가 있으면 된다. 그런데\n",
        "`q_table`와 같이 테이블 형식으로는 힘들것 같다. $\\to$ `q_net`를 만들자.\n",
        "\n",
        "-   4x4 grid: 상태공간의 차원은 2차원이며 가질수 있는 값은 16개, 각\n",
        "    상태공간에서 할수 있는 행동이 4개 -\\> 총 16\\*4의 경우의 수에 대한\n",
        "    reward만 조사하면 되었음\n",
        "-   LunarLander: 상태공간의 차원은 8차원이지만 가질수 있는 값의 범위는\n",
        "    무한대 -\\> 무수히 많은 경우에 대한 reward 값을 조사하는건 현실적으로\n",
        "    불가능\n",
        "\n",
        "`-` 4x4 코드\n",
        "\n",
        "``` python\n",
        "class AgentGreedy(AgentRandom):\n",
        "    def __init__(self,env):\n",
        "        super().__init__(env)\n",
        "        #--#\n",
        "        self.q_table = np.zeros([4,4,4])\n",
        "    def learn(self): # q_table \n",
        "        s1,s2 = self.current_state\n",
        "        ss1,ss2 = self.next_state\n",
        "        a = self.action\n",
        "        r = self.reward\n",
        "        q_hat = self.q_table[s1,s2,a] \n",
        "        if self.terminated:\n",
        "            q = r \n",
        "        else:\n",
        "            future_reward = self.q_table[ss1,ss2,:].max()\n",
        "            q = r + 0.99 * future_reward \n",
        "        diff = q - q_hat\n",
        "        self.q_table[s1,s2,a] = q_hat + 0.05 * diff         \n",
        "    def act(self):\n",
        "        if self.n_experiences < 3000:\n",
        "            self.action = self.action_space.sample()\n",
        "        else: \n",
        "            s1,s2 = self.current_state \n",
        "            self.action = self.q_table[s1,s2,:].argmax()\n",
        "```\n",
        "\n",
        "`수정 1`. `agent.q_table` 에 대응하는 과정"
      ],
      "id": "a4cf5147-0a57-4584-a0d5-7a7c2d15a8f3"
    },
    {
      "cell_type": "code",
      "execution_count": 102,
      "metadata": {},
      "outputs": [],
      "source": [
        "agent.q_net = torch.nn.Sequential(\n",
        "    torch.nn.Linear(8,256),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(256,128),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(128,64),\n",
        "    torch.nn.ReLU(),    \n",
        "    torch.nn.Linear(64,4)\n",
        ")    "
      ],
      "id": "30790a05-1a92-4033-af74-7beb4170dc06"
    },
    {
      "cell_type": "code",
      "execution_count": 103,
      "metadata": {},
      "outputs": [],
      "source": [
        "agent.q_net # <- 8개의 숫작가 들어가면 4개의 숫자가 나옴 "
      ],
      "id": "9113fb6c-0938-45d5-88e6-1577c948a291"
    },
    {
      "cell_type": "code",
      "execution_count": 104,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "s = torch.tensor(agent.current_state)\n",
        "agent.q_net(s)"
      ],
      "id": "8ce8d1d7-84f5-45d5-90f6-64beb165b0c6"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "-   `q_net`은 8개의 숫자가 입력으로 오면 4개의 숫자가 리턴되는 함수이다.\n",
        "-   해석을 하면 8개의 숫자는 state를 나타내는 숫자로 이해할 수 있고\n",
        "    4개의 숫자는 각 action에 대한 q값으로 해석할 수 있다.\n",
        "-   출력되는 4개의 숫자는 합리적인 숫자가 아님 (최초의 숫자임) 그렇지만\n",
        "    데이터를 학습하여 점점 합리적으로 변할 것임.\n",
        "\n",
        "`2`. `q_hat`"
      ],
      "id": "3631ca8c-f016-41b9-8f81-4f3ca826a53d"
    },
    {
      "cell_type": "code",
      "execution_count": 105,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "s = torch.tensor(agent.current_state)\n",
        "a = torch.tensor(agent.action)\n",
        "q_hat = agent.q_net(s)[a]"
      ],
      "id": "48ce4790-c57a-4671-8df5-06dc9680081a"
    },
    {
      "cell_type": "code",
      "execution_count": 106,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "q_hat"
      ],
      "id": "23602a73-59af-4334-bf0a-0c14d777acef"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`3`. `q` ($q = r + 0.99 \\times {\\tt future\\_reward}$)"
      ],
      "id": "c9bc4bcb-cf2e-43be-b1a8-60141e3ab677"
    },
    {
      "cell_type": "code",
      "execution_count": 107,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "s = torch.tensor(agent.current_state)\n",
        "a = torch.tensor(agent.action)\n",
        "r = torch.tensor(agent.reward)\n",
        "ss = torch.tensor(agent.next_state)"
      ],
      "id": "a6eca33d-7cae-4adf-9ee0-0fc17b62332c"
    },
    {
      "cell_type": "code",
      "execution_count": 108,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "if agent.terminated:\n",
        "    q = r\n",
        "else:\n",
        "    future_reward = agent.q_net(ss).max().data\n",
        "    q = r + 0.99 * future_reward "
      ],
      "id": "463c6b32-8a7c-413e-97c1-8661942ec932"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`4`. `q_hat` 을 점점 `q` 와 비슷하게 만드는 과정"
      ],
      "id": "1c686cfc-eb8d-4a8d-b2ae-d77c7c0ec447"
    },
    {
      "cell_type": "code",
      "execution_count": 109,
      "metadata": {},
      "outputs": [],
      "source": [
        "memory = list(zip(agent.current_states,agent.actions,agent.rewards,agent.next_states,agent.terminations))"
      ],
      "id": "e0c6814a-a808-4833-af49-ea34b7bd3e0b"
    },
    {
      "cell_type": "code",
      "execution_count": 110,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "agent.batch_size = 4 \n",
        "minibatch = random.sample(memory,agent.batch_size)\n",
        "minibatch[0] # s,a,r,ss,t "
      ],
      "id": "62a22c4c-6b22-4a2e-9060-70adc6f7df5b"
    },
    {
      "cell_type": "code",
      "execution_count": 111,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "agent.optimizr = torch.optim.Adam(agent.q_net.parameters())\n",
        "for epoc in range(5):\n",
        "    memory = list(zip(agent.current_states,agent.actions,agent.rewards,agent.next_states,agent.terminations))\n",
        "    minibatch = random.sample(memory,agent.batch_size)\n",
        "    ## step 1~2 \n",
        "    loss = 0 \n",
        "    for s,a,r,ss,terminated in minibatch:\n",
        "        # step1: q_hat \n",
        "        q_hat = agent.q_net(s)[a]        \n",
        "        # step2: loss를 계산한다. \n",
        "        if agent.terminated:\n",
        "            q = r\n",
        "        else:\n",
        "            future_reward = agent.q_net(ss).max().data\n",
        "            q = r + 0.99 * future_reward\n",
        "        loss = loss + (q_hat-q)**2 \n",
        "    loss = loss / agent.batch_size \n",
        "    # step3 \n",
        "    loss.backward()\n",
        "    # step4 \n",
        "    agent.optimizr.step()\n",
        "    agent.optimizr.zero_grad() "
      ],
      "id": "f8b90555-06e5-4f16-9b53-1ef68cc36b0f"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "`5`. 행동..?"
      ],
      "id": "b6134b04-ef3f-4be1-a5da-1c7dbb17242c"
    },
    {
      "cell_type": "code",
      "execution_count": 112,
      "metadata": {},
      "outputs": [],
      "source": [
        "agent.q_net(s).argmax().item()"
      ],
      "id": "0f26f726-cba4-440d-91a1-1a0a265daa71"
    },
    {
      "cell_type": "code",
      "execution_count": 113,
      "metadata": {},
      "outputs": [],
      "source": [
        "agent.eps = 0.5 \n",
        "if np.random.rand() < agent.eps: \n",
        "    agent.action = agent.action_space.sample()\n",
        "else: \n",
        "    s = torch.tensor(agent.current_state)\n",
        "    agent.q_net(s).argmax().item()    "
      ],
      "id": "44db8edb-24b9-4c79-9561-facdce2fc7fe"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## B. 클래스의 설계"
      ],
      "id": "5c00abc6-87d7-4216-8552-7d4e6d03d7c2"
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [],
      "source": [
        "class AgentExplorer(AgentRandom):\n",
        "    def __init__(self,env):\n",
        "        super().__init__(env)\n",
        "        self.eps = 0 \n",
        "        self.q_net = torch.nn.Sequential(\n",
        "            torch.nn.Linear(8,256),\n",
        "            torch.nn.ReLU(),\n",
        "            torch.nn.Linear(256,128),\n",
        "            torch.nn.ReLU(),\n",
        "            torch.nn.Linear(128,64),\n",
        "            torch.nn.ReLU(),    \n",
        "            torch.nn.Linear(64,4)\n",
        "        )\n",
        "        self.optimizr = torch.optim.Adam(self.q_net.parameters(),lr=0.0001)\n",
        "        self.batch_size = 64\n",
        "    def act(self):\n",
        "        if np.random.rand() < self.eps: \n",
        "            self.action = self.action_space.sample()\n",
        "        else: \n",
        "            s = torch.tensor(self.current_state)\n",
        "            self.action = self.q_net(s).argmax().item() \n",
        "    def learn(self):\n",
        "        if self.n_experiences < self.batch_size:\n",
        "            pass \n",
        "        else: \n",
        "            for epoc in range(1):\n",
        "                memory = list(zip(\n",
        "                    self.current_states,\n",
        "                    self.actions,\n",
        "                    self.rewards,\n",
        "                    self.next_states,\n",
        "                    self.terminations\n",
        "                ))\n",
        "                minibatch = random.sample(memory,self.batch_size)\n",
        "                ## step 1~2 \n",
        "                loss = 0 \n",
        "                for s,a,r,ss,terminated in minibatch:\n",
        "                    # step1: q_hat \n",
        "                    q_hat = self.q_net(s)[a]        \n",
        "                    # step2: loss를 계산한다. \n",
        "                    if self.terminated:\n",
        "                        q = r\n",
        "                    else:\n",
        "                        future_reward = self.q_net(ss).max().data\n",
        "                        q = r + 0.99 * future_reward\n",
        "                    loss = loss + (q_hat-q)**2 \n",
        "                loss = loss / self.batch_size \n",
        "                # step3 \n",
        "                loss.backward()\n",
        "                # step4 \n",
        "                self.optimizr.step()\n",
        "                self.optimizr.zero_grad() "
      ],
      "id": "df5ee74a-5ede-4860-bdd1-894e3f16b2c8"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## C. 환경과 상호작용"
      ],
      "id": "bc830ae8-1ca0-4bae-afe0-6a7475fb995f"
    },
    {
      "cell_type": "code",
      "execution_count": 70,
      "metadata": {
        "tags": []
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "에피소드:100    경험(t):10838 점수(에피소드):-176.71    게임시간(에피소드):108.38   돌발행동(에피소드):0.61\n",
            "에피소드:200    경험(t):51270 점수(에피소드):-56.40 게임시간(에피소드):404.32   돌발행동(에피소드):0.37\n",
            "에피소드:300    경험(t):127024    점수(에피소드):97.89  게임시간(에피소드):757.54   돌발행동(에피소드):0.22\n",
            "에피소드:400    경험(t):181605    점수(에피소드):161.62 게임시간(에피소드):545.81   돌발행동(에피소드):0.13\n",
            "---game cleared in 468 episodes!---"
          ]
        }
      ],
      "source": [
        "env = gym.make(\"LunarLander-v2\",render_mode = 'rgb_array')\n",
        "agent = AgentExplorer(env)\n",
        "agent.eps = 1 \n",
        "#--#\n",
        "for _ in range(2000): \n",
        "    # STEP1: 에피소드 준비 \n",
        "    agent.current_state,_ = env.reset()\n",
        "    agent.score = 0\n",
        "    # STEP2: 에피소드 진행 \n",
        "    for t in range(1,1001):\n",
        "        ## step1: 행동 \n",
        "        agent.act()\n",
        "        ## step2: 보상\n",
        "        agent.next_state, agent.reward, agent.terminated, _, _ = env.step(agent.action)\n",
        "        ## step3: 저장 & 학습 \n",
        "        agent.save_experience()\n",
        "        agent.learn()\n",
        "        ## step4: \n",
        "        agent.current_state = agent.next_state \n",
        "        if agent.terminated: break \n",
        "    # STEP3: 다음에피소드준비 \n",
        "    agent.scores.append(agent.score) \n",
        "    agent.playtimes.append(t) \n",
        "    agent.n_episodes = agent.n_episodes + 1\n",
        "    agent.eps = agent.eps * 0.995 \n",
        "    if np.mean(agent.scores[-100:]) > 200: \n",
        "        print(f\"---game cleared in {agent.n_episodes} episodes! ---\")\n",
        "        torch.save(agent.q_net.state_dict(),f\"q_net_final.pth\")\n",
        "        break \n",
        "    #---#\n",
        "    logfreq = 100\n",
        "    if (agent.n_episodes % logfreq) == 0: \n",
        "        print(\n",
        "            f\"에피소드:{agent.n_episodes}\\t\"\n",
        "            f\"경험(t):{agent.n_experiences}\\t\"            \n",
        "            f\"점수(에피소드):{np.mean(agent.scores[-logfreq:]):.2f}\\t\"\n",
        "            f\"게임시간(에피소드):{np.mean(agent.playtimes[-logfreq:]):.2f}\\t\"\n",
        "            f\"돌발행동(에피소드):{agent.eps:.2f}\"\n",
        "        )\n",
        "        torch.save(agent.q_net.state_dict(),f\"q_net_{agent.n_episodes}.pth\")"
      ],
      "id": "f4801de3-ee06-4857-83cd-951ffb3dfac1"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "------------------------------------------------------------------------\n",
        "\n",
        "***아래코드 실행하면 제가 실습에 사용한 파일 받아올수있어요***\n",
        "\n",
        "``` python\n",
        "!wget https://github.com/guebin/DL2024/raw/main/posts/LunarLander/q_net_100.pth\n",
        "!wget https://github.com/guebin/DL2024/raw/main/posts/LunarLander/q_net_200.pth\n",
        "!wget https://github.com/guebin/DL2024/raw/main/posts/LunarLander/q_net_300.pth\n",
        "!wget https://github.com/guebin/DL2024/raw/main/posts/LunarLander/q_net_400.pth\n",
        "!wget https://github.com/guebin/DL2024/raw/main/posts/LunarLander/q_net_final.pth\n",
        "```\n",
        "\n",
        "------------------------------------------------------------------------"
      ],
      "id": "1b3a2d5d-6ba8-430e-8d8c-34e986b1ed24"
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "agent_dummy = AgentExplorer(env) \n",
        "#agent_dummy.q_net = agent.q_net # 비법전수 \n",
        "agent_dummy.q_net.load_state_dict(torch.load(\"q_net_final.pth\"))\n",
        "\n",
        "agent_dummy.current_state, _ = env.reset()\n",
        "agent_dummy.terminated = False \n",
        "ims = [] \n",
        "ims.append(env.render())\n",
        "for t in range(1001):\n",
        "    agent_dummy.act() \n",
        "    agent_dummy.next_state, agent_dummy.reward, agent_dummy.terminated, _, _  = env.step(agent_dummy.action)\n",
        "    ims.append(env.render())\n",
        "    agent_dummy.current_state = agent_dummy.next_state\n",
        "    if agent_dummy.terminated: break "
      ],
      "id": "043e8f48-7e6c-4055-820c-a35e1b94b064"
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "tags": []
      },
      "outputs": [],
      "source": [
        "show(ims)"
      ],
      "id": "de9ee484-ac4c-42f6-a51e-dac36ffe1a70"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# A1. 신경망관련 용어\n",
        "\n",
        "`-` 은근히 용어가 헷갈리는데, 뜻을 좀 살펴보자.\n",
        "\n",
        "-   ANN: 인공신경망\n",
        "-   MLP: 다층퍼셉트론 (레이어가 여러개 있어요)\n",
        "-   DNN: 깊은신경망, 심층신경망\n",
        "-   CNN: 합성곱신경망\n",
        "-   RNN: 순환신경망\n",
        "\n",
        "`# 예시1` – MLP, DNN\n",
        "\n",
        "``` python\n",
        "net = torch.nn.Sequential(\n",
        "    torch.nn.Linear(in_features=1,out_features=2),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(in_features=2,out_features=2),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(in_features=2,out_features=1),    \n",
        "    torch.nn.Sigmoid()\n",
        ")\n",
        "```\n",
        "\n",
        "-   ANN: O\n",
        "-   MLP: O\n",
        "-   DNN: O\n",
        "-   CNN: X (합성곱레이어가 없으므로)\n",
        "-   RNN: X (순환구조가 없으므로)\n",
        "\n",
        "`#`\n",
        "\n",
        "`# 예시2` – MLP, Shallow Network\n",
        "\n",
        "``` python\n",
        "net = torch.nn.Sequential(\n",
        "    torch.nn.Linear(in_features=1,out_features=2),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(in_features=2,out_features=1),\n",
        "    torch.nn.Sigmoid()\n",
        ")\n",
        "```\n",
        "\n",
        "-   ANN: O\n",
        "-   MLP: O\n",
        "-   DNN: X? (깊은 신경망으로 생각하려면 더 많은 레이어가 필요함. 합의된\n",
        "    기준은 히든레이어 2장이상, 이걸 설명하기 위해서 얕은 신경망이란\n",
        "    용어도 씀)\n",
        "-   CNN: X (합성곱레이어가 없으므로)\n",
        "-   RNN: X (순환구조가 없으므로)\n",
        "\n",
        "`#`\n",
        "\n",
        "`# 예시3` – MLP, DNN, Wide NN\n",
        "\n",
        "``` python\n",
        "net = torch.nn.Sequential(\n",
        "    torch.nn.Linear(in_features=1,out_features=1048576),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(in_features=1048576,out_features=1048576),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(in_features=1048576,out_features=1),\n",
        "    torch.nn.Sigmoid(),    \n",
        ")\n",
        "```\n",
        "\n",
        "-   ANN: O\n",
        "-   MLP: O\n",
        "-   DNN: O? (깊긴한데 이정도면 모양이 깊다기 보다는 넓은 신경망임,\n",
        "    그래서 어떤 연구세어는 이걸 넓은 신경망이라 부르기도 함)\n",
        "-   CNN: X (합성곱레이어가 없으므로)\n",
        "-   RNN: X (순환구조가 없으므로)\n",
        "\n",
        "`# 예시4` – CNN\n",
        "\n",
        "``` python\n",
        "net = torch.nn.Sequential(\n",
        "    # Layer1\n",
        "    torch.nn.Conv2d(1, 64, kernel_size=4, stride=2, padding=1, bias=False),\n",
        "    torch.nn.LeakyReLU(0.2),\n",
        "    # Layer2\n",
        "    torch.nn.Conv2d(64, 128, kernel_size=4, stride=2, padding=1, bias=False),\n",
        "    torch.nn.BatchNorm2d(128),\n",
        "    torch.nn.LeakyReLU(0.2),\n",
        "    # Layer3\n",
        "    torch.nn.Conv2d(128, 256, kernel_size=4, stride=2, padding=1, bias=False),\n",
        "    torch.nn.BatchNorm2d(256),\n",
        "    torch.nn.LeakyReLU(0.2),\n",
        "    # Layer4\n",
        "    torch.nn.Conv2d(256, 512, kernel_size=4, stride=2, padding=1, bias=False),\n",
        "    torch.nn.BatchNorm2d(512),\n",
        "    torch.nn.LeakyReLU(0.2),\n",
        "    # Layer5\n",
        "    torch.nn.Conv2d(512, 1, kernel_size=4, stride=1, padding=0, bias=False),\n",
        "    torch.nn.Sigmoid(),\n",
        "    torch.nn.Flatten()\n",
        ")\n",
        "```\n",
        "\n",
        "-   ANN: O\n",
        "-   MLP: X (합성곱연결이 포함되어있으므로, MLP가 아님, 완전연결만\n",
        "    포함해야 MLP임)  \n",
        "-   DNN: O\n",
        "-   CNN: O (합성곱레이어를 포함하고 있으므로)\n",
        "-   RNN: X (순환구조가 없으므로)\n",
        "\n",
        "`#`\n",
        "\n",
        "`# 예시5` – CNN\n",
        "\n",
        "``` python\n",
        "net = torch.nn.Sequential(\n",
        "    torch.nn.Conv2d(1,16,(5,5)),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.MaxPool2d((2,2)),\n",
        "    torch.nn.Flatten(),\n",
        "    torch.nn.Linear(2304,1),\n",
        "    torch.nn.Sigmoid()\n",
        ")\n",
        "```\n",
        "\n",
        "-   ANN: O\n",
        "-   MLP: X\n",
        "-   DNN: X? (히든레이어가 1장이므로..)\n",
        "-   CNN: O (합성곱레이어를 포함하고 있으므로)\n",
        "-   RNN: X (순환구조가 없으므로)\n",
        "\n",
        "> 근데 대부분의 문서에서는 CNN, RNN은 DNN의 한 종류로 설명하고\n",
        "> 있어서요.. 이런 네트워크에서는 개념충돌이 옵니다.\n",
        "\n",
        "`#`\n",
        "\n",
        "`# 예시6` – RNN\n",
        "\n",
        "``` python\n",
        "class Net(torch.nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.rnn = torch.nn.RNN(4,2)\n",
        "        self.linr = torch.nn.Linear(2,2) \n",
        "    def forward(self,X):\n",
        "        h = self.rnn(X) \n",
        "        netout = self.linr(h)\n",
        "        return netout \n",
        "net = Net()     \n",
        "```\n",
        "\n",
        "-   ANN: O\n",
        "-   MLP: X\n",
        "-   DNN: X? (히든레이어가 1장이므로..)\n",
        "-   CNN: X (합성곱레이어가 없으므로)\n",
        "-   RNN: O\n",
        "\n",
        "> 이것도 비슷한 개념충돌\n",
        "\n",
        "`#`\n",
        "\n",
        "# A2. 학습\n",
        "\n",
        "`-` 모든 인공지능 관련 알고리즘은 아래의 분류로 가능함.\n",
        "\n",
        "|      **특징**       | **지도학습 (Supervised Learning)** | **비지도학습 (Unsupervised Learning)** | **강화학습 (Reinforcement Learning)**  |\n",
        "|:----------:|:------------------:|:-------------------:|:------------------:|\n",
        "|      **정의**       | 입력 데이터와 정답(레이블)을 사용  |           입력 데이터만 사용           |  에이전트가 환경과 상호작용하며 학습   |\n",
        "|      **목표**       |   입력에 대한 정확한 출력을 예측   |    데이터의 숨겨진 구조나 패턴 발견    | 최대 보상을 얻기 위한 최적의 정책 학습 |\n",
        "|      **예시**       |      이미지 분류, 스팸 필터링      |           군집화, 차원 축소            |         게임 플레이, 로봇 제어         |\n",
        "|  **주요 알고리즘**  |   선형 회귀, 로지스틱 회귀, SVM    |        K-평균, PCA, 오토인코더         |      Q-러닝, DQN, 정책 경사 방법       |\n",
        "|      **활용**       |             분류, 예측             |       데이터의 숨겨진 패턴 발견        |     복잡한 의사결정 문제 해결 가능     |\n",
        "| **데이터 요구사항** |       레이블링이 반드시 필요       |         많은 양의 데이터 필요          |     시뮬레이션 또는 실제 환경 필요     |\n",
        "\n",
        "`-` 그런데 사실 불가능함"
      ],
      "id": "5ec01faf-6b91-4bf4-9c41-c3f3f31a302d"
    }
  ],
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3 (ipykernel)",
      "language": "python"
    },
    "language_info": {
      "name": "python",
      "codemirror_mode": {
        "name": "ipython",
        "version": "3"
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.8"
    }
  }
}