{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 13wk-1: 순환신경망 (4) – LSTM 계산과정 재현, LSTM은 왜 강한가?\n", "\n", "최규빈 \n", "2024-05-27\n", "\n", "\n", "\n", "# 1. 강의영상\n", "\n", "\n", "\n", "# 2. Import" ], "id": "19a64556-4d23-44f6-a9f5-5ed8a9ae6df3" }, { "cell_type": "code", "execution_count": 33, "metadata": { "tags": [] }, "outputs": [], "source": [ "import torch\n", "import pandas as pd\n", "import matplotlib.pyplot as plt" ], "id": "46662b63-dee5-453b-8539-9d1d0e7929c2" }, { "cell_type": "code", "execution_count": 34, "metadata": { "tags": [] }, "outputs": [], "source": [ "soft = torch.nn.Softmax(dim=1)\n", "sig = torch.nn.Sigmoid()\n", "tanh = torch.nn.Tanh()" ], "id": "61756f58-25dc-4f07-8f93-0ed355f58c02" }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3. Data – `abcabC`" ], "id": "c0810f3e-53e3-428c-be63-ab1bcb2120f2" }, { "cell_type": "code", "execution_count": 35, "metadata": { "tags": [] }, "outputs": [], "source": [ "txt = list('abcabC')*50\n", "txt[:10]" ], "id": "86a18822-5d02-4b72-915d-f434ba4a5a69" }, { "cell_type": "code", "execution_count": 36, "metadata": { "tags": [] }, "outputs": [], "source": [ "df_train = pd.DataFrame({'x':txt[:-1], 'y':txt[1:]})\n", "df_train[:5]" ], "id": "94c9e9bc-e4e4-4e04-b717-a3080d09f2e4" }, { "cell_type": "code", "execution_count": 37, "metadata": { "tags": [] }, "outputs": [], "source": [ "x = torch.tensor(df_train.x.map({'a':0,'b':1,'c':2,'C':3}))\n", "y = torch.tensor(df_train.y.map({'a':0,'b':1,'c':2,'C':3}))\n", "X = torch.nn.functional.one_hot(x).float()\n", "y = torch.nn.functional.one_hot(y).float()" ], "id": "910fd73a-46c9-4545-b2bc-db0e1720d441" }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4. LSTM 계산과정 재현\n", "\n", "## A. `torch.nn.LSTMCell`" ], "id": "af9068d2-f393-431a-bf70-d8c4ba8265e2" }, { "cell_type": "code", "execution_count": 38, "metadata": { "tags": [] }, "outputs": [], "source": [ "torch.manual_seed(43052)\n", "lstmcell = torch.nn.LSTMCell(4,2) # 숙성담당\n", "cook = torch.nn.Linear(2,4)\n", "loss_fn = torch.nn.CrossEntropyLoss()\n", "optimizr = torch.optim.Adam(list(lstmcell.parameters())+list(cook.parameters()),lr=0.1) \n", "#---#\n", "L = len(X)\n", "for epoc in range(1):\n", " # 1~2 \n", " loss = 0 \n", " ht = torch.zeros(2)\n", " ct = torch.zeros(2)\n", " for t in range(L):\n", " Xt,yt = X[t],y[t]\n", " ht,ct = lstmcell(Xt,(ht,ct))\n", " netout_t = cook(ht)\n", " loss = loss + loss_fn(netout_t,yt) \n", " loss = loss/L\n", " # 3\n", " loss.backward()\n", " # 4 \n", " optimizr.step()\n", " optimizr.zero_grad()" ], "id": "5ab20de4-0757-45a2-82b0-71ca68896365" }, { "cell_type": "code", "execution_count": 39, "metadata": { "tags": [] }, "outputs": [], "source": [ "ht,ct" ], "id": "c92202a8-2847-48e5-86f8-771b497bc3cc" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## B. 직접구현\n", "\n", "### ***t=0 $\\to$ t=1***\n", "\n", "`-` `lstmcell`을 이용하여 $t=0 \\to t=1$을 구현해보자. (결과비교용)" ], "id": "1908978d-d771-4ace-bf51-968deae6801e" }, { "cell_type": "code", "execution_count": 40, "metadata": { "tags": [] }, "outputs": [], "source": [ "torch.manual_seed(43052)\n", "lstmcell = torch.nn.LSTMCell(4,2) # 숙성담당\n", "cook = torch.nn.Linear(2,4)\n", "loss_fn = torch.nn.CrossEntropyLoss()\n", "optimizr = torch.optim.Adam(list(lstmcell.parameters())+list(cook.parameters()),lr=0.1) \n", "#---#\n", "L = len(X)\n", "for epoc in range(1):\n", " # 1~2 \n", " loss = 0 \n", " ht = torch.zeros(2)\n", " ct = torch.zeros(2)\n", " for t in range(1):\n", " Xt,yt = X[t],y[t]\n", " ht,ct = lstmcell(Xt,(ht,ct))\n", " netout_t = cook(ht)\n", " loss = loss + loss_fn(netout_t,yt) \n", " loss = loss/L\n", " # 3\n", " loss.backward()\n", " # 4 \n", " optimizr.step()\n", " optimizr.zero_grad()" ], "id": "206cb5a9-00c4-409b-9d1b-6e3bf3c54b97" }, { "cell_type": "code", "execution_count": 41, "metadata": { "tags": [] }, "outputs": [], "source": [ "ht,ct" ], "id": "a714f717-0571-4c61-8ba4-9e737891a8e9" }, { "cell_type": "markdown", "metadata": {}, "source": [ "- 이런결과를 어떻게 만드는걸까?\n", "- \n", "\n", "`-` 직접계산\n", "(${\\boldsymbol i}_t, {\\boldsymbol f}_t, {\\boldsymbol g}_t, {\\boldsymbol o}_t$)" ], "id": "f70a55a1-5b26-4f47-a04d-e1bfee389133" }, { "cell_type": "code", "execution_count": 42, "metadata": { "tags": [] }, "outputs": [], "source": [ "torch.manual_seed(43052)\n", "lstmcell = torch.nn.LSTMCell(4,2) # 숙성담당" ], "id": "32fc61c5-071d-4f1e-91e2-4a1d66661040" }, { "cell_type": "code", "execution_count": 43, "metadata": { "tags": [] }, "outputs": [], "source": [ "Wih = lstmcell.weight_ih.T\n", "Wih02 = Wih[:,0:2]\n", "Wih24 = Wih[:,2:4]\n", "Wih46 = Wih[:,4:6]\n", "Wih68 = Wih[:,6:8]\n", "#--#\n", "Whh = lstmcell.weight_hh.T\n", "Whh02 = Whh[:,0:2]\n", "Whh24 = Whh[:,2:4]\n", "Whh46 = Whh[:,4:6]\n", "Whh68 = Whh[:,6:8]\n", "#--#\n", "bih02 = lstmcell.bias_ih[0:2]\n", "bih24 = lstmcell.bias_ih[2:4]\n", "bih46 = lstmcell.bias_ih[4:6]\n", "bih68 = lstmcell.bias_ih[6:8]\n", "#--#\n", "bhh02 = lstmcell.bias_hh[0:2]\n", "bhh24 = lstmcell.bias_hh[2:4]\n", "bhh46 = lstmcell.bias_hh[4:6]\n", "bhh68 = lstmcell.bias_hh[6:8]" ], "id": "bd84a2d6-23a7-4e88-be20-b4649b4f2a65" }, { "cell_type": "code", "execution_count": 44, "metadata": { "tags": [] }, "outputs": [], "source": [ "ht = torch.zeros(2)\n", "ct = torch.zeros(2)\n", "#--#\n", "it = sig(Xt@Wih02 + bih02 + ht@Whh02 + bhh02)\n", "ft = sig(Xt@Wih24 + bih24 + ht@Whh24 + bhh24)\n", "gt = tanh(Xt@Wih46 + bih46 + ht@Whh46 + bhh46)\n", "ot = sig(Xt@Wih68 + bih68 + ht@Whh68 + bhh68)" ], "id": "081f8f9b-cc35-4cb9-9cfe-58117ab96309" }, { "cell_type": "code", "execution_count": 45, "metadata": { "tags": [] }, "outputs": [], "source": [ "it,ft,gt,ot" ], "id": "6a6f42e8-47f2-486c-81c0-cddf869a51a5" }, { "cell_type": "markdown", "metadata": {}, "source": [ "그런데 아래와 같이 계산할수도 있음." ], "id": "42796800-ced8-496c-a500-9682064ee544" }, { "cell_type": "code", "execution_count": 46, "metadata": { "tags": [] }, "outputs": [], "source": [ "ifgo = Xt @ lstmcell.weight_ih.T + lstmcell.bias_ih +\\\n", "ht @ lstmcell.weight_hh.T + lstmcell.bias_hh" ], "id": "b00426c0-d330-4e1a-b41a-a097b58904fb" }, { "cell_type": "code", "execution_count": 47, "metadata": { "tags": [] }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "it = tensor([0.3081, 0.5949], grad_fn=)\n", "ft = tensor([0.5065, 0.4264], grad_fn=)\n", "gt = tensor([0.5629, 0.4518], grad_fn=)\n", "ot = tensor([0.6216, 0.4072], grad_fn=)" ] } ], "source": [ "print(f\"it = {sig(ifgo[0:2])}\")\n", "print(f\"ft = {sig(ifgo[2:4])}\")\n", "print(f\"gt = {tanh(ifgo[4:6])}\")\n", "print(f\"ot = {sig(ifgo[6:8])}\")" ], "id": "986e497c-ce5d-4be0-8a69-b2b85b40908e" }, { "cell_type": "markdown", "metadata": {}, "source": [ "`-` 직접계산 (${\\boldsymbol c}_t, {\\boldsymbol h}_t$)" ], "id": "93f47018-7f49-469a-9a9e-948951402e1e" }, { "cell_type": "code", "execution_count": 48, "metadata": { "tags": [] }, "outputs": [], "source": [ "ct = it*gt + ft*ct \n", "ht = ot*tanh(ct)" ], "id": "af028ce3-a056-4083-b6bd-e33fe00f98f0" }, { "cell_type": "code", "execution_count": 49, "metadata": { "tags": [] }, "outputs": [], "source": [ "ht,ct" ], "id": "735c92bf-d197-4a63-946f-fcfe54c138df" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `#` ***t=0 $\\to$ t=L***\n", "\n", "`-` `lstmcell`을 이용하여 구현해보자. (결과비교용)" ], "id": "57780d85-bd69-47f1-9fb5-cd12c73e21f8" }, { "cell_type": "code", "execution_count": 52, "metadata": { "tags": [] }, "outputs": [], "source": [ "torch.manual_seed(43052)\n", "lstmcell = torch.nn.LSTMCell(4,2) # 숙성담당\n", "cook = torch.nn.Linear(2,4)\n", "loss_fn = torch.nn.CrossEntropyLoss()\n", "optimizr = torch.optim.Adam(list(lstmcell.parameters())+list(cook.parameters()),lr=0.1) \n", "#---#\n", "L = len(X)\n", "for epoc in range(1):\n", " # 1~2 \n", " loss = 0 \n", " ht = torch.zeros(2)\n", " ct = torch.zeros(2)\n", " for t in range(L):\n", " Xt,yt = X[t],y[t]\n", " ht,ct = lstmcell(Xt,(ht,ct))\n", " netout_t = cook(ht)\n", " loss = loss + loss_fn(netout_t,yt) \n", " loss = loss/L\n", " # 3\n", " loss.backward()\n", " # 4 \n", " optimizr.step()\n", " optimizr.zero_grad()" ], "id": "4219bb6f-0275-41f4-81d9-c97b35803feb" }, { "cell_type": "code", "execution_count": 53, "metadata": { "tags": [] }, "outputs": [], "source": [ "ht,ct" ], "id": "0aa701ff-b7e6-4036-b8c6-ee4cd69b80a9" }, { "cell_type": "markdown", "metadata": {}, "source": [ "`-` 직접구현" ], "id": "f413bac9-f4b3-4f29-9685-d4f721550b90" }, { "cell_type": "code", "execution_count": 54, "metadata": { "tags": [] }, "outputs": [], "source": [ "torch.manual_seed(43052)\n", "lstmcell = torch.nn.LSTMCell(4,2) # 숙성담당\n", "cook = torch.nn.Linear(2,4)\n", "loss_fn = torch.nn.CrossEntropyLoss()\n", "optimizr = torch.optim.Adam(list(lstmcell.parameters())+list(cook.parameters()),lr=0.1) \n", "#---#\n", "L = len(X)\n", "for epoc in range(1):\n", " # 1~2 \n", " loss = 0 \n", " ht = torch.zeros(2)\n", " ct = torch.zeros(2)\n", " for t in range(L):\n", " Xt,yt = X[t],y[t]\n", " ifgo = Xt @ lstmcell.weight_ih.T + lstmcell.bias_ih +\\\n", " ht @ lstmcell.weight_hh.T + lstmcell.bias_hh\n", " it,ft,gt,ot = sig(ifgo[0:2]),sig(ifgo[2:4]),tanh(ifgo[4:6]),sig(ifgo[6:8])\n", " ct = it*gt + ft*ct\n", " ht = ot*tanh(ct)\n", " netout_t = cook(ht)\n", " loss = loss + loss_fn(netout_t,yt) \n", " loss = loss/L\n", " # 3\n", " loss.backward()\n", " # 4 \n", " optimizr.step()\n", " optimizr.zero_grad()" ], "id": "46c85c22-20c5-4648-b362-7b706fc69ff2" }, { "cell_type": "code", "execution_count": 55, "metadata": { "tags": [] }, "outputs": [], "source": [ "ht,ct" ], "id": "aa871391-e37a-4f3d-aa5d-7dad101f6f1c" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## C. `torch.nn.LSTM`" ], "id": "a69eab5e-ace9-4db1-be10-292dbd3ace95" }, { "cell_type": "code", "execution_count": 80, "metadata": { "tags": [] }, "outputs": [], "source": [ "torch.manual_seed(43052)\n", "lstmcell = torch.nn.LSTMCell(4,2) # 숙성담당\n", "cook = torch.nn.Linear(2,4)\n", "lstm = torch.nn.LSTM(4,2) # <-- 이거로 학습\n", "lstm.weight_ih_l0.data = lstmcell.weight_ih.data\n", "lstm.weight_hh_l0.data = lstmcell.weight_hh.data\n", "lstm.bias_ih_l0.data = lstmcell.bias_ih.data\n", "lstm.bias_hh_l0.data = lstmcell.bias_hh.data\n", "loss_fn = torch.nn.CrossEntropyLoss()\n", "optimizr = torch.optim.Adam(list(lstm.parameters())+list(cook.parameters()),lr=0.1) \n", "#---#\n", "L = len(X)\n", "Water = torch.zeros(1,2)\n", "for epoc in range(1):\n", " # 1 \n", " h,(hL,cL) = lstm(X,(Water,Water))\n", " netout = cook(h)\n", " # 2 \n", " loss = loss_fn(netout,y)\n", " # 3\n", " loss.backward()\n", " # 4 \n", " optimizr.step()\n", " optimizr.zero_grad()" ], "id": "3bc756ae-a856-46e0-989f-89bd3ab24be4" }, { "cell_type": "code", "execution_count": 76, "metadata": { "tags": [] }, "outputs": [], "source": [ "hL,cL" ], "id": "7f78e4d4-91ec-44a9-a72a-e2f203601eb2" }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 5. LSTM은 왜 강한가?\n", "\n", "## A. 적합 및 시각화\n", "\n", "`-` 적합" ], "id": "6d6c72a3-390a-4026-9531-54470eff7587" }, { "cell_type": "code", "execution_count": 27, "metadata": { "tags": [] }, "outputs": [], "source": [ "torch.manual_seed(43052)\n", "lstm = torch.nn.LSTM(4,2)\n", "cook = torch.nn.Linear(2,4)\n", "loss_fn = torch.nn.CrossEntropyLoss() \n", "optimizr = torch.optim.Adam(list(lstm.parameters())+ list(cook.parameters()),lr=0.1)\n", "#--#\n", "Water = torch.zeros(1,2) \n", "for epoc in range(200): \n", " ## step1 \n", " h, (hL,cL) = lstm(X,(Water,Water))\n", " netout = cook(h)\n", " ## step2\n", " loss = loss_fn(netout,y) \n", " ## step3\n", " loss.backward()\n", " ## step4 \n", " optimizr.step()\n", " optimizr.zero_grad() " ], "id": "bced9cca-dcf7-4471-9bcf-99e30264db34" }, { "cell_type": "markdown", "metadata": {}, "source": [ "`-` 시각화" ], "id": "47051f25-0c38-4a8a-9224-820a75b500f6" }, { "cell_type": "code", "execution_count": 28, "metadata": { "tags": [] }, "outputs": [], "source": [ "L = len(X)\n", "i = input_gate = torch.zeros(L,2)\n", "f = forget_gate = torch.zeros(L,2)\n", "g = gihhen_state = torch.zeros(L,2)\n", "o = output_gate = torch.zeros(L,2)\n", "c = cell_state = torch.zeros(L,2)\n", "h = hidden_state = torch.zeros(L,2) \n", "#--#\n", "water = torch.zeros(2)\n", "ifgo = X[0] @ lstm.weight_ih_l0.T + lstm.bias_ih_l0 +\\\n", " water @ lstm.weight_hh_l0.T + lstm.bias_hh_l0 \n", "# 위에서 water는 h0에 대응\n", "i[0] = sig(ifgo[0:2])\n", "f[0] = sig(ifgo[2:4])\n", "g[0] = tanh(ifgo[4:6])\n", "o[0] = sig(ifgo[6:8])\n", "c[0] = f[0] * water + i[0] * g[0] # water는 c0에 대응 \n", "h[0] = o[0] * tanh(c[0])\n", "#--#\n", "for t in range(1,L): \n", " ## 1: calculate ifgo \n", " ifgo = X[t] @ lstm.weight_ih_l0.T + lstm.bias_ih_l0 +\\\n", " h[t-1] @ lstm.weight_hh_l0.T + lstm.bias_hh_l0 \n", " ## 2: decompose ifgo \n", " i[t] = sig(ifgo[0:2])\n", " f[t] = sig(ifgo[2:4])\n", " g[t] = tanh(ifgo[4:6])\n", " o[t] = sig(ifgo[6:8])\n", " ## 3: calculate ht,ct \n", " c[t] = f[t] * c[t-1] + i[t] * g[t]\n", " h[t] = o[t] * tanh(c[t])" ], "id": "92933a9f-34e7-48af-93a2-214369af9a89" }, { "cell_type": "code", "execution_count": 29, "metadata": { "tags": [] }, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAGVCAYAAABElA6NAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90\nbGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9h\nAAAPYQGoP6dpAAA3wUlEQVR4nO3de1hU5b4H8O+AMHhhBpW4JQqlGwW8IKiAabpV1NKsXYK79mQd\no8edl4zcFV3R52zJOhWZaek2yTTl9BBpJ8WgEi3BDAEvucm2GqMxmiYzQDkorPOHMTUOl5k1s1gL\n5vt5nvXkvPOuNb81IT9/73rXu1SCIAggIiJSIA+5AyAiImoNkxQRESkWkxQRESkWkxQRESkWkxQR\nESkWkxQRESkWkxQRESkWkxQRESkWkxQRESkWkxQRESkWkxQRURe0d+9ezJw5EyEhIVCpVPjoo4/a\n3aeoqAixsbHw8fHBTTfdhLfeesumT25uLiIjI6FWqxEZGYm8vDwJov8dkxQRURdUX1+P4cOHY/Xq\n1Xb1P3XqFG677TaMGzcOZWVlePrpp7F48WLk5uZa+hQXFyMlJQU6nQ4VFRXQ6XRITk7GgQMHpDoN\nqLjALBFR16ZSqZCXl4c777yz1T5PPvkkduzYgePHj1va5s+fj4qKChQXFwMAUlJSYDKZsGvXLkuf\nadOmoXfv3ti6dasksXeT5KhERAQAuHz5MhoaGpw+jiAIUKlUVm1qtRpqtdrpYwPXqqSkpCSrtqlT\np2LDhg24cuUKvLy8UFxcjMcee8ymT1ZWlktiaAmTFBGRRC5fvozw7t1hcMGxevXqhbq6Oqu2F154\nARkZGS44OmAwGBAYGGjVFhgYiKtXr+LChQsIDg5utY/B4IozbBmTFBGRRBoaGmAAoFepoHHiOCYA\noXV10Ov10Gh+P5Krqqhm11dqzVeD/tjeUp/r21yJSYqISGIaDw9onPlFLghAYyM0Go1VknKloKAg\nm4ro/Pnz6NatG/r27dtmn+urK1fi7D4iIkJCQgIKCgqs2j799FPExcXBy8urzT6JiYmSxcVKiohI\nah4egAsqKUfU1dXh+++/t7w+deoUysvL0adPH/Tv3x/p6ek4e/YsNm3aBODaTL7Vq1cjLS0Nqamp\nKC4uxoYNG6xm7T366KMYP348Vq5ciVmzZmH79u0oLCzEl19+Kf7c2iMQEZEkjEajAEAwqtWC4OMj\nejOq1deOYzTa/dlffPGFAMBmmzt3riAIgjB37lzh1ltvtdpnz549QkxMjODt7S2EhYUJa9eutTnu\nBx98IERERAheXl7C4MGDhdzcXGe+onbxPikiIomYTCZotVoY1WqnrkmZBAFasxlGo1Gya1JKxeE+\nIiKpuWK4z00xSRERSY1JSjTO7iMiIsViJUVEJDVWUqIxSRERSY1JSjQO93Uxjz/+OGbOnCl3GDaU\nGhcRKRuTVBdTXl6O4cOHyx2GDSXGNX78eKhUKqhUKnh5eSEiIgLvv/8+Y+oEcSktnnZ5eDi/uSn3\nPfMuqqKiQnHJAFBeXIIgoLy8HJmZmaiursZ3332HW265BXPnzsWpU6cYk4LjUlo8dlGpnEtQEi7g\nqnSdJkkdO3YM48ePR/fu3TFixAh89dVXUKlUqKioYEy/0ev1uHjxIjw8PDBlyhT06NEDERERkj41\ns7PGdeLECdTW1uKWW25BUFAQwsPD8fTTT+Pq1as4fPgwY1JwXEqLxy6spETrFGd+7NgxxMfHWx5r\n/Pzzz+Oee+6Bl5cXhgwZwph+U15eDgB44403kJ6ejoqKCvTv3x9PPfWULPEoOa7S0lKoVCoMGzbM\n0nbmzBkAkHRF584WE6C8uJQWD0lM0kWXXGTChAlCcnKyVdtdd90lDB8+XBAEQfj444+FP/3pT8LA\ngQOF9evXKyKmO++8U/Dz8xPuvvvuDolHEARh+fLlQu/evYVz585Z2lavXi1ERUUJgiDP92RPXHJ8\nV0uXLhVuuukmy+t///vfQlxcnDBixAihsbGxw+JwJKaPP/5YWLhwoeLiUmI8GzduFGJjY4WoqChh\nyJAhwpYtWzo8TkH4w9p9AQGCEBQkejMGBDi8dl9XofhK6vTp09izZw+ef/55q3a1Wo3hw4fj6tWr\nSEtLw+eff45Dhw5h5cqV+Pnnn2WNCQAWL15sWV24o5SXl2PWrFkICAiwtJ08eRIDBw6U5XuyJy5A\nnu+qtLQUp0+fRq9eveDj44MRI0YgKioK+fn58JBpaKW9mA4fPoyYmBjFxaW0eP71r39h48aNyM/P\nx9GjR1FUVIRGB1cQdzkO94mm+DOvqKiAt7c3oqKirNqPHz+OESNG4Ouvv0ZUVBRuvPFG+Pr64rbb\nbsPu3btljQkAJk6cCF9fX0njuF55eTkSEhKs2srKymT7nuyJC5DnuyorK8M//vEPlJeX4+TJk/jl\nl1+QnZ1tGS567733MGbMGAwdOhR33HEHGhoaZI/p8OHDOH78OGJjYxEVFYXvvvtO8pjsieuHH37A\njBkzLMni7NmzssVTV1eH//7v/0ZOTg78/f0BADfccAN0Op2kMZF0FJ+kPD09cfXqVVy+fNnSVlRU\nZJkt9uOPP+LGG2+0vNevXz/J/5K0F5McamtrcerUKZt/aZeXl2PEiBGyfE/2xCWHkydPoqamBlOm\nTMHAgQMREhJi8/jr2267DQcOHMCRI0fg7++Pffv2yR7T4cOH0b9/f5SWlmLRokV49dVXJY3Jnrga\nGhpw++2344knnkB5eTn27dsn6XWh9uL53//9X8THxyMoKEiyGERhJSWa4s88NjYWXl5e+Mc//oGT\nJ0/i//7v/zBv3jwAwIgRIyC0cCf29X+5OzomOZSXl8PDwwNDhw61tP3www+4dOmSbN+TPXHJofnC\ne2xsbIvvC4KAdevWYdSoURg+fDjy8vLg4+Mja0xmsxkNDQ1YsGABAGDYsGG4cOGCpDHZE1deXh7i\n4+Mxfvx4AECfPn3QrZt0C9m0F8/Ro0etJlQoBpOUaIo/8+DgYLzzzjvYvn07hg0bhnfeeQcPPvgg\nBg4ciD59+uDGG2+0qgjOnDmD4OBgWWOSQ0VFBQYPHmz1y7SsrAx+fn4ICwuT5XuyJy45HDp0CAMH\nDoSfn1+L72dnZ+P777/H3r17UVFRAY1Gg8jISFljOnbsGCIjIy3XgMrKyjrkl3F7cR05cgSjRo2S\nPA574+nZsyeampo6LB7qADJP3HBYY2OjMH78eCE9PV0QBEG4cuWKMHDgQOHMmTOCyWQSBg4cKFy4\ncEHWmJp98cUXHTpjrS1K+J7aoqTv6vHHHxf+9a9/CYIgCFlZWUK/fv1kjujabLXIyEihoaFBOHfu\nnBAXFyf89NNPcoclrFq1yjLj8OrVq8LFixdljeerr74ShgwZYvnZrqmpETZv3ixbPJbZff37C0JY\nmOjN2L+/287uU/wCs3v37sVPP/2EmJgYXLhwAS+//DJOnz6NvLw8AEC3bt3wyiuvYOLEiWhqasIT\nTzyBvn37yhoTAEydOhWHDh1CfX09+vXrh7y8vA79F+f15Pie7KW070qn02HWrFnYtGkTbr31Vquh\nSrkcOXIEd999N0aNGoWmpia89tprlokBcnrggQeQkpKC6OhoeHl54e2338bo0aNliycxMRFpaWmY\nOHEiBEGAh4cHHn/8cdnisXDzITtnKP7x8R988AGeeuopnD17FoGBgZg8eTJWrFgh6017SoyJiJTH\n8vj4sDBonEhSpqYmaE+fdsvHxys+SRERdVaWJHXTTc4nqZMn3TJJKX64j4io02teYFYsN64lOEhK\nRESKxUqKiEhqzk6ccONKikmKiEhqTFKiMUkREUmNSUq0TntNymw2IyMjA2azWe5QLJQYE6DMuBiT\nfRiT/ZQaFzmn005Bt0ztVNCUTCXGBCgzLsZkH8ZkPyXGZYkpKgoaT0/xx2lshPbYMUWdW0fhcB8R\nkdQ43Cdapx3uIyKirq/DK6mmpib8+OOP8PX1depRESaTyeq/SqDEmABlxsWY7MOY7OequARBQG1t\nLUJCQlz35GFWUqJ1+DWpM2fOIDQ0tCM/kojIYXq9Hv369XPqGJZrUjExzl+TKivjNamO0PyYcG9v\nPVQq5XzZ5/9TK3cINoReHftIdXup6pTxXdXjF4T4/gkA8KNwFj3RU+aIrnPDDXJHYKu6Wu4IWibh\ngxIdZTKZENq/v+V3Fcmrw38ymof4VCqNopKURiP9U2odpdgk5aGM78oTnsBvX5FG0CgvSXXAk48d\nptR/hSsoSTVz6ZOrOdwnmvJ+MoiIuhpnF5h146cNc3YfEREpFpMUEZHUmof7nNlEWLNmDcLDw+Hj\n44PY2Fjs27ev1b4PPPAAVCqVzRYVFWXpk52d3WKfy5cvi4rPHkxSRERSkyFJ5eTkYMmSJXjmmWdQ\nVlaGcePGYfr06aiqqmqx/+uvv47q6mrLptfr0adPH8yePduqn0ajsepXXV0NHx8fUV+LPZikiIi6\noFdffRXz5s3DQw89hCFDhiArKwuhoaFYu3Zti/21Wi2CgoIs2zfffINLly7hwQcftOqnUqms+gUF\nBUl6HkxSRERSc1ElZTKZrLbWFtNtaGhAaWkpkpKSrNqTkpKwf/9+u0LesGEDJk+ejAEDBli119XV\nYcCAAejXrx9mzJiBsrIyEV+I/ZikiIik5qIkFRoaCq1Wa9kyMzNb/LgLFy6gsbERgYGBVu2BgYEw\nGAzthltdXY1du3bhoYcesmofPHgwsrOzsWPHDmzduhU+Pj4YO3YsTpw4IfKLaR+noBMRSc3Z+6R+\n21ev11utOKFWq9vc7fp7vQRBsOv+r+zsbPj5+eHOO++0ao+Pj0d8fLzl9dixYzFy5Ei88cYbWLVq\nVbvHFYNJioiok9BoNHYti+Tv7w9PT0+bqun8+fM21dX1BEHAO++8A51OB29v7zb7enh4YNSoUZJW\nUhzuIyKSWgfP7vP29kZsbCwKCgqs2gsKCpCYmNjmvkVFRfj+++8xb968dj9HEASUl5cjODjYofgc\nwUqKiEhqLhruc0RaWhp0Oh3i4uKQkJCAdevWoaqqCvPnzwcApKen4+zZs9i0aZPVfhs2bMCYMWMQ\nHR1tc8xly5YhPj4egwYNgslkwqpVq1BeXo4333xT3HnZgUmKiKgLSklJwcWLF7F8+XJUV1cjOjoa\nO3futMzWq66utrlnymg0Ijc3F6+//nqLx6ypqcHDDz8Mg8EArVaLmJgY7N27F6NHj5bsPEQ9qmPN\nmjV4+eWXUV1djaioKGRlZWHcuHF27du8dL1abVTUArO/nlfGyt5/pNgFZhWzCno9evleG2aoE2qV\nt8CshDc4ilZXJ3cELVPQArMmkwlaPz+XPBbD8qiOyZOhceIcTVevQltY6JaP6nC4hnT0LmYiIrfX\nvMCs2E2JK+p3EIeTlKN3MZvNZpsb0IiIiOzhUJIScxdzZmam1c1nfCovEbkdmRaY7QocOnMxdzGn\np6fDaDRaNr1eLz5aIqLOiElKNFFX8hy5i1mtVrd7VzQREVFLHEpSztzFTETktmS4T6qrcOjMnbmL\nmYjIbXG4TzSHh/vau4uZiIjIVRxOUu3dxUxERNfhcJ9ooiZOPPLII3jkkUdcHQsRUdfEJCWactYi\nISLqqpikRHPfMyciIsVjJUVEJDVWUqIxSRERSa15gVln9ndT7pueiYhI8VhJERFJjcN9ojFJERFJ\njUlKNPc9cyIiUjxWUkREUmMlJRqTFBGR1JikRJMtSZnNcn1yy1QaX7lDsNHKcyRll5ysjO+q0ccD\n2C13FG345Re5I7Dl4yN3BC1raJA7AlIoVlJERFJjJSUakxQRkdSYpERz3zMnIiLFYyVFRCQ1VlKi\nMUkREUmNSUo0JikiIqlxgVnR3Dc9ExGR4rGSIiKSGof7RGOSIiKSGpOUaO575kREpHispIiIpMZK\nSjQmKSIiqTFJiea+Z05ERIrHSoqISGqspERjkiIikhqTlGgOn/nevXsxc+ZMhISEQKVS4aOPPpIg\nLCIictaaNWsQHh4OHx8fxMbGYt++fa323bNnD1Qqlc3273//26pfbm4uIiMjoVarERkZiby8PEnP\nweEkVV9fj+HDh2P16tVSxENE1PU0V1LObA7KycnBkiVL8Mwzz6CsrAzjxo3D9OnTUVVV1eZ+lZWV\nqK6utmyDBg2yvFdcXIyUlBTodDpUVFRAp9MhOTkZBw4ccDg+ezk83Dd9+nRMnz7d7v5msxnmPzyG\n12QyOfqRRESdmwzDfa+++irmzZuHhx56CACQlZWF3bt3Y+3atcjMzGx1v4CAAPj5+bX4XlZWFqZM\nmYL09HQAQHp6OoqKipCVlYWtW7c6HKM9JB/ozMzMhFartWyhoaFSfyQRUZdkMpmstj8WAH/U0NCA\n0tJSJCUlWbUnJSVh//79bX5GTEwMgoODMWnSJHzxxRdW7xUXF9scc+rUqe0e0xmSJ6n09HQYjUbL\nptfrpf5IIiJlaV4FXez22yrooaGhVv/ob60iunDhAhobGxEYGGjVHhgYCIPB0OI+wcHBWLduHXJz\nc/Hhhx8iIiICkyZNwt69ey19DAaDQ8d0Bcln96nVaqjVaqk/hohIuVw03KfX66HRaCzN7f1uVV33\niA9BEGzamkVERCAiIsLyOiEhAXq9Hv/zP/+D8ePHizqmK7jvvEYioo7iookTGo3GamstSfn7+8PT\n09Omwjl//rxNJdSW+Ph4nDhxwvI6KCjI6WM6ikmKiKiL8fb2RmxsLAoKCqzaCwoKkJiYaPdxysrK\nEBwcbHmdkJBgc8xPP/3UoWM6yuHhvrq6Onz//feW16dOnUJ5eTn69OmD/v37uzQ4IqIuQYbZfWlp\nadDpdIiLi0NCQgLWrVuHqqoqzJ8/H8C1+QJnz57Fpk2bAFybuRcWFoaoqCg0NDRg8+bNyM3NRW5u\nruWYjz76KMaPH4+VK1di1qxZ2L59OwoLC/Hll1+KP7d2OJykvvnmG0ycONHyOi0tDQAwd+5cZGdn\nuywwIqIuQ4YklZKSgosXL2L58uWorq5GdHQ0du7ciQEDBgAAqqurre6ZamhowNKlS3H27Fl0794d\nUVFR+OSTT3DbbbdZ+iQmJmLbtm149tln8dxzz+Hmm29GTk4OxowZI/7c2qESBEGQ7OgtMJlM0Gq1\nAIwANO11d2sSTphxSnKy3BFc0+hTj6929wIA1Am16ImeMkd0naYmuSOw5eMjdwQta2iQOwILk8kE\nrZ8fjEaj1SQF0cfSamHMyIDGie/edPkytBkZLomps+HafUREUuPafaIxSRERSY1JSjT3PXMiIlI8\nVlJERFJjJSUakxQRkdSYpERz3zMnIiLFYyVFRCS15gVmndnfTTFJERFJjcN9ojFJERFJjUlKNPc9\ncyIiUjxWUr8RoLwx31EzOnTFKrsd7HGr3CEAAOrRiF7NL3r5Ar/IGU0LFLgsktBwRe4QWqSCMn/W\nXYaVlGhMUkREUmOSEs19z5yIiBSPlRQRkdRYSYnGJEVEJDUmKdHc98yJiEjxWEkREUmNlZRoTFJE\nRFJjkhLNfc+ciIgUj5UUEZHUuMCsaExSRERS43CfaO575kREpHispIiIpMZKSjQmKSIiqTFJicYk\nRUQkNSYp0dz3zImISPEcSlKZmZkYNWoUfH19ERAQgDvvvBOVlZVSxUZE1DU0V1LObG7KoTMvKirC\nggULUFJSgoKCAly9ehVJSUmor6+XKj4ios6PSUo0h65J5efnW73euHEjAgICUFpaivHjx7s0MCIi\nIqcmThiNRgBAnz59Wu1jNpthNpstr00mkzMfSUTU+XDihGiiz1wQBKSlpeGWW25BdHR0q/0yMzOh\n1WotW2hoqNiPJCLqnDjcJ5roM1+4cCEOHz6MrVu3ttkvPT0dRqPRsun1erEfSUREbkbUcN+iRYuw\nY8cO7N27F/369Wuzr1qthlqtFhUcEVGXwOE+0RxKUoIgYNGiRcjLy8OePXsQHh4uVVxERF0HV0EX\nzaEktWDBArz//vvYvn07fH19YTAYAABarRbdu3eXJEAiInJfDqX2tWvXwmg0YsKECQgODrZsOTk5\nUsVHRNT5yTRxYs2aNQgPD4ePjw9iY2Oxb9++Vvt++OGHmDJlCm644QZoNBokJCRg9+7dVn2ys7Oh\nUqlstsuXL4uKzx4OD/cREZGDZLgmlZOTgyVLlmDNmjUYO3Ys3n77bUyfPh3ffvst+vfvb9N/7969\nmDJlClasWAE/Pz9s3LgRM2fOxIEDBxATE2Ppp9FobFYa8vHxcfyc7MQFZomIuqBXX30V8+bNw0MP\nPQQAyMrKwu7du7F27VpkZmba9M/KyrJ6vWLFCmzfvh0ff/yxVZJSqVQICgqSNPY/ct8pI0REHcVF\nw30mk8lq++NCCX/U0NCA0tJSJCUlWbUnJSVh//79doXc1NSE2tpam8Ua6urqMGDAAPTr1w8zZsxA\nWVmZiC/EfkxSRERSc1GSCg0NtVocoaWKCAAuXLiAxsZGBAYGWrUHBgZaJry155VXXkF9fT2Sk5Mt\nbYMHD0Z2djZ27NiBrVu3wsfHB2PHjsWJEydEfjHt43AfEZHUXHRNSq/XQ6PRWJrbuwdVdd3UdUEQ\nbNpasnXrVmRkZGD79u0ICAiwtMfHxyM+Pt7yeuzYsRg5ciTeeOMNrFq1yq5TcRSTFBFRJ6HRaKyS\nVGv8/f3h6elpUzWdP3/eprq6Xk5ODubNm4cPPvgAkydPbrOvh4cHRo0aJWklxeE+IiKpdfAUdG9v\nb8TGxqKgoMCqvaCgAImJia3ut3XrVjzwwAN4//33cfvtt7f7OYIgoLy8HMHBwQ7F5whWUkREUpNh\nCnpaWhp0Oh3i4uKQkJCAdevWoaqqCvPnzwdwbV3Vs2fPYtOmTQCuJaj7778fr7/+OuLj4y1VWPfu\n3aHVagEAy5YtQ3x8PAYNGgSTyYRVq1ahvLwcb775pvhzaweTFBFRF5SSkoKLFy9i+fLlqK6uRnR0\nNHbu3IkBAwYAAKqrq1FVVWXp//bbb+Pq1atYsGABFixYYGmfO3cusrOzAQA1NTV4+OGHYTAYoNVq\nERMTg71792L06NGSnYdK6OA7dE0m029Z2Qig/bHVjiJAeWtjjYpT5s3TB3vcKncIAIB6n0b02v0V\nAKCuJ9DzF5kDul5Tk9wR2FDizzkAqKCcn3WTyQStnx+MRqNd13/aPZZWC+Nnn0HTs6f449TXQztp\nkkti6mxYSRERSY0LzIrGiRNERKRYslVSvr4K+8fBV0fkjsDG11FyR9CKY9JdJHWI6hcAY679ua4W\ngPjhFEl4esodgQ1VY6PcIbgnPk9KNA73ERFJjUlKNPc9cyIiUjxWUkREUmMlJRqTFBGR1JikRHPf\nMyciIsVjJUVEJDVWUqIxSRERSY1JSjQmKSIiqTFJiea+Z05ERIrHSoqISGqspERjkiIikhoXmBXN\nfdMzEREpHispIiKpcbhPNCYpIiKpMUmJ5tCZr127FsOGDYNGo4FGo0FCQgJ27dolVWxEROTmHKqk\n+vXrhxdffBEDBw4EALz77ruYNWsWysrKEBWl1IcfERHJjJWUaA4lqZkzZ1q9/uc//4m1a9eipKSE\nSYqIqDVMUqKJvibV2NiIDz74APX19UhISGi1n9lshtlstrw2mUxiP5KIiNyMw0nqyJEjSEhIwOXL\nl9GrVy/k5eUhMjKy1f6ZmZlYtmyZU0ESEXVqrKREc/jMIyIiUF5ejpKSEvz973/H3Llz8e2337ba\nPz09HUaj0bLp9XqnAiYi6nSak5Qzm5tyuJLy9va2TJyIi4vDwYMH8frrr+Ptt99usb9arYZarXYu\nSiIicktO3yclCILVNSciIroOh/tEcyhJPf3005g+fTpCQ0NRW1uLbdu2Yc+ePcjPz5cqPiKizo9J\nSjSHktS5c+eg0+lQXV0NrVaLYcOGIT8/H1OmTJEqPiKizo8LzIrmUJLasGGDVHEQERHZ4Np9RERS\n43CfaExSRERSY5ISzX3PnIiIFI+VFBGR1FhJicYkRUQkNSYp0dz3zImISPFYSRERSY2VlGhMUkRE\nUmOSEs19z5yIqItbs2YNwsPD4ePjg9jYWOzbt6/N/kVFRYiNjYWPjw9uuukmvPXWWzZ9cnNzERkZ\nCbVajcjISOTl5UkVPgAmKSIi6cnwqI6cnBwsWbIEzzzzDMrKyjBu3DhMnz4dVVVVLfY/deoUbrvt\nNowbNw5lZWV4+umnsXjxYuTm5lr6FBcXIyUlBTqdDhUVFdDpdEhOTsaBAwdEfzXtUQmCIEh29BaY\nTCZotVr4+hqhUmk68qPbZPzqqNwh2BCiouUOoUWqY8r4rupVv6BX1BgAQJ1Qi57oKXNE1/H0lDsC\nW42NckegeCaTCVo/PxiNRmg0zv2Oav59Z6ypcepYYmIaM2YMRo4cibVr11rahgwZgjvvvBOZmZk2\n/Z988kns2LEDx48ft7TNnz8fFRUVKC4uBgCkpKTAZDJh165dlj7Tpk1D7969sXXrVrGn1ybZrknV\n1sr1yS1TDVVeQpg8We4IWtbQoIzvqtGnHtgtdxRtUGJCMJnkjqBl/fvLHcHvOvbf7Q4xXff/r7Xn\n9TU0NKC0tBRPPfWUVXtSUhL279/f4rGLi4uRlJRk1TZ16lRs2LABV65cgZeXF4qLi/HYY4/Z9MnK\nyhJxNvbhcB8RkcQEqJzeACA0NBRardaytVQRAcCFCxfQ2NiIwMBAq/bAwEAYDIYW9zEYDC32v3r1\nKi5cuNBmn9aO6Qqc3UdEJLGmpmubM/sDgF6vtxrua++p56rrHvEhCIJNW3v9r2939JjOYpIiIpKY\nq5KURqOx65qUv78/PD09bSqc8+fP21RCzYKCglrs361bN/Tt27fNPq0d0xU43EdE1MV4e3sjNjYW\nBQUFVu0FBQVITExscZ+EhASb/p9++ini4uLg5eXVZp/WjukKrKSIiCTmqkrKEWlpadDpdIiLi0NC\nQgLWrVuHqqoqzJ8/HwCQnp6Os2fPYtOmTQCuzeRbvXo10tLSkJqaiuLiYmzYsMFq1t6jjz6K8ePH\nY+XKlZg1axa2b9+OwsJCfPnll+JPrh1MUkREEpMjSaWkpODixYtYvnw5qqurER0djZ07d2LAgAEA\ngOrqaqt7psLDw7Fz50489thjePPNNxESEoJVq1bh7rvvtvRJTEzEtm3b8Oyzz+K5557DzTffjJyc\nHIwZM0b8ybVDtvukACMA5dwnpUTKnYIudwTXNPrU46vdvQAo9D4pJeIU9HaZBAHa2lqX3idlMDh3\nLJPJhKAgrUti6mxYSRERSUyOSqqrYJIiIpIYk5R4nN1HRESKxUqKiEhirKTEY5IiIpIYk5R4HO4j\nIiLFYiVFRCQxQXCuGlLwwuySY5IiIpIYh/vEc2q4LzMzEyqVCkuWLHFROEREXU9zknJmc1eik9TB\ngwexbt06DBs2zJXxEBERWYhKUnV1dbjvvvuwfv169O7du82+ZrMZJpPJaiMiciespMQTlaQWLFiA\n22+/HZPtWFwuMzPT6kmSoaGhYj6SiKjTYpISz+EktW3bNhw6dKjVxxZfLz09HUaj0bLp9XqHgyQi\nIvfk0Ow+vV6PRx99FJ9++il8fHzs2ketVrf7iGMioq6Ms/vEcyhJlZaW4vz584iNjbW0NTY2Yu/e\nvVi9ejXMZjM8PT1dHiQRUWfGJCWeQ0lq0qRJOHLkiFXbgw8+iMGDB+PJJ59kgiIiIpdyKEn5+voi\nOjraqq1nz57o27evTTsREV3DSko8rjhBRCQxJinxnE5Se/bscUEYREREtlhJERFJjAvMisckRUQk\nMQ73icfnSRERkWKxkiIikhgrKfGYpIiIJMYkJR6TFBGRxJikxOM1KSIiUixWUkREEmMlJR6TFBGR\nxJikxONwHxERKZZsldTKlUD37nJ9uq1Fi1Vyh2Bj473KvM38wXfGyR0CAKC+qRG9ml/4+QG/KOz/\n4S+/yB2BLY1G7ghaVlMjdwS/M5mu/Ty5ECsp8TjcR0QkMSYp8TjcR0REisVKiohIYlxgVjwmKSIi\niXG4TzwO9xERkWIxSRERSay5knJmk9KlS5eg0+mg1Wqh1Wqh0+lQ08aMyytXruDJJ5/E0KFD0bNn\nT4SEhOD+++/Hjz/+aNVvwoQJUKlUVtucOXMcio1JiohIYkpPUvfeey/Ky8uRn5+P/Px8lJeXQ6fT\ntdr/l19+waFDh/Dcc8/h0KFD+PDDD/Hdd9/hjjvusOmbmpqK6upqy/b22287FBuvSRERubHjx48j\nPz8fJSUlGDNmDABg/fr1SEhIQGVlJSIiImz20Wq1KCgosGp74403MHr0aFRVVaF///6W9h49eiAo\nKEh0fKykiIgk5qpKymQyWW1ms9np2IqLi6HVai0JCgDi4+Oh1Wqxf/9+u49jNBqhUqngd92N0Fu2\nbIG/vz+ioqKwdOlS1NbWOhQfKykiIom5anZfaGioVfsLL7yAjIwM8QcGYDAYEBAQYNMeEBAAg8Fg\n1zEuX76Mp556Cvfeey80f1jV5L777kN4eDiCgoJw9OhRpKeno6KiwqYKawuTFBGRxFyVpPR6vVUS\nUKvVre6TkZGBZcuWtXncgwcPAgBUKtslxQRBaLH9eleuXMGcOXPQ1NSENWvWWL2Xmppq+XN0dDQG\nDRqEuLg4HDp0CCNHjmz32ACTFBFRp6HRaKySVFsWLlzY7ky6sLAwHD58GOfOnbN576effkJgYGCb\n+1+5cgXJyck4deoUPv/883ZjGzlyJLy8vHDixAkmKSIipZDjZl5/f3/4+/u32y8hIQFGoxFff/01\nRo8eDQA4cOAAjEYjEhMTW92vOUGdOHECX3zxBfr27dvuZx07dgxXrlxBcHCw3efBiRNERBJT8hT0\nIUOGYNq0aUhNTUVJSQlKSkqQmpqKGTNmWM3sGzx4MPLy8gAAV69exT333INvvvkGW7ZsQWNjIwwG\nAwwGAxoaGgAA//nPf7B8+XJ88803OH36NHbu3InZs2cjJiYGY8eOtTs+JikiIje3ZcsWDB06FElJ\nSUhKSsKwYcPw3nvvWfWprKyE0WgEAJw5cwY7duzAmTNnMGLECAQHB1u25hmB3t7e+OyzzzB16lRE\nRERg8eLFSEpKQmFhITw9Pe2OzaHhvpYuxAUGBto9A4SIyB0pfYHZPn36YPPmze3E8HsQYWFhVq9b\nEhoaiqKiIqdjc/iaVFRUFAoLCy2vHcmIRETuiAvMiudwkurWrZtDdw+bzWarG85MJpOjH0lERG7K\n4WtSJ06cQEhICMLDwzFnzhycPHmyzf6ZmZmWRQu1Wq3NzWhERF2dkidOKJ1DSWrMmDHYtGkTdu/e\njfXr18NgMCAxMREXL15sdZ/09HQYjUbLptfrnQ6aiKgzYZISz6HhvunTp1v+PHToUCQkJODmm2/G\nu+++i7S0tBb3UavVbd4VTURE1Bqnbubt2bMnhg4dihMnTrgqHiKiLocTJ8Rz6j4ps9mM48ePO3T3\nMBGRu+Fwn3gOJamlS5eiqKgIp06dwoEDB3DPPffAZDJh7ty5UsVHRERuzKHhvjNnzuCvf/0rLly4\ngBtuuAHx8fEoKSnBgAEDpIqPiKjT43CfeA4lqW3btkkVBxFRl8UkJR5XQScikhiTlHhcYJaIiBSL\nlRQRkcRYSYnHJEVEJDGlr4KuZBzuIyIixWIlRUQkMQ73icckRUQkMSYp8TjcR0REisVKiohIYqyk\nxGOSIiKSGJOUeLIlqWefBVQquT7d1qLCQrlDsPHAn+WOoBVhy+WO4BqPXwHcfu3PP/8MoKec0djy\n8pI7AluNjXJHQOQQVlJERBJjJSUekxQRkcSYpMTj7D4iIlIsVlJERBJjJSUekxQRkcSYpMRjkiIi\nkhgXmBWP16SIiEixWEkREUmMw33iMUkREUmMSUo8DvcREZFisZIiIpIYKynxmKSIiCTGJCUeh/uI\niEixWEkREUmMlZR4TFJERBJjkhLP4eG+s2fP4m9/+xv69u2LHj16YMSIESgtLZUiNiIi6gCXLl2C\nTqeDVquFVquFTqdDTU1Nm/s88MADUKlUVlt8fLxVH7PZjEWLFsHf3x89e/bEHXfcgTNnzjgUm0NJ\n6tKlSxg7diy8vLywa9cufPvtt3jllVfg5+fn0IcSEbmT5krKmU1K9957L8rLy5Gfn4/8/HyUl5dD\np9O1u9+0adNQXV1t2Xbu3Gn1/pIlS5CXl4dt27bhyy+/RF1dHWbMmIFGBx6+6dBw38qVKxEaGoqN\nGzda2sLCwhw5BBGR21HycN/x48eRn5+PkpISjBkzBgCwfv16JCQkoLKyEhEREa3uq1arERQU1OJ7\nRqMRGzZswHvvvYfJkycDADZv3ozQ0FAUFhZi6tSpdsXnUCW1Y8cOxMXFYfbs2QgICEBMTAzWr1/f\n5j5msxkmk8lqIyIix13/u9RsNjt9zOLiYmi1WkuCAoD4+HhotVrs37+/zX337NmDgIAA/OlPf0Jq\nairOnz9vea+0tBRXrlxBUlKSpS0kJATR0dHtHvePHEpSJ0+exNq1azFo0CDs3r0b8+fPx+LFi7Fp\n06ZW98nMzLSMc2q1WoSGhjrykUREnV7zKuhit+ZV0ENDQ61+n2ZmZjodm8FgQEBAgE17QEAADAZD\nq/tNnz4dW7Zsweeff45XXnkFBw8exJ///GdL4jQYDPD29kbv3r2t9gsMDGzzuNdzaLivqakJcXFx\nWLFiBQAgJiYGx44dw9q1a3H//fe3uE96ejrS0tIsr00mExMVEbkVVw336fV6aDQaS7tarW51n4yM\nDCxbtqzN4x48eBAAoFKpbN4TBKHF9mYpKSmWP0dHRyMuLg4DBgzAJ598gr/85S+t7tfeca/nUJIK\nDg5GZGSkVduQIUOQm5vb6j5qtbrNL5KIqKtzVZLSaDRWSaotCxcuxJw5c9rsExYWhsOHD+PcuXM2\n7/30008IDAy0O8bg4GAMGDAAJ06cAAAEBQWhoaEBly5dsqqmzp8/j8TERLuP61CSGjt2LCorK63a\nvvvuOwwYMMCRwxARkcT8/f3h7+/fbr+EhAQYjUZ8/fXXGD16NADgwIEDMBqNDiWTixcvQq/XIzg4\nGAAQGxsLLy8vFBQUIDk5GQBQXV2No0eP4qWXXrL7uA5dk3rsscdQUlKCFStW4Pvvv8f777+PdevW\nYcGCBY4chojIrSh5CvqQIUMwbdo0pKamoqSkBCUlJUhNTcWMGTOsZvYNHjwYeXl5AIC6ujosXboU\nxcXFOH36NPbs2YOZM2fC398fd911FwBAq9Vi3rx5ePzxx/HZZ5+hrKwMf/vb3zB06FDLbD97OFRJ\njRo1Cnl5eUhPT8fy5csRHh6OrKws3HfffY4chojIrSh5CjoAbNmyBYsXL7bMxLvjjjuwevVqqz6V\nlZUwGo0AAE9PTxw5cgSbNm1CTU0NgoODMXHiROTk5MDX19eyz2uvvYZu3bohOTkZv/76KyZNmoTs\n7Gx4enraHZtKEJrnjXQMk8kErVYLLy8jVCr7xlY7gnnnZ3KHYEP48yS5Q2iRas8XcocAAKj3+BW9\nbr0dAFDXaERP9JQ5out4eckdgS0HbqJ0VyaTCVo/PxiNRruv/7R5LK0Wjz5qhFot/lhmswmvv651\nSUydDdfuIyKSmNIrKSVjkiIikhiTlHh8nhQRESkWKykiIomxkhKPSYqISGJMUuJxuI+IiBSLlRQR\nkcSaF5h1Zn93xSRFRCQxDveJxyRFRCQxJinxeE2KiIgUi5UUEZHEWEmJJ1uSunJFrk9umWqy8tbJ\nKyyUO4KWLV8+Ue4QAACNPvXArb+98PCA4gYGlPZDDgA9esgdQctqauSO4HcNDS4/JJOUeAr7W01E\nRPQ7DvcREUmMlZR4TFJERBJjkhKPw31ERKRYrKSIiCTGSko8JikiIokxSYnH4T4iIlIsVlJERBLj\nArPiMUkREUmMw33icbiPiIgUi5UUEZHEWEmJxyRFRCQxJinxmKSIiCTGJCUer0kREZFiOZSkwsLC\noFKpbLYFCxZIFR8RUafXXEk5s7krh4b7Dh48iMbGRsvro0ePYsqUKZg9e7bLAyMi6io43CeeQ0nq\nhhtusHr94osv4uabb8att97ayh5ERETiiZ440dDQgM2bNyMtLQ0qlarVfmazGWaz2fLaZDKJ/Ugi\nok6JlZR4oidOfPTRR6ipqcEDDzzQZr/MzExotVrLFhoaKvYjiYg6JV6TEk90ktqwYQOmT5+OkJCQ\nNvulp6fDaDRaNr1eL/YjiYjIzYga7vvhhx9QWFiIDz/8sN2+arUaarVazMcQEXUJXGBWPFFJauPG\njQgICMDtt9/u6niIiLocXpMSz+HhvqamJmzcuBFz585Ft25csIKIiKTjcJIqLCxEVVUV/uu//kuK\neIiIuhylT5y4dOkSdDqdZYKbTqdDTU1Nm/u0tLCDSqXCyy+/bOkzYcIEm/fnzJnjUGwOl0JJSUkQ\n3HmAlIjIQUof7rv33ntx5swZ5OfnAwAefvhh6HQ6fPzxx63uU11dbfV6165dmDdvHu6++26r9tTU\nVCxfvtzyunv37g7FxvE6IiI3dvz4ceTn56OkpARjxowBAKxfvx4JCQmorKxEREREi/sFBQVZvd6+\nfTsmTpyIm266yaq9R48eNn0dwQVmiYgk5qrhPpPJZLX9caEEsYqLi6HVai0JCgDi4+Oh1Wqxf/9+\nu45x7tw5fPLJJ5g3b57Ne1u2bIG/vz+ioqKwdOlS1NbWOhQfKykiIom5arjv+sUQXnjhBWRkZIg/\nMACDwYCAgACb9oCAABgMBruO8e6778LX1xd/+ctfrNrvu+8+hIeHIygoCEePHkV6ejoqKipQUFBg\nd3xMUkREEnNVktLr9dBoNJb2tu5BzcjIwLJly9o87sGDBwGgxaXtBEFoc8m7P3rnnXdw3333wcfH\nx6o9NTXV8ufo6GgMGjQIcXFxOHToEEaOHGnXsZmkiIg6CY1GY5Wk2rJw4cJ2Z9KFhYXh8OHDOHfu\nnM17P/30EwIDA9v9nH379qGyshI5OTnt9h05ciS8vLxw4sQJJikiIqWQY3afv78//P392+2XkJAA\no9GIr7/+GqNHjwYAHDhwAEajEYmJie3uv2HDBsTGxmL48OHt9j127BiuXLmC4ODg9k/gN5w4QUQk\nMSXfJzVkyBBMmzYNqampKCkpQUlJCVJTUzFjxgyrmX2DBw9GXl6e1b4mkwkffPABHnroIZvj/uc/\n/8Hy5cvxzTff4PTp09i5cydmz56NmJgYjB071u74mKSIiNzcli1bMHToUCQlJSEpKQnDhg3De++9\nZ9WnsrISRqPRqm3btm0QBAF//etfbY7p7e2Nzz77DFOnTkVERAQWL16MpKQkFBYWwtPT0+7YONxH\nRCQxpS8w26dPH2zevLmdGGyDePjhh/Hwww+32D80NBRFRUVOx8YkRUQksaYmwM6Jcq3u7646PEn9\nno35hN721NfLHUHLrl6VO4JrGq/WW36MTIIJjWiUN6DrKfE3i1KXNFPQE7tNv91syuXflKHDk9Tv\ndxvzCb3tmTVL7gg6Ae21/4TgRnnjIOe0cDOp3Gpra6HVal1yLFZS4nV4kgoJCYFer4evr6/dN4q1\nxGQyITQ01ObmNjkpMSZAmXExJvswJvu5Ki5BEFBbW9vuU8cdwSQlXocnKQ8PD/Tr189lx3Pk5raO\nosSYAGXGxZjsw5js54q4XFVBkfM4cYKISGKspMRjkiIikhiTlHid9mZetVqNF154oc0FFjuaEmMC\nlBkXY7IPY7KfUuMi56gEzrMkIpKEyWSCVqtFRIQRnp7ir5M1NppQWamF0WhU5HVAKXG4j4hIYhzu\nE49JiohIYkxS4nXaa1JERNT1sZIiIpKY0heYVTImKSIiiTk7XMfhPiIiIgViJUVEJDFWUuIxSRER\nSYxJSjwO9xERkWKxkiIikhgrKfGYpIiIJMYkJR6H+4iISLFYSRERSYyVlHhMUkREEmOSEo/DfURE\npFispIiIJMZKSjwmKSIiiXGBWfGYpIiIJObs86TcOUnxmhQRESkWKykiIomxkhKPSYqISGJMUuJx\nuI+IiBSLlRQRkcRYSYnHJEVEJDEmKfE43EdERIrFSoqISGKspMRjkiIikhiTlHgc7iMicnP//Oc/\nkZiYiB49esDPz8+ufQRBQEZGBkJCQtC9e3dMmDABx44ds+pjNpuxaNEi+Pv7o2fPnrjjjjtw5swZ\nh2JjkiIiklhTk/OblBoaGjB79mz8/e9/t3ufl156Ca+++ipWr16NgwcPIigoCFOmTEFtba2lz5Il\nS5CXl4dt27bhyy+/RF1dHWbMmIHGxkb7gxOIiEgSRqNRACAARuHaoJ3Y7dpxjEajpPFu3LhR0Gq1\n7fZramoSgoKChBdffNHSdvnyZUGr1QpvvfWWIAiCUFNTI3h5eQnbtm2z9Dl79qzg4eEh5Ofn2x0T\nKykiIsmZXLABJpPJajObzR18HtecOnUKBoMBSUlJlja1Wo1bb70V+/fvBwCUlpbiypUrVn1CQkIQ\nHR1t6WMPJikiIol4e3sjKCgIQCgArRNbKHr16oXQ0FBotVrLlpmZ2eHnBAAGgwEAEBgYaNUeGBho\nec9gMMDb2xu9e/dutY89OLuPiEgiPj4+OHXqFBoaGpw+liAIUF03RVCtVrfaPyMjA8uWLWvzmAcP\nHkRcXJzomK6Pp6UYr2dPnz9ikiIikpCPjw98fHw6/HMXLlyIOXPmtNknLCxM1LGvVYfXqqXg4GBL\n+/nz5y3VVVBQEBoaGnDp0iWraur8+fNITEy0+7OYpIiIuiB/f3/4+/tLcuzw8HAEBQWhoKAAMTEx\nAK7NECwqKsLKlSsBALGxsfDy8kJBQQGSk5MBANXV1Th69Cheeukluz+LSYqIyM1VVVXh559/RlVV\nFRobG1FeXg4AGDhwIHr16gUAGDx4MDIzM3HXXXdBpVJhyZIlWLFiBQYNGoRBgwZhxYoV6NGjB+69\n914AgFarxbx58/D444+jb9++6NOnD5YuXYqhQ4di8uTJdsfGJEVE5Oaef/55vPvuu5bXzdXRF198\ngQkTJgAAKisrYTQaLX2eeOIJ/Prrr3jkkUdw6dIljBkzBp9++il8fX0tfV577TV069YNycnJ+PXX\nXzFp0iRkZ2fD09PT7thUguDOC24QEZGScQo6EREpFpMUEREpFpMUEREpFpMUEREpFpMUEREpFpMU\nEREpFpMUEREpFpMUEREpFpMUEREpFpMUEREpFpMUEREp1v8DhdbYf1mMv0gAAAAASUVORK5CYII=\n" } } ], "source": [ "mat = torch.concat([g,h,soft(netout)],axis=1)[:8].data\n", "plt.matshow(mat,cmap='bwr',vmin=-1,vmax=1);\n", "plt.axvline(x=3.5,color=\"lime\")\n", "plt.xticks(\n", " ticks=range(mat.shape[-1]),\n", " labels=[r\"$g_0$\",r\"$g_1$\",r\"$h_0$\",r\"$h_1$\",\n", " r\"$P_a$\",r\"$P_b$\",r\"$P_c$\",r\"$P_C$\"]\n", ")\n", "plt.colorbar()" ], "id": "2a840f6a-c978-4a9e-9fa2-a9982e405fb9" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## B. 시각화1: $({\\boldsymbol g}_t, {\\boldsymbol c}_{t-1}) \\to {\\boldsymbol c}_{t}$" ], "id": "ffa2840d-529d-45fe-9595-12a180d78845" }, { "cell_type": "code", "execution_count": 30, "metadata": { "tags": [] }, "outputs": [], "source": [ "mat1 = torch.concat([g[1:9],i[1:9],g[1:9]*i[1:9]],axis=1).data\n", "mat2 = torch.concat([c[:8],f[1:9],c[:8]*f[1:9]],axis=1).data\n", "mat3 = torch.concat([g[1:9]*i[1:9],c[:8]*f[1:9],c[1:9]],axis=1).data" ], "id": "3dd1ed7b-47af-4cbf-84fb-9762ec630baf" }, { "cell_type": "code", "execution_count": 32, "metadata": { "tags": [] }, "outputs": [], "source": [ "c[:8].min(),c[:8].max() " ], "id": "60a3f9bc-998c-49c6-a36e-8ebdc6409014" }, { "cell_type": "markdown", "metadata": {}, "source": [ "- ${\\boldsymbol c}_t$의 경우 원래 출력값이 -1, 1 사이라는 보장은\n", " 없으나, 이 예제의 경우에는 거의 -1, 1 사이임.\n", "- `matshow`로 시각화할때 `vmin=-1`, `vmax=1`로 설정해도\n", " ${\\boldsymbol c}_t$를 표현함에 부족함이 없을듯" ], "id": "bcea141f-3c5e-4839-b3e7-c965538274d6" }, { "cell_type": "code", "execution_count": 33, "metadata": { "tags": [] }, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA94AAAG9CAYAAADjkq72AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90\nbGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9h\nAAAPYQGoP6dpAABK0klEQVR4nO3de5ybZZ3///edZCaTZA4Ve26HtiAIbSkiLVhU2iLglhZ1sS3u\norawrgoF12Vh16q71D1QFL+7ugvyXVABvx7A4oIIVYSVtqI/pMUTbUUL9MihB6STmZymk1y/PzJJ\np9KBZmbuue4r9+vZRx7eU2bMp/PJ9c79Se7ct2eMMQIAAAAAAL6I2C4AAAAAAIB6xuANAAAAAICP\nGLwBAAAAAPARgzcAAAAAAD5i8AYAAAAAwEcM3gAAAAAA+IjBGwAAAAAAHzF4AwAAAADgIwZvAAAA\nAAB8xOANAAAAAICPGLwBAAAAAPARgzecMnnyZHmep5UrV9ouBYNAH4GhZ4zRtddeq/HjxysSicjz\nPG3fvt12WRhi9DlceL50A+sSRyNmuwCgFqeddprGjh2riRMn2i4Fg0AfgaF333336Ytf/KIk6eST\nT1Zra6vi8bjlqjDU6HO48HzpBtYljgbveMMp9957rx5//HF95CMfsV0Ket19990688wzlUgkNHLk\nSP3lX/6l/vCHP7zmz9BHYOht3rxZkjRy5Eht3rxZjz/+uMaNG2e5KryeWjOUPruL50t3sC7hBwbv\n1/HKK6/o4osvVjKZ1LHHHqtbbrlFc+fOled5mjt3ru3yQodDrobOUDy2P/WpT+lTn/qUPvOZz2jf\nvn36zW9+oze96U0666yz9MQTT/T7c/TRDcYYfeUrX9Fpp52mRCKhlpYWnXHGGfr1r39tuzT8iblz\n5+of//EfJUn79++vHuqIIwvKY7vWDKXPdvB8OTxYl/UvKD22xuA1XXTRRUaSkWTe/OY3m1QqZVKp\nlJFk5syZY7u80Jk0aZKRZK677jrbpThvsI/tn/zkJ2bSpEnm5ZdfftV/W716tTn++OPNwYMHj/iz\n9NENV155ZfUx8sY3vtFMmzbNNDY2mnvvvdd2afgTl19+uZkwYYKRZBobG82ZZ55pzjzzTNtlBVYQ\nHtsDyVD6bAfPl8ODdVn/gtBjmxi8X8MzzzxTfXBcc801xhhjfve735lYLPaqsH322WfN/fffb6nS\n8PjTJyB+7wMzFI/tSy+91Hz+85/v9z5OPfVUs379+iP+tyPtSNDLYNm2bZvxPM9IMhdddJEpFArG\nGGP27t1rdu7caYwZfM/e9773mREjRpj3v//9Q1Jz2F133XVGkpk0adJhf8/aOlxQHtsDzVD6PLx4\nvhwerMv6Nxw9DjoONX8Nlc9rSNKSJUskSSeddJJmzJjxqu/94Q9/qKeffnrYakMZv/eBGYrH9u7d\nu/XmN7+5+vWIESMOOxTuxBNP1O7du4+6JnoZLBs2bJAxRpJ09dVXq7GxUZI0atQotbe3Sxp8zz7x\niU/oG9/4xuCLxWtibR0uKI9tMtQNPF8OD9Zl/RuOHgcdg/dR6vtZjcqDpmLdunX67Gc/q9tuu02n\nnXaacrnccJcXSvzeh8ZAH9sTJkzQ1q1bq1+feOKJGjlyZPXrZ555RuPHjz+qGuile4aiZ/PmzVNL\nS4sP1aGCtVW74Xpsk6Hu4fnSHtZl/QvF78viu+2B9+yzz1YPL1qxYoUxpv/Di97xjndUD5OAf/70\nkCt+7wMzFI/thx9+2Bx33HHmwIEDr/pv9913n5k8ebLp7u4+4v0f6dA5ehksfQ8JW7RoUfWQsP37\n95tdu3YZY4amZ48++iiHmg+R/g51ZG0dLiiP7YFmKH0eXjxfDg/WZf0brh4HGe94v4bjjjtOF110\nkSRp1apVOvnkkzVz5kw1NDS86nt3795dPUwCw4ff+8AMxWP73HPP1Xve8x6dccYZWrNmjQ4cOKAd\nO3bo+uuv12WXXaZvfetbR/z/6w+9DJbJkydr+fLlkqR77rlHEyZM0CmnnKIJEyZo48aNkvrv2emn\nn67p06e/6vbCCy8M678BZaytwwXlsU2GuoHny+HBuqx/g+lxvYjZLiDovvrVryoajeqBBx5QZ2en\nbrjhBt15553auHGjEomEpPKDZMKECZYrDR9+74MzFI/t//iP/9Bpp52mFStWaNOmTWpqatL555+v\nn/70p5o6depR10Ivg+k///M/dfLJJ+u2227T008/rW3btmnGjBmaPHnya/bsySefHOZK0R/W1pEF\n5bFNhrqB58vhwbqsfwPtcd2w/ZZ70O3cudPkcrnq188884xpamoyksynPvUpY4wx69evN4sXL7ZV\nYmjxex+coX5sF4vFAddCL90zVD3jUHN/sbZqZ+uxTYYGF8+X9rEu618Yfl8cav46vve972nixIl6\n97vfrfnz5+vUU09VPp/XmDFjdNVVV0mSpk+frq1bt+qUU06p6zPxBQ2/98EZ6sd2JDLwOKGX7hmK\nnr373e/W4sWLtWbNGk2cOFEbNmwY4irB2qqdrcc2GRpcPF/ax7qsf2H4fXnG/MlpGXGY//3f/9Wn\nP/1pPf3008pmsxo7dqzOO+88rVy5Uscee6zt8oAB47ENAMDr4/kSwFBg8AYAAAAAwEccag4AAAAA\ngI8YvAEAAAAA8BGDNwAAAAAAPmLwBgAAAADARwzeAAAAAAD4iMF7AAqFglauXKlCoWC7lFCjD8FA\nH9AXj4dgoA/BQS9QwWMhOOhFMIStD1xObADS6bTa2trU0dGh1tZW2+WEFn0IBvqAvng8BAN9CA56\ngQoeC8FBL4IhbH3gHW8AAAAAAHzE4A0AAAAAgI9iw32HpVJJL7zwglpaWuR53nDf/ZBIp9OH/S/s\noA/BUA99MMaos7NT48ePVyRi7/VI8hFDhT4ERz30IggZST5iKNGLYKiHPtSSj8P+Ge/du3ervb19\nOO8SAI7Krl27NHHiRGv3Tz4CCDKbGUk+Agiyo8nHYX/Hu6WlpXdrl6T6/xB9ICUy0kvjJUkv7P21\nUkpZLmgQTjjBdgUDlklI418qb/967wtu90FOt0JSWlJ7n3yyo3L/O3fucvokI15Xp+0SBiyjrMa3\nnChJemGclMpaLmgwDhywXcGAZZTReG+CJOl5434+enL7PLbpdFrtxx5rNSMr9/3LX+5SS4u7+ej2\nc6UO24d0eW26vib7ZqTr+5DOr4ka9iGHffA+dHhQqxi8LfGi1V99a67F6cXqsqinah9acq30IQBs\nH75Yuf/W1la3B++Im4eBSlJUUan3ubPVk1KOHtIqSXL4MRRVVOr91bca9/PR9Z38CpsZWbnvlpZW\npwdv5/Xdh3R4bbq+JvtmJPuQwXA0+cjJ1QAAAAAA8BGDNwAAAAAAPmLwBgAAAADAR8P+GW8EwMEG\n6cZrdM0VGTWowXY1odVwULrmxt7tD9MHIAga1KBruj8h/ed/quGg7WrCq0ENusb8XXUbQEAcbNDf\nmWsksTZtqmRkJuvRB4cM++XE0um02traJHWIk6vZZV7aY7uEwRs71nYFQ2LPS26f5ENyvRVpSW3q\n6OiwelKzSj4eOGC3jsFy+azmVW1ttisYvGLRdgVDwsjhE9z1cv1ETul0Wm0jRljNyEo+bt3a4fTJ\n1dx+riwrlWxXMHiur8mKPXvdz0f318TR70NyqDkAAAAAAD5i8A4jryRN2q7tkZ0qqQ5etnRUyZO2\nTyrf6AMQDCWVtN3boe2TjEpefbwj4qKSStre+4d8BALEY20GQSUjd0bog0v4jHcYJXLS9imaIqlr\nz7Nc+8+SXEKasr28/eyeHH0AAiCnnKY0nyJtk7qapVTWdkXhlFNOU7zjJEmdpot8BIIikdNx3hRJ\nrE2bqhk5Snp2D31wBe94AwAAAADgIwZvAAAAAAB8xOANAAAAAICPGLwBAAAAAPARgzcAAAAAAD5i\n8AYAAAAAwEdcTiyMemLSzVfoikuzivEQsCbWI11xc+/2IvoABEFMMV3R/RHpq19VrMd2NeEVU0xX\nmMur2wACoiemy80VklibNlUyMpvz6INDPGOMGc47TKfTamtrk9QhqXU47xp/wry0x3YJgzd2rO0K\nhsSel4Z1GfrC7VakJbWpo6NDra32cqmSjwcO2K1jsLyuTtslDF5bm+0KBq9YtF3BkDDybJcwaJ7c\nzvh0Oq22ESOsZmQlH7du7VBLi7v56PZzZVmpZLuCwXN9TVbs2et+Prq/Jo5+H5JDzQEAAAAA8BGD\ndygZaeQ+7fP2y9TJK34uMpL2jSzf6AMQDEZG+7z92jfSsC4tMjLa1/uHPgBBwtoMgkpG7vfog0sG\nNHh/5Stf0ZQpU9TU1KTTTz9dP/3pT4e6LvgpmZX2jdbo0dOVVdZ2NaGVTUqj95Vv9KF+kI9uyyqr\n0c3HafTe8hqFHVllNdobo9HeGPKxzpCRjktmNcYbrTHeaNamRZWMnD6aPrik5sH77rvv1ic/+Ul9\n5jOf0a9+9Su9853v1Pz587Vz504/6gMAZ5CPANA/MhJAmNU8eP/7v/+7/uqv/kof+chHdPLJJ+tL\nX/qS2tvbdcstt/hRHwA4g3wEgP6RkQDCrKbBu7u7W08++aTOP//8w/7+/PPP189//vMj/kyhUFA6\nnT7sBgD1hnwEgP7VmpHkI4B6U9PgvX//fhWLRY0ZM+awvx8zZoxeeumlI/7MqlWr1NbWVr21t7cP\nvFoACCjyEQD6V2tGko8A6s2ATq7meYdfM84Y86q/q1ixYoU6Ojqqt127dg3kLgHACeQjAPTvaDOS\nfARQb2K1fPPIkSMVjUZf9crk3r17X/UKZkU8Hlc8Hh94hQDgAPIRAPpXa0aSjwDqTU3veDc2Nur0\n00/Xww8/fNjfP/zwwzrrrLOGtDD4qCcm3bFUS3NLFKvttRcMoViPtPSO8o0+uI98rA8xxbT04F+W\n12WP7WrCK6aYlpqlWmqWko91goysEz2szSCoZOSSHH1wSc2duvrqq/WhD31IM2fO1OzZs3Xrrbdq\n586d+vjHP+5HffBDd1y69A7dMX+P7UpCLd4t3XFpeXvPS7yqXw/IR/fFFdcd+f8rXfYd26WEWlxx\n3aHbJUlGR/6oBtxDRtaB7rhu1x22qwi9SkbuSZOPLql58L744ov18ssv65//+Z/14osvavr06Vqz\nZo0mTZrkR30A4AzyEQD6R0YCCDPPGGOG8w7T6bTa2tokdUhqHc67RpWRkll1PbdHSSXlufxuwtix\ntisYMCMpmyxvdz5XcrsPcroVktKS2tTR0aHWVnu5VMnHAwfs1jFYXlen7RIGzMgoq6w0bpySWbm9\nLotF2xUMWLUPkhJKud0HSZ6GdVdryKXTabWNGGE1Iyv5uHVrh1pa3M1Ht58rJcmos1Remy7vQ7q+\nJisZuWev53QfpHpYE0e/Dzmgs5rDccmslGlW85jjqzs2GH7ZpNScKd/oAxAMWWXV3DJOzV2HXhjD\n8Msqq2avRc1eC/kIBEkyqxavWS1eM2vTokpGHj+GPriEwRsAAAAAAB8xeAMAAAAA4CMGbwAAAAAA\nfMTgDQAAAACAjxi8AQAAAADwEYM3AAAAAAA+itkuABYUo9LqRVp0YV5RRW1XE1rRorRode/22fQB\nCIKoolp08H3Sffcp6u5lsJ0XVVSLzKLqNoCAKLI2g6CSkfkCfXCJZ4wZ1ivIp9NptbW1SeqQ9NoX\nGYe/zEt7bJcweGPH2q5gSOx5aViXoS/cbkVaUps6OjrU2movlyr5eOCA3ToGy+vqtF3C4LW12a5g\n8Ir18cqBkWe7hEHz5HbGp9NptY0YYTUjK/m4dWuHWlrczUe3nyvLSiXbFQye62uyYs9e9/PR/TVx\n9PuQHGoOAAAAAICPGLwBAAAAAPARg3cYJTOS8eSNGauMMrarCa1MUvJM+UYfgGDIKCOvpVVeySiT\nrI9DEV2UUUaeF5HnRchHIEiSGUU8TxHPY21aVMnIsWPog0sYvAEAAAAA8BGDNwAAAAAAPmLwBgAA\nAADARwzeAAAAAAD4iMEbAAAAAAAfxWwX4Cojdy9Yn5HUXPniuOOlrMViIEk6/jg53weX10RaUpvt\nIvrYt0/K521XMXBjx7bYLmHgkhFVTxBrJBmHz2xeKtmuYBBKUrS8tXevlLJbzKCNHOluPkpSsRSc\n+k84wXYFg+Pyc6V0+D6ky2vT+TUpr5qRru9DmliD7RIGJW2M2opH970M3iEULUoXPHhoG3b07cOa\nYtRuMQDKilHpwQt0gdaQjxZFFdUFZn51G0AwRIvSuwoXlLdZm9ZEFdV8c4F+uEbl5y04wTNmeF/O\nT6fTamtrk9QhqXU473pIuf6KJYLFk8PvqvVyeU1U3vHu6OhQa6u9XKrk49atHWppcTcfx461XcHg\nufx4rurpsV3BkNiz3/2dypEjbVcwOOl0Wscc02Y1I9l/DI49L7m/z+L6mqyI1cFbqPXxjnfxqPKR\nz3gDAAAAAOAjBm8AAAAAAHzE4B1CmaSU6irfMknb1YRX3z4omXnd7wcwDJIZqStFPlqWUUapSKtS\nkVZlRD4CQZFJSlNGpzRldIq1aVFGGbVGUlJXin1Ih9TBJwMwEFlXT0NZZ+gDEECprMsniK0bWY8u\nAEGUY20GQtbLunta+ZDiHW8AAAAAAHzE4A0AAAAAgI8YvAEAAAAA8BGDNwAAAAAAPmLwBgAAAADA\nR5zVPIQiJWnO2kPbsKNvH9aVeA0MCIRSRFo7R3O0jny0KKKI5pizq9sAgiFSkmZ3zylvszatiSii\ns80crV+n8vMWnMDgHUKJvLR2nu0q0LcPnhJ2iwFQlk9I89ZqrTzblYRaQgmtLf1EkrRHUcvVAKhI\n5KV7X1lru4zQSyihn5TWKsb+vFN4iQQAAAAAAB8xeAMAAAAA4CMG7xDKJKVRe8u3TNJ2NeHVtw9K\nZmyXA0Aqr8W9o8hHyzLKaFRkrEZFxioj8hEIikxSmjpqlKaOGsXatCijjMZGRkl7R7EP6RA+4x1S\n+0fZrgASfQACadR+7bddA7TfowtAEP0xwtoMgv3efon9SKfwjjcAAAAAAD5i8AYAAAAAwEcM3gAA\nAAAA+KjmwXv9+vW68MILNX78eHmep/vuu8+HsgDAPeQjABwZ+Qgg7GoevDOZjE499VTddNNNftQD\nAM4iHwHgyMhHAGFX81nN58+fr/nz5/tRC4ZJpCTN3HBoG3b07cPGEp/6qAfkYx0oRaQNMzVTG8lH\niyKKaKaZWd2G+8jH+hApSaceZG3aVsnIjRtVft6CE3y/nFihUFChUKh+nU6n/b5LvI5EXtpwhu0q\n0LcPnhJ2i4EV5GMA5RPSGRu0QZ7tSkItoYQ2lB6XJO1R1HI1sIF8DKZEXnrojxtslxF6CSX0eGmD\nYuzPO8X3l0hWrVqltra26q29vd3vuwQAJ5CPAHBk5COAeuP74L1ixQp1dHRUb7t27fL7LgHACeQj\nABwZ+Qig3vh+qHk8Hlc8Hvf7blCDbEKauqW8vWWqlMzZrSes+vZBU7NSLmm1Hgw/8jGAEllpy1RN\nFvloU1ZZTY2cIkl6VFuUFPkYNuRjMGUT0syRkyVJ6/ezNm3JKqtTIlOlbSrvTLIP6QTfB28Ej/Gk\nHZMPbcOOvn2QZ2yWAqDCM9LkHdoh8tEmI6Md3o7qNoBgMJ60O8ratK2akZPFPqRDah68u7q69Mwz\nz1S/3rZtm37961/rmGOO0bHHHjukxQGAS8hHADgy8hFA2NU8eG/cuFHz5s2rfn311VdLkpYuXao7\n7rhjyAoDANeQjwBwZOQjgLCrefCeO3eujOGQBgD4U+QjABwZ+Qgg7LjiOgAAAAAAPmLwBgAAAADA\nR5zVPIQ8I03dfGgbdvTtwxZOnwwEg/GkzVM1VVvIR4s8eZpqpla3AQSDZ6QTe1ibtlUycssWcQkO\nhzB4h1AyJ22ebrsK9O2Dx3UwgWDIJaXpm7WZHUqrkkpqc+m3kqQ9ilquBkBFMietf3mz7TJCL6mk\nflvarBj7807hUHMAAAAAAHzE4A0AAAAAgI8YvEMom5CmbSrfsgnb1YRX3z4okbVdDgCpvBY3TSMf\nLcsqq2mRGZoWmaGsyEcgKLIJ6ew3TtPZb5zG2rQoq6xmRKZJm6axD+kQPuMdQsaTtkw7tA07+vaB\nszgBAeEZadoWcb4au4yMtnhbqtsAgsF40h9irE3bqhk5TexDOoR3vAEAAAAA8BGDNwAAAAAAPmLw\nBgAAAADARwzeAAAAAAD4iMEbAAAAAAAfcVbzEPKMNGn7oW3Y0bcPOzh9MhAMxpO2T9Ik7SAfLfLk\naZKZVN0GEAyekSYWWZu2VTJyxw5xCQ6HMHiHUDInbZ9iuwr07YOnpN1iAJTlktKU7drODqVVSSW1\nvfSsJGmPoparAVCRzEkb92+3XUboJZXUs6XtirE/7xQONQcAAAAAwEcM3gAAAAAA+MjaoeYtLZLn\n8pF8P3vKdgUDlvPyOnvKMimR0HqtU0IJ2yUNXMTd145yTdLZ68vbmZk5t/sgSZvdXRPq6pJmz7Zd\nRdUJJ9iuYHBMrMF2CQOWazI6+9GiZjUltH7ZFCUK7maM0/monM7WHEnS82etl5d3Ox+f3+32CQOi\nkeDUf+CA1Npqu4pBcPm5UuV9yPYXZ0mSRr7f3bXp+pqsZGTDr9zugyTpoV/ZrmBwatiH5DPeIVRS\nSRuTm8vbpmS5mvAqRaSNs3q36QMQCKWItHGmJOVUcndudV5JJW30NkqSxnolPnEPBERJJR18S3lt\nGtamNdWMfAt9cAm7FQAAAAAA+IjBGwAAAAAAHzF4AwAAAADgIwZvAAAAAAB8xOANAAAAAICPOKt5\nSI3seYMUjdouI/RG7qtsWC0DQB8j94l8DICRhmAEgijyMmszCEaakfrjH21XgVoweIdQyiS17+n1\n0rRptksJtVRW2je6vG1KKbvFAJAkpbKe9o2PSSedZLuUUEsppX3aK0makONCOUBQpExSY2fse/1v\nhK8qGTlhBvnoEg41BwAAAADARwzeAAAAAAD4iME7hHJeXnOnXKq5mqeccrbLCa1ckzT30fKNPgDB\nkGsymvtIj+Z+/Tnl4iXb5YRWTjnN1TzN1TyZJvIRCIqcl9f+1XO1f/Vc1qZFlYykD27hM94hVFJJ\n61Iby9uGHUtbShFp3dzebfoABEIpIq2bI0lZlXhp2pqSSlrnrZMkjfVK4lOMQDCUVFL3WeW1aVib\n1lQz8iz64BJ2KwAAAAAA8BGDNwAAAAAAPmLwBgAAAADARwzeAAAAAAD4iMEbAAAAAAAfcVbzkEqW\nEpLHORBtS2YqG1bLANBHMiPyMQCShmAEgsjLsjaDIGmSynElMacweIdQyiSV2fKENG2a7VJCLZWV\nMs3lbVNK2S0GgCQplfWUGRGTTjrJdimhllJKGXVJkibkeBEECIqUSWrcCZnX/0b4qpKRE04gH13C\noeYAAAAAAPiopsF71apVmjVrllpaWjR69Gi9733v0+9//3u/agMAZ5CPANA/MhJA2NU0eK9bt07L\nly/X448/rocfflg9PT06//zzlclwyIlL8l5BCyZdoQVaqLzytssJrXxcWvBA+UYf3Ec+1od83GjB\n94tacPN25RtLtssJrbzyWqCFWqCFMnHysR6QkfUh7xX08jcW6OVvLGBtWlTJSPrglpo+4/2jH/3o\nsK9vv/12jR49Wk8++aTOPvvsIS0M/imqqDUtPy1vm6LlasKrGJXWLOjdpg/OIx/rQzEqrbnASOpS\nMWq7mvAqqqg13hpJ0thIUXyK0X1kZH0oqqjCu8pr07A2ralm5Lvog0sGdXK1jo4OSdIxxxzT7/cU\nCgUVCoXq1+l0ejB3CQBOIB8BoH+vl5HkI4B6M+CTqxljdPXVV+sd73iHpk+f3u/3rVq1Sm1tbdVb\ne3v7QO8SAJxAPgJA/44mI8lHAPVmwIP3lVdeqd/+9rf6zne+85rft2LFCnV0dFRvu3btGuhdAoAT\nyEcA6N/RZCT5CKDeDOhQ86uuukr333+/1q9fr4kTJ77m98bjccXj8QEVBwCuIR8BoH9Hm5HkI4B6\nU9PgbYzRVVddpXvvvVdr167VlClT/KoLAJxCPgJA/8hIAGFX0+C9fPlyffvb39b3v/99tbS06KWX\nXpIktbW1KZFI+FIgALiAfASA/pGRAMKups9433LLLero6NDcuXM1bty46u3uu+/2qz74IGWSMpue\nkjElpZSyXU5opbKS8co3+uA+8rE+pLKeTENM5pTpSuUGfBoUDFJKKRlTkjElRXLkYz0gI+tDyiQ1\nfoLR+AmGtWlRJSPpg1tqPtQcAPBq5CMA9I+MBBB2vJwPAAAAAICPGLxDKO8VtLj9ai3WEuWVt11O\naOXj0uLvlm/0AQiGfNxo8XeKWvx/dirfWLJdTmjllddiLdFiLZGJk49AUOS9gv7434v1x/9ezNq0\nqJKR9MEtDN4hVFRR97Q9rHu8e1RU0XY5oVWMSvcsLt/oAxAMxah0zyKje85Pqxi1XU14FVXUPd49\nuse7RyZCPgJBUVRR+YX3KL+QtWlTJSPpg1sYvAEAAAAA8BGDNwAAAAAAPmLwBgAAAADARwzeAAAA\nAAD4iMEbAAAAAAAfMXgDAAAAAOCjmO0CMPySJqGuzb+Qpk5VUknb5YRWMit1pcrbiS76AARBMit1\ntUWlN79ZyZxnu5zQSiqpLtMpSTohRz4CQZE0CY19U5ckyWNtWlPJyDed4NEHhzB4h5AnTymTlJSy\nXUqoeZJS2fK2ETv4QBB48srrMscBYTZ58pTqfY7yyEcgMDx5iuTYf7StkpERXiB2CnsWAAAAAAD4\niME7hApet5ZN+IyW6VIVVLBdTmgVGqVlt5dv9AEIhkKj0bKvFbXsX3er0FCyXU5oFVTQMl2qZbpU\nppF8BIKi4HXrlf9Yplf+Yxlr06JKRtIHtzB4h1CPenTnG+7Xnd6d6lGP7XJCqycm3bmsfKMPQDD0\nxKQ7P2x053sPqIcPY1nTox7d6d2pO707ZaLkIxAUPepRbsmdyi1hbdpUyUj64BYGbwAAAAAAfMTg\nDQAAAACAjxi8AQAAAADwkbVPsHV22rrnoeGdMt12CQOXzEiZ3u2uLknGZjVQuQ2ud6HV5TWhtO0C\nDjN2rBRx+WXRh35lu4KB87KSzixvb94iZa1WA5XXgtPrAUOqq8vtx4Pbz5U6bB+StRkMmUzvU5ej\noqe6vSaMOfp9SJYLAAAAAAA+YvAGAAAAAMBHXCwljLJJadRe7d3WpaSStqsJrWRW2juqvB1/jj4A\nQZA0Ce393Trp7DlKOnzonuuSSmqv2SNJOi1HPgKBkU1q3Kl7JUkea9OaSkYed7xX3q+HExi8Q8mT\n9o/SKNNku5BQ8ySN2l/e7pRntRYAZZ48jSoeI+23XUm4efI0SqOq2wCCwlP0j6NsFxF6lYyMvEw+\nuoRDzQEAAAAA8BGDdxg1FqSblmt5/GoVVLBdTWgVGqXlN5Vv9AEIhoLXreXj/rW8LhttVxNeBRW0\nvPePaSQfgcBoLOiVf12uV/6VtWlTJSNzX6QPLvGMMcN6FaN0Oq22tjZJHZJah/OuUZHMSJlmSVJX\n54tKKWW5oEFodfcxlElKzb2X5Hixs8vtPsjpVqh8ObE2dXR0qNXiP6SSj2PHdigScfcX+vxDm2yX\nMGAZL6vmaeXLiXWlpJTLn/MulWxXMGAZZdTstUiSxp/YpUjO7XzctdPtC0am02m1jRhhNSMr+bh7\nt92cHiyHSy/rsw/p8tp0fU32zciWcV3ysm72Qeq9srHDjEnLmKPbh+QdbwAAAAAAfMTgDQAAAACA\njxi8AQAAAADwEYM3AAAAAAA+YvAGAAAAAMBHDN4AAAAAAPgoZrsAWJBLSJO3adumLiWUsF1NaCVy\n0rbJvdtP0QcgCBKmSdt+/yPp3X+mRM52NeGVUELbzHOSpHfkyUcgMHIJjZ29TZLksTatqWTkKTO8\n8n49nMDgHUYmIu2YrMmm03YloRYx0uQd5e1ODj4BAiGiiCYfnCDtsF1JuEUU0WRNliR5xrNbDIBD\nTESx3ZNtVxF6lYyM7CQfXcLePgAAAAAAPmLwDqOGbukL1+ra+GfVrW7b1YRWd4N07RfKN/oABEO3\nd1DXjvk/5XXZYLua8OpWt67t/WMayEcgMBq6deAz1+rAZ1ibNlUyMv8v9MElnjHGDOcdptNptbW1\nSeqQ1Dqcd42KZEbKNEuSujpfVEopywUNQqu7j6FMUmrOlLdf7Oxyuw9yuhWS0pLa1NHRoVaL/5BK\nPo4d26FIxN1f6PMPbbJdwoBlvKyap50pSepKSams5YIGo1SyXcGAZZRRs9ciSRp/YpciObfzcdfO\nYd3VGnLpdFptI0ZYzchKPu7ebTenB8vh0sv67EO6vDZdX5N9M7JlXJe8rJt9kKSuLtsVDI4xaRlz\ndPuQvOMNAAAAAICPGLwBAAAAAPBRTYP3LbfcohkzZqi1tVWtra2aPXu2fvjDH/pVGwA4g3wEgP6R\nkQDCrqbBe+LEibrhhhu0ceNGbdy4Ueecc47e+973avPmzX7VBwBOIB8BoH9kJICwq+k63hdeeOFh\nX//bv/2bbrnlFj3++OOaNm3akBYGAC4hHwGgf2QkgLCrafDuq1gsavXq1cpkMpo9e3a/31coFFQo\nFKpfp9Ppgd4lADiBfASA/h1NRpKPAOpNzYP3U089pdmzZyufz6u5uVn33nuvpk6d2u/3r1q1Sp/7\n3OcGVSSGWC4hTdukTU9klFDCdjWhlchJm3pf5E88Th/qAfnovoRp0qat90rv+3MlcrarCa+EEtpk\nnpIkvTtPPtaLWjKSfAyoXEJj3lW+ZKTH2rSmkpFnvs0r79fDCTVfx7u7u1s7d+7UgQMH9L3vfU9f\n/epXtW7dun6D80ivWLa3t4vreNtn0p22Sxg85y+IWdaZdvt6kpLrrRia63gPVT5yHe8AOOUU2xUM\nnsPX8e6r/VjPdgmD5vo1g4fqOt61ZGR/+ch1vO2bONF2BYPn+pqsaBvhfj6G6TreNb/j3djYqDe9\n6U2SpJkzZ2rDhg368pe/rP/+7/8+4vfH43HF4/Fa7wYAnEM+AkD/aslI8hFAvRn0dbyNMYe9IgkH\nNHRL163Uysbr1a1u29WEVneDtPK68o0+1Cfy0T3d3kGtHP2V8rpssF1NeHWrWyt7/5gG8rFekZEO\nauhWx9+uVMffsjZtqmRk/lP0wSU1HWr+6U9/WvPnz1d7e7s6Ozt111136YYbbtCPfvQjnXfeeUf1\n/5FOp9XW1iYONbcomZEyzZKkrs4XlVLKckGD4PAxW5mk1Jwpb7/Y2eV2H+R0KzQUh5oPZT5yqLk9\nGS+r5mlnSpK6UlIqa7mgwXD4UPOMMmr2WiRJ40/sUiTndj66fljrUBxqPtiMrOQjh5pb1mcf0uW1\n6fqa7JuRLeO65GXd7IPEoeb92rNnjz70oQ/pxRdfVFtbm2bMmFHTTiUA1CvyEQD6R0YCCLuaBu+v\nfe1rftUBAE4jHwGgf2QkgLAb9Ge8AQAAAABA/xi8AQAAAADwEYM3AAAAAAA+YvAGAAAAAMBHNZ1c\nDXUi3yTNekJPrM2oSU22qwmtprz0xKze7Z/QByAImkxcTzz7HekDf6GmvO1qwqtJTXrC/EKS9OcF\n8hEIjHyTRi98QpLksTatqWTkvHO88n49nMDgHUalqLRxlmaVOm1XEmrRkjRrY3m7U1G7xQCQJEUV\n1azcdGmj7UrCLaqoZqn8yqRX8ixXA6CqFFXjb2bZriL0KhkZ/SX56BIONQcAAAAAwEcM3mHU0C1d\nc6NubPiyutVtu5rQ6m6QbrymfKMPQDB0ewd148jby+uywXY14dWtbt3Y+8c0kI9AYDR0q/NjN6rz\nY6xNmyoZWfgEfXCJZ4wxw3mH6XRabW1tkjoktQ7nXaMimZEyzZKkrs4XlVLKckGD0OruYyiTlJoz\n5e0XO7vc7oOcboWktKQ2dXR0qNXiP6SSj2PHdigScfcX+vxDm2yXMGAZL6vmaWdKkrpSUipruaDB\nKJVsVzBgGWXU7LVIksaf2KVIzu183LVzWHe1hlw6nVbbiBFWM7KSj7t3283pwXK49LI++5Aur03X\n12TfjGwZ1yUv62YfJKmry3YFg2NMWsYc3T4k73gDAAAAAOAjBm8AAAAAAHzE4A0AAAAAgI8YvAEA\nAAAA8BGDNwAAAAAAPmLwBgAAAADARzHbBcCCfJM091E9uiarJjXZria0mvLSo3N7t39AH4AgaDJx\nPbrt69Kll6kpb7ua8GpSkx41P5EkfbBAPgKBkW/SyMWPSpI81qY1lYxcsNAr79fDCQzeYVSKSuvm\nam6x03YloRYtSXPXlbc7FbVbDABJUlRRzc3MktbZriTcoopqruZKkrySZ7cYAIeUomp6fK7tKkKv\nkpGxx8hHl1gbvD//eSmRsHXvg3fVJ+rggf7eebYrQK+f/MR2BYNn5O6aSEtqs11EH5mM5Ln761T0\n1Om2Sxi04gsv2C4BvTZutF3B4Lmcj1Kw6l+/XkombVcxcEH6XQ7Uno3GdgmDVg99kKQvfMF2BYP3\nsY+73Yta9iF5xzuEDsakWz8q6cTd+ugD49VQ5KP+NlT7IGmsd1Ax02C3IAAysYPSX9+qm5Md+mj2\ng2oQ69KGgzqoW3WrJOm9+hh9AALiYEz6euJmSdKHch9lbVpSycifTZPe+buPKlqiDy5g8A6h7kbp\nypslaauWPTRODUXbFYXToT5Idz3QrViR0ASsa+yWuelKXSlpWe5iNfCCmBXd6taV3lWSpPm6lJ17\nICC6G6VPt14pSbo4t4y1aUm3unWVd6X0Dmn275cxeDuCtzoBAAAAAPARgzcAAAAAAD5i8AYAAAAA\nwEcM3gAAAAAA+IjBGwAAAAAAHzF4AwAAAADgIy4nFkLxgvTAAkkzTlG82+2L1rus2gdJhY/F7RYD\noKwQl3fhA/rBnX9U3DTaria04orrAfOD6jaAYIgXpP/3ygPlbdamNXHF9QPzgH70IylWpA+uYPAO\noVhRWrBGUm6k7VJCrdoHSd//KEsRCAKvGJPWLNCCwou2Swm1mGJaoPIrk3vEC8RAUMSK0nndC2yX\nEXqVjHx+p+1KUAsONQcAAAAAwEe8zRZCB2PSty6RdPKLuuSRMWoo8vqLDdU+SGrxDipmGuwWBEAm\ndlC65Fu6I3FAl+QuUoNYlzYc1EF9S9+SJJ2rD9IHICAOxqS7mu6QJL0/fwlr05JKRv78ROnMZy5R\ntEQfXMDgHULdjdKld0jS01q8brQaipYLCqlDfZDueqBbsSKhCVjX2C3z9Ut1qaTF+QvVwAtiVnSr\nW5d6l0mSntUSdu6BgOhulD7Zdqkk6cL8YtamJd3q1mXepdI86fTnFjN4O4K3OgEAAAAA8BGDNwAA\nAAAAPmLwBgAAAADARwzeAAAAAAD4iMEbAAAAAAAfMXgDAAAAAOAjLicWQvGC9N3FkqZPU7zbs11O\naFX7ICn6objdYgCUFeLyLv6u7v6/ryhuGm1XE1pxxfVdc3d1G0AwxAvSrQe+W95mbVoTV1x3m+/q\nkUekWJE+uGJQ73ivWrVKnufpk5/85BCVg+EQK0qL75EWrxutWImDHmyp9uEeKWp4DazekI9u8oox\nefcs1uL8hYrx2rQ1McW0uPcPfag/5KO7YkXpPYXFek+BtWlTJSNPf24x+5AOGfDUtWHDBt16662a\nMWPGUNYDAM4jHwHgyMhHAGE1oMG7q6tLl1xyiW677Ta94Q1vGOqa4LOeqLR6kbR6zl71REq2ywmt\nah8WSUWvx3Y5GCLko9tMtEdm0WqtbvqBesS6tKVHPVrd+4c+1A/y0X09Uen++GrdH2dt2lTJyCeP\nW80+pEMGNHgvX75cCxYs0Lnnnvu631soFJROpw+7wa5CXFqyWlpy3WYVGo3tckKr2ofV0sFIwXY5\nGCLko+PiBZm7l2jJGz6mgtdtu5rQKqigJd7FWuJdrILIx3pBPrqvEJc+OmKJPjpiCWvTooIKuthb\notvOW6KeKH1wRc0fCrjrrrv0y1/+Uhs2bDiq71+1apU+97nP1VwYALiGfASAIyMfAYRdTe9479q1\nS3/zN3+jb37zm2pqajqqn1mxYoU6Ojqqt127dg2oUAAIMvIRAI6MfASAGt/xfvLJJ7V3716dfvrp\n1b8rFotav369brrpJhUKBUWj0cN+Jh6PKx7nNPcA6hv5CABHRj4CQI2D97ve9S499dRTh/3dpZde\nqpNOOkn/8A//8KrQBICwIB8B4MjIRwCocfBuaWnR9OnTD/u7VCqlN77xja/6ewAIE/IRAI6MfASA\nQVzHGwAAAAAAvL6az2r+p9auXTsEZWA4NXZLty+TdPJJajzo2S4ntKp9kBS7qNFqLfAH+eig7kZ5\nl92ur//7ATWaBtvVhFajGnW7+Xp1G/WHfHRTY7f0pY7by9usTWsa1aivm9u1dq0UK9EHVwx68IZ7\nGnqkZXdKmjfOdimhVu2DpO//OTv4QBB4PQ3Sncu07N9etF1KqDWoQcu0TJK0R7xADARFQ4/0gfwy\n22WEXiUju/9guxLUgkPNAQAAAADwEYN3CPVEpQcvkB48c796IiXb5YRWtQ8XSEWvx3Y5ACSZaI/M\nBQ/qwfgj6hHr0pYe9ejB3j/0AQiOnqj0cOODeriRtWlTJSOfOvZB9iEdwqHmIVSISwsflKSn1HXB\n2YrlbVcUTof6IN31QEHRIssRsC5ekPnBQi2U1PXSM4oZ1qUNBRW00LtQkvSsuhRjdwUIhEJc+tAb\nFkqSnt3D2rSloIIu9BZK86Uvf61L0R764ALe8QYAAAAAwEcM3gAAAAAA+IjBGwAAAAAAHzF4AwAA\nAADgIwZvAAAAAAB8xOANAAAAAICPOPd8CDV2Szctl3TiCWo86NkuJ7SqfZAUe3ej3WIAlHU3yrvy\nJv3Xv3Wo0TTYria0GtWom8x/VbcBBENjt3R9+qbyNmvTmkY16r/MTfrZz6RYiT64gsE7hBp6pOVf\nkTRvou1SQq3aB0nfP58dfCAIvJ4G6ZblWv6ZF22XEmoNatBylV+Z3CNeIAaCoqFHuiy33HYZoVfJ\nyIbNtitBLTjUHAAAAAAAHzF4h1AxIq2dI6099RUVI8Z2OaFV7cMcqaii7XIASDKRosyctVrb+HPW\npUVFFbW29w99AIKjGJF+1rBWP2tgbdpUycjfj1urkkcfXMGh5iGUb5LmrZWkX6vrgrOVykctVxRO\nh/og3fVAXtFiymo9ACQ15WV+Mk/zJHW99IxSJmm7olDKK6953jmSpGfVpZTIRyAI8k3S+4+ZJ0l6\ndg9r05a88jrHmye9R/ry17oU76EPLuAdbwAAAAAAfMTgDQAAAACAjxi8AQAAAADwkbXPeH/2s5Ln\n8BVCrnrkEdslDFwkJ+nC8vaDD0oufz4n6vDn071DJ7ZbsMDpLpS5vCYyGem977VdRdU//7OUSNiu\nYuA+9nF3wz1TkporX9xwg9Tt8KX+br7ZdgVD4tlnpUTJdhWDM2a02ycy9RSc+i+4QGpttV3FILj8\nXCkdtg/p8tqspzV58snu9kGS+2uihn1I3vEGAAAAAMBHDN4AAAAAAPiIy4mFUIOJ6Qtb/1p605vU\nIIcPo3Rcw0HpC3/fu/15+gAEQcNB6QvXSpr9NjUUeW3algY16Avm85KkmCEfgaBoMDFdvu0Lklib\nNlUyctt2jz44hME7hBpNg67debH0pnNslxJqjQc9XfvF8rb5fKPdYgBIkhoPqrwuLz/Ndimh1qhG\nXatrJUmPGXfPGQDUm0bToL98/lrbZYReJSMfe558dAkv5wMAAAAA4CPe8Q6hoor6ZctWSS16q96q\nqBw+M7jDihGjX761vH2aivQBCIBiROV1OWmv3rpzpKKG16dtKKqoX+qXvdunk49AQBRV1O+aN0iS\nTuxiH9KWSkb+rtmjDw5h8A6hfKRbZ5xxpSSpy3Qq5f6FrJyUb5LOeKK83Wny9AEIgHyTdMYGSfqe\nuj75EaW6GbxtyCuvM7wzJUk/jnQpUSIfgSDIR7r10becIUn68c9Zm7ZUM/It9MEl7FEAAAAAAOAj\nBm8AAAAAAHzE4A0AAAAAgI8YvAEAAAAA8BGDNwAAAAAAPmLwBgAAAADAR1xOLIQaTEzXPfchacoU\nNajBdjmh1XBQuu5zvdv/RB+AIGg4KF23UtKsmWoo8tq0LQ1q0HXmnyRJMUM+AkHRYGK6dOd1klib\nNlUycucujz44hME7hBpNg1ZuWypNOcd2KaHWeNDTyt7B2/xTo91iAEiSGg+qvC4vn2W7lFBrVKNW\naqUk6THj2S0GQFWjadBlO1faLiP0Khn52E7y0SW8nA8AAAAAgI8YvEOopJI2p7ZrszarpJLtckKr\n5Bltnlq+0QcgGEqetHmqtHncH1XyjO1yQqukkjb3/iEfgeAoqaRtyc3almRt2lTJSPrgFg41D6Fc\npKDpb/uIJKnLdCqllOWKwimXkKZvKm93mhx9AAIgl5Cmb5aku9X1yY8o1c1n52zIKafp3imSpB9H\nupQokY9AEOQiBX34rdMlST/+OWvTlmpGvpU+uIR3vAEAAAAA8FFNg/fKlSvled5ht7Fjx/pVGwA4\ng3wEgP6RkQDCruZDzadNm6ZHHnmk+nU0Gh3SggDAVeQjAPSPjAQQZjUP3rFYjFcoAeAIyEcA6B8Z\nCSDMav6M99atWzV+/HhNmTJFH/jAB/Tcc8+95vcXCgWl0+nDbgBQj8hHAOhfLRlJPgKoNzUN3mee\neaa+8Y1v6KGHHtJtt92ml156SWeddZZefvnlfn9m1apVamtrq97a29sHXTQABA35CAD9qzUjyUcA\n9aamQ83nz59f3T7llFM0e/ZsHX/88brzzjt19dVXH/FnVqxYcdh/S6fThKdlDSama3Yslo49Vg3i\nUjm2NByUrvli7/bf0QfXkY/1oeGgdM2Nkt5yqhqKXPjDlgY16Brzd5KkmCEf60GtGUk+BlODiekD\nu6+RxNq0qZKRu5/36INDBnUd71QqpVNOOUVbt27t93vi8bji8fhg7gZDrNE06MZnPiYde47tUkKt\n8aCnG/++vG3+rtFuMRhy5KObGg+qvC4vP8t2KaHWqEbdqBslSY8Zz3I18MPrZST5GEyNpkHLt99o\nu4zQq2TkY9vJR5cM6uX8QqGg3/3udxo3btxQ1QMAdYF8BID+kZEAwqamwfuaa67RunXrtG3bNv3i\nF7/QokWLlE6ntXTpUr/qgw9KKml700varu0qqWS7nNAqeUbbJ5Vv9MF95GN9KHnS9knS9mPSKnnG\ndjmhVVJJ23v/kI/1gYysDyWV9GJ8u16MszZtqmQkfXBLTYea7969W3/xF3+h/fv3a9SoUXrb296m\nxx9/XJMmTfKrPvggFyloyts/KEnqMp1KKWW5onDKJaQp28rbnSZHHxxHPtaHXEKasl2SvqWuT35E\nqW4+O2dDTjlN8Y6TJP040qVEiXx0HRlZH3KRgpbMmiJJ+vHPWZu2VDNyFn1wSU2D91133eVXHQDg\nNPIRAPpHRgIIO07ZCgAAAACAjxi8AQAAAADwEYM3AAAAAAA+YvAGAAAAAMBHDN4AAAAAAPioprOa\noz7ETFRX7H6PNGGCYjwErIn1SFd8pXf7cvoABEGsR7riZknTpylW4rVpW2KK6QpzuSQpashHIChi\nJqo/f+EKSaxNmyoZ+eKLHn1wCJ0Kobhp1M2//4Q04RzbpYRavNvTzVeWt83lcbvFAJAkxbtVXpeX\nn227lFCLK66bdbMk6THjWa4GQEXcNOrq5262XUboVTLysefIR5fwcj4AAAAAAD5i8A4hI6N9DQe0\nT/tkZGyXE1pGRvtGlm/0AQgGI2nfSGlfc451aZGR0b7eP/QBCA4jo1di+/RKjLVpUyUj6YNbGLxD\nKBvJa/TZizTaG6OssrbLCa1sUhq9t3yjD0AwZJPS6H3S6C/coWxjj+1yQiurrEZ7YzTaG6N8hHwE\ngiIbyes9bxut97xtNGvTokpG0ge3MHgDAAAAAOAjBm8AAAAAAHzE4A0AAAAAgI8YvAEAAAAA8BGD\nNwAAAAAAPmLwBgAAAADARzHbBWD4xUxUS184Xxo3TjEeAtbEeqSld/RuL6UPQBBU1+Wb36xYidem\nbYkppqVmqSQpashHIChiJqo/28PatK2SkXv20geX0KkQiptG3fG7v5fGnWO7lFCLd3u647Lytlka\nt1sMAElSvFu641JJl5OPNsUV1x26XZL0mPEsVwOgIm4a9Zmtd9guI/QqGfnYVvLRJdYG74MHbd3z\n0PDOfZftEgbNZHO2Sxg8Y2xXMCTyedsVDF7S6TWRtl3AYU4+WUqlbFcxCI88YruCwTv3XNsVDN7N\nN9uuABhy+bzU2Gi7ioFz+7my7Kc/tV0BKnJ1sCsf/TO314QxR78PyTveoWSkZFYZ5ZRUUp54tcwG\nIymbLG97MvQBCAAjo2wkLyWlZFasSkuMjLLK9m6nyEcgMIxykfLabCqxD2lLJSNzEY8+OIQPsIVR\nMitlmtWcGFXdscHwyyal5kz5Rh+AYMhG8mqed2F5XSZtVxNeWWXV7LWo2WtRPkI+AoGRzOr8s5p1\n/lnNrE2LKhn53nfRB5cweAMAAAAA4CMGbwAAAAAAfMTgDQAAAACAjxi8AQAAAADwEYM3AAAAAAA+\nYvAGAAAAAMBHXMc7jIpRafUiLbqoqKiitqsJrWhRWrS6d3shfQCCIKqoFu05W1q/XtGi7WrCK6qo\nFplFkqSIIR+BwChGNXc/a9O2Skbu2Sv25R3C4B1GhSZpyWqtzuZsVxJqTQVp9ZLydi7bZLcYAJKk\nplKjVm/6J2nJubZLCbUmNWm1vitJesx4lqsBUFVo0r88vdp2FaFXyciHf0s+uoRDzQEAAAAA8BGD\nNwAAAAAAPmLwDqNkRjKevERSGWVsVxNamaTkmfKNPgDBkInk5L3r3PK6TNquJrwyysjzIvK8iHIR\n8hEIjGRG73yHp3e+w2NtWlTJyPPPow8uYfAGAAAAAMBHDN4AAAAAAPiIwRsAAAAAAB8xeAMAAAAA\n4CMGbwAAAAAAfMTgDQAAAACAj2K2C4AFxaj04AW64M+Kiipqu5rQihalCx7s3T6HPgBBEFVUF+w/\nQ/rFE4oWbVcTXlFFdYG5QJIUMeQjEBjFqN72R9ambZWM3L9f7Ms7pOZ3vJ9//nl98IMf1Bvf+EYl\nk0m95S1v0ZNPPulHbfBLoUla+KAe7L5XTWqyXU1oNRWkBxeWb/ShPpCP7msqNerB31xfXpcF29WE\nV5Oa9KAe0IN6QHFDPtYLMrIOFJp045YHdeOWB1mbFlUy8l9//aAaS/TBFTW94/3KK6/o7W9/u+bN\nm6cf/vCHGj16tJ599lmNGDHCp/IAwA3kIwD0j4wEEHY1Dd6f//zn1d7erttvv736d5MnT37NnykU\nCioUDr1tkE6na6sQABxAPgJA/2rNSPIRQL2p6VDz+++/XzNnztTixYs1evRonXbaabrtttte82dW\nrVqltra26q29vX1QBWMIJDNSV0qpppHKKGO7mtDKJKVUV/lGH9xHPtaHTCSn1NyF5XWZtF1NeGWU\nUUrNSqlZuQj5WA9qzUjyMaCSGZ03O6XzZqdYmxZVMvLCc+iDS2oavJ977jndcsstOuGEE/TQQw/p\n4x//uD7xiU/oG9/4Rr8/s2LFCnV0dFRvu3btGnTRGAKprLJe1nYVoZdNlW9wH/lYP7LRPOsyALIe\nz1P1pNaMJB+DKx/NKh9lbdqW9bIq0Aen1HSoealU0syZM3X99ddLkk477TRt3rxZt9xyiz784Q8f\n8Wfi8bji8fjgKwWAACMfAaB/tWYk+Qig3tT0jve4ceM0derUw/7u5JNP1s6dO4e0KABwDfkIAP0j\nIwGEXU2D99vf/nb9/ve/P+zv/vCHP2jSpElDWhQAuIZ8BID+kZEAwq6mwftv//Zv9fjjj+v666/X\nM888o29/+9u69dZbtXz5cr/qAwAnkI8A0D8yEkDY1TR4z5o1S/fee6++853vaPr06fqXf/kXfelL\nX9Ill1ziV30A4ATyEQD6R0YCCLuaTq4mSQsXLtTChQv9qAXDpRSR1s7RnHeWFKnttRcMoUhJmrO2\nd/tM+lAPyEf3RRTRnFdmSL/5rSIl29WEV0QRzTFzqtuoD2RkHShF9JYO1qZtlYx85RX64JKaB2/U\ngXxCmrdWa7M525WEWiIvrZ1X3s5lE3aLASBJSpTiWvvLf5fOPdd2KaGWUEJr9agk6bGSZ7kaAFX5\nhP7rqbW2qwi9SkY+/CT56BJeIgEAAAAAwEcM3gAAAAAA+IjBO4ySGWnvKI1qOlYZZWxXE1qZpDRq\nb/lGH4BgyERyGvXO95fXZdJ2NeGVUUajNFqjNFq5CPkIBEYyo4VnjtLCM0exNi2qZOTiOfTBJXzG\nO6xG7dd+2zVA+0f1bvBxeyAw9jd2SKNe//vgr/0ez1JAEHU0sDaDYL+3X2q0XQVqwTveAAAAAAD4\niMEbAAAAAAAfMXgDAAAAAOAjBm8AAAAAAHzE4A0AAAAAgI84q3kYlSLShpmaeXpJEV57sSZSkmZu\n6N2eTh+AIIgoopnpE6Xf/0GRku1qwiuiiGaamdVtAAFRiuikTtambZWMTKfpg0sYvMMon5DO2KAN\nWa5hZVMiL204o7ydyybsFgNAkpQoxbVhw1ekc8+1XUqoJZTQBj0hSXqs5FmuBkBVPqHbfrPBdhWh\nV8nIh58gH13CSyQAAAAAAPiIwRsAAAAAAB8xeIdRIittm6zJ8ZOUVdZ2NaGVTUiTt5Vv9AEIhmwk\nr8lnXVJel3wCxJqsspqsKZqsKcpHyEcgMBJZLZ45WYtnTmZtWlTJyA+9gz64hM94h5FnpMk7tEOS\nkbFdTWgZT9oxuXc7Rx+AIDAy2pHYI00ur1HYYWS0w9tR3QYQEJ7RS02sTduqGZmgDy7hHW8AAAAA\nAHzEO94DZOTuWyEZSc2VL0aOEkc52zdqpJzvg8trIi2pzXYRfeRyUsThl0Wjf/Yu2yUMmElmpE7b\nVaCvXE6S45d2czkfpWDVP3as7QoGJ0i/y4Houw/p8tp0vQ9963/ve+T0PqTxHN7hkZQ25qj3Id3+\nlwIAAAAAEHAM3gAAAAAA+IjBGwAAAAAAH/EZ7xDyjDR186Ft2NG3D1s4fTIQDMaTNk/VVG0hHy3y\n5GmqmVrdBhAMnpEmdbE2batk5JYt4hIcDmHwDqFkTto83XYV6NsHT0m7xQCQJHm5pLwZm7W5xI6M\nTUkltVmbJEkP0wsgMJI56bb/b7PtMkIvqaQ2abMi7M87hUPNAQAAAADwEYM3AAAAAAA+YvAOoWxC\nmrapfMsmbFcTXn37oITDF2AE6ohJZFX67TTy0bKsspqm6Zqm6cpHyEcgKLIJ6a9nT9Nfz57G2rQo\nq6yma5q0aRr7kA7hM94hZDxpy7RD27Cjbx84ixMQEJ6Rpm0R56uxy8hoi7elug0gGIwn7WhmbdpW\nzchpYh/SIbzjDQAAAACAjxi8AQAAAADwEYM3AAAAAAA+YvAGAAAAAMBHDN4AAAAAAPiIs5qHkGek\nSdsPbcOOvn3YwemTgWAwnrR9kiZpB/lokSdPk8yk6jaAYPCMNCbH2rStkpE7dohLcDiEwTuEkjlp\n+xTbVaBvHzwl7RYDQJLk5ZLyjt+u7SV2ZGxKKqnt2iZJepheAIGRzEn/77HttssIvaSS2qbtirA/\n7xQONQcAAAAAwEcM3gAAAAAA+IjBO4RyTdKsJ8q3XJPtasKrbx/UlLNdDgBJpimn0uOzyEfLcspp\nls7QLJ2hQoR8BIIi1yRdecYsXXnGLNamRTnldIZmSU/MYh/SIXzGO4RKEWnjrEPbsKNvHxQpWa0F\nQK9ISZq1URtFPtpUUkkbvY3VbQDBUIpIf2hjbdpWzchZYh/SITXtVkyePFme573qtnz5cr/qAwAn\nkI8A0D8yEkDY1fSO94YNG1QsFqtfb9q0Seedd54WL1485IUBgEvIRwDoHxkJIOxqGrxHjRp12Nc3\n3HCDjj/+eM2ZM2dIiwIA15CPANA/MhJA2A34M97d3d365je/qauvvlqe1/81NguFggqFQvXrdDo9\n0LsEACeQjwDQv6PJSPIRQL0Z8Klj7rvvPh04cEDLli17ze9btWqV2traqrf29vaB3iUAOIF8BID+\nHU1Gko8A6s2AB++vfe1rmj9/vsaPH/+a37dixQp1dHRUb7t27RroXWIIjdxXvsEu+lCfyEfH7RvJ\nugyAkWakRpqRtsuAD44mI8nH4GrrHqm2btambSPNSGkffXDJgA4137Fjhx555BH9z//8z+t+bzwe\nVzweH8jdwCeprLRvtO0q0LcPnlJ2i8GQIR/d5mVT8sbu075S/x8RgP9SSmmf9kqSHqYXdeVoM5J8\nDKZUVlq9jlcmbUsppb3apwj7804Z0Dvet99+u0aPHq0FCxYMdT0A4DTyEQD6R0YCCKuaB+9SqaTb\nb79dS5cuVSw24HOzAUDdIR8BoH9kJIAwq3nwfuSRR7Rz505ddtllftSDYZBrkuY+Wr7lmmxXE159\n+6CmnO1yMATIR/eZppxK/zuXfLQsp5zmap7map4KEfKxXpCR7ss1SdecPlfXnD6XtWlRTjnN01zp\n0bnsQzqk5pcbzz//fBlj/KgFw6QUkdbNPbQNO/r2QZGSxUowVMjHOhApSXPXaZ3IR5tKKmmdt06S\ndLXIx3pBRrqvFJF+e0x5bZZYm9ZUM3Ku2Id0CLsVAAAAAAD4iMEbAAAAAAAfMXgDAAAAAOAjBm8A\nAAAAAHzE4A0AAAAAgI+4iGJIJTO2K4B0qA9Zu2UA6CuTVJJVaV3SJG2XAOAI4kXWZhAkTVJZnqqc\nwuAdQqmslGm2XQX69sFTym4xACRJXjYlrzWjTMmzXUqopZRSRl2SpIfpBRAYqaz0g5/w7o1tKaXU\npYwi7M87hUPNAQAAAADwEYM3AAAAAAA+YvAOoXxcWvBA+ZaP264mvPr2QfG87XIASDLxvEo/WEA+\nWpZXXgu0UAu0UN0R8hEIinxc+uxbFuizb1nA2rQor7wWaoH0wAL2IR3CZ7xDqBiV1iw4tA07+vZB\n0aLVWgD0ihalC9ZojchHm4oqao23RpL0MZGPQFAUo9ITo8prs8jatKaakQvEPqRDeMcbAAAAAAAf\nMXgDAAAAAOAjBm8AAAAAAHzE4A0AAAAAgI8YvAEAAAAA8NGwn9XcGNO7lR7uux5SLlefMar+A9JG\nnJPSkr59kEnL9U64vCaqbajmkx2V+89mXf5tSpZ/jYNiTKZ+8jHt7uMoo4zklbezmbRKJac74XIr\nJEnp3n+AzYxk/zEY+u67uLw2XV+TfTPS9X3ItMs7DaptH9Izw5yiu3fvVnt7+3DeJQAclV27dmni\nxInW7p98BBBkNjOSfAQQZEeTj8M+eJdKJb3wwgtqaWmR53mv/wMBlE6n1d7erl27dqm1tdV2OaFF\nH4KhHvpgjFFnZ6fGjx+vSMTeJ3DIRwwV+hAc9dCLIGQk+YihRC+CoR76UEs+Dvuh5pFIxOo7SkOp\ntbXV2QdJPaEPweB6H9ra2myXQD5iyNGH4HC9F7YzknyEH+hFMLjeh6PNR06uBgAAAACAjxi8AQAA\nAADwEYP3AMTjcV133XWKx+O2Swk1+hAM9AF98XgIBvoQHPQCFTwWgoNeBEPY+jDsJ1cDAAAAACBM\neMcbAAAAAAAfMXgDAAAAAOAjBm8AAAAAAHzE4A0AAAAAgI8YvAEAAAAA8BGDNwAAAAAAPmLwBgAA\nAADARwzeAAAAAAD46P8H8IffEnpa9TcAAAAASUVORK5CYII=\n" } } ], "source": [ "fig,ax = plt.subplots(1,3,figsize=(10,10))\n", "ax[0].matshow(mat1,cmap='bwr',vmin=-1,vmax=1);\n", "ax[0].axvline(x=1.5,linestyle=\"dashed\",color=\"lime\")\n", "ax[0].axvline(x=3.5,linestyle=\"dashed\",color=\"lime\")\n", "ax[0].set_xticks(ticks= [0.5,2.5,4.5],labels=[r'${\\bf g}_t$',r'${\\bf i}_t$',r'${\\bf g}_t \\odot {\\bf i}_t$']);\n", "ax[1].matshow(mat2,cmap='bwr',vmin=-1,vmax=1);\n", "ax[1].axvline(x=1.5,linestyle=\"dashed\",color=\"lime\")\n", "ax[1].axvline(x=3.5,linestyle=\"dashed\",color=\"lime\")\n", "ax[1].set_xticks(ticks= [0.5,2.5,4.5],labels=[r'${\\bf c}_{t-1}$',r'${\\bf f}_t$',r'${\\bf c}_{t-1} \\odot {\\bf f}_t$']);\n", "ax[2].matshow(mat3,cmap='bwr',vmin=-1,vmax=1);\n", "ax[2].axvline(x=1.5,linestyle=\"dashed\",color=\"lime\")\n", "ax[2].axvline(x=3.5,linestyle=\"dashed\",color=\"lime\")\n", "ax[2].set_xticks(ticks= [0.5,2.5,4.5],labels=[r'${\\bf g}_t \\odot {\\bf i}_t$',r'${\\bf c}_{t-1} \\odot {\\bf f}_t$',r'${\\bf c}_t$']);\n", "fig.tight_layout()" ], "id": "20c4d4a6-2449-4540-a458-7993f1f58bbc" }, { "cell_type": "markdown", "metadata": {}, "source": [ "`-` ${\\boldsymbol g}_t$ 특징: 보통 -1,1 중 하나의 값을 가지도록 학습되어\n", "있다. (마치 RNN의 hidden node처럼!)\n", "\n", "- $\\boldsymbol{g}_t = \\tanh({\\boldsymbol x}_t {\\bf W}_{ig} + {\\boldsymbol h}_{t-1} {\\bf W}_{hg}+ {\\boldsymbol b}_{ig}+{\\boldsymbol b}_{hg})$\n", "\n", "`-` ${\\boldsymbol c}_t$ 특징: ${\\boldsymbol g}_t$와 매우 비슷하지만 약간\n", "다른값을 가진다. 그래서 ${\\boldsymbol g}_t$와는 달리 -1,1 이외의 값도\n", "종종 등장.\n", "\n", "- ${\\boldsymbol c}_t$의 값은 이론상 제한이 없음. (꼭 -1,1 사이에 있지\n", " 않음)\n", "\n", "## C. 시각화2: ${\\boldsymbol g}_t \\to {\\boldsymbol h}_{t}$" ], "id": "a774b4f0-47a6-4c5c-b73a-59755b4db16a" }, { "cell_type": "code", "execution_count": 44, "metadata": { "tags": [] }, "outputs": [], "source": [ "mat = torch.concat([g[:8],tanh(c[:8]),o[:8],h[:8]],axis=1).data" ], "id": "42001553-4640-4a69-a8ec-69b8f0e0846a" }, { "cell_type": "code", "execution_count": 46, "metadata": { "tags": [] }, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAGnCAYAAACO4pEQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90\nbGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9h\nAAAPYQGoP6dpAAApDklEQVR4nO3dfXxU5Z338e+ZSTJ5JApJUCASlFYhAaUGLNVKaAULiLou4Fpr\nQe3eWxstLqttsQ8ibY1V19UW4S6uBV1bH7DSF7egrboGpBYNFlSCWlAegvIQKGRgkkySmev+gzpt\nIKAT5jpnJvN5v17zMjOcyfl5zcl855xzze84xhgjAAAs8XldAACgZyNoAABWETQAAKsIGgCAVQQN\nAMAqggYAYBVBAwCwiqABAFhF0AAArCJoAABWETRIOv/xH/+hyZMnH3eZffv2qaSkRFu3bu32eqZM\nmaL77ruv288H4lVWVibHcTRnzhyvS3EVQYNuu/nmm3X55Zcn/PeuX79eZ5999nGXqamp0eTJk1VW\nVtbt9fzoRz/ST3/6UwWDwW7/DgCfjKBBt9XV1WnUqFEJ/71vvvnmcYOmpaVFDz/8sL7xjW+c0HqG\nDx+usrIy/frXvz6h3wPg+AiaLuzfv19XXnmlcnNzddppp2nBggWqqqqS4ziqqqryujzPtbe3Kysr\nS6+++qq+//3vy3EcnXfeeZKkuXPnatiwYcrLy1Pfvn11ww03qL29XZK0efNmOY6j5cuX68tf/rJy\nc3N15pln6rXXXov97oaGBu3bt08+n0/jxo3rcpnnnntOGRkZGj16dOwxY4zmz5+vESNGKCcnRwUF\nBRo1apTWr19/3P+XSy+9VI8//ngCRyd1RCIR3XvvvRo6dKgCgYAKCws1fvx4rV692uvSery2tjbN\nmjVLRUVFKikp0cyZM9XR0eF1WfYYHOWKK64wkowkc+aZZ5q8vDyTl5dnJJkxY8Z4XZ7nIpGIee21\n14wks379erNz506zf/9+E41GzY9+9CPzxz/+0WzdutWsWLHCFBUVmfnz5xtjjHn66aeN4zhm7Nix\n5n//93/NX/7yF3PRRReZqqqq2O9etmxZbJxfeumlLpeZOXOm+cpXvtKpphtvvDH2mvXp08eUl5eb\nrKwss3Tp0uP+v6xYscIEAgHT2tqauAFKEddff31szAYPHmx69+5tJJmMjAxTW1vrdXk90sCBA40k\nk5OTY3r37m369+8few0WLlzodXnWEDRH2Lx5c+yFv+WWW4wxxrzzzjsmIyPjqKB5//33zbJlyzyq\n1FtLly41ffr0+cTlrrrqKvPtb3/bGGPMD3/4Q3PSSSeZPXv2xP593rx5pry8PHZ/7ty55uSTTza7\nd+8+5jKXXXaZue6662L3t2zZYhzHMZLMFVdcYcLhsDHGmD179pjt27fHluvq9XrzzTeNJLN169ZP\n+7/eI7z//vuxMZs5c6YxxpgDBw7E3ggvvPDC2HLpuo3b8PH4Dho0yBw4cMC0tLSYfv36GUnmyiuv\njC3X08adQ2dHqK+vj/08bdo0SdJZZ52l4cOHH7Xsc889p3fffde12pLJunXrjjqPsm3bNt14442q\nqKjQySefrPz8fD311FMaMGCApMPnXiZPnqzi4uLYcz744AMNHjw4dn/9+vW67LLLVFJScsxlWlpa\nlJ2dHbtfV1cn87fr982aNUtZWVmSpOLiYpWWlsaW6+r1ysnJkSQ1Nzd3byBS1Nq1a2Nj9tWvflWS\nVFhYqIkTJ8b+XUrvbdymSy+9VIWFhcrOztagQYMkSbt37479e08bd4LmOBzHif1sjrgQ6cqVK/WD\nH/xADz30kEaMGKGWlha3y/PUkTPD9u7dq1GjRmnv3r267777tHr1av3pT3+S3+/XOeecI+lw0Pzj\neRXpcGB9/O8f/95PWqaoqEj79++Pq95jvV5//etfJalT+KWbf9zO/1G6b+M2nXTSSbGfMzIyJP39\nPaYnjjtBc4SKiorYz88884wk6d1339Xbb7/dabkxY8aooqJCL730ktatWxf7ZJwu3n777U57eStW\nrFBHR4cef/xxjR8/XuXl5Vq1apXa2tp0zjnnqKmpSdu2bdOIESM6/Z7169fHQuTgwYPasmXLcZeR\npBEjRmjjxo2x+yNHjoy9Wd5///1qa2uTdPi7Njt27JB07Ndrw4YNGjBggIqKihIzMCni3HPPjY3Z\nx7PumpqatGLFCklSZWVl2m/jXumJ407QHOH000/XFVdcIenwdzWGDBmiyspKZWZmHrXsjh07Oh2a\nSSfRaFRvvfWWPvroIzU1Nal3794KBoNatmyZNm3apPvuu09z5sxR//79VVxcrDfffFN+v7/TXtC2\nbdu0f//+WIisX79ePp9Pw4YNO+YyknTxxRervr4+tldTVlam6upqSdLTTz+t/v37a9iwYerfv3/s\nEJDU9ev1yiuvaPz48YkenqR3xhln6LrrrpMkPfDAA/rMZz6j008/Xdu2bVNGRobuuOMOSem9jXup\np407QdOF//7v/9bUqVOVk5OjgwcP6q677lJ5ebmkvx/T37Fjh/r37+9lmZ76yU9+oieffFL9+/fX\n3LlzNWnSJF1//fW65pprdMEFF+jDDz/UtGnTOh02O+usszp9Olu3bp1OOumk2JcuP17mH8+/HLmM\nJA0bNkyVlZV66qmnYo/9/Oc/14MPPqhzzjlHhw4d0pYtW2Lfk5G6fr1aW1u1dOlS/eu//mtiBydF\n/PKXv9Tdd9+tIUOGaPv27Wpvb9dFF12kl19+WVVVVWm/jXulR467t3MRktP27dtNS0tL7P7mzZtN\ndna2kWS+973vGWOMWbVqlZk6dapXJaa95cuXmyFDhphIJPKplu/q9Zo3b54ZN26cjfJ6BLZxb/TE\ncWePpgu//e1vNWDAAF188cWaMGGCzj77bLW2tqpv37666aabJB0+l7Np0yYNGzasR80OSRUTJ07U\nv/3bv+nDDz/8VMt39XplZmbqF7/4hc0yUxrbuDd64rg7xhwxnQp66aWXdNttt+ndd99Vc3OzTjnl\nFI0bN05z5szRaaed5nV5AJBSCBoAgFUcOgMAWEXQAACsImgAAFYRNAAAqwgaAIBVBM2nFA6HNWfO\nHIXDYa9LSRuMufsYc/elw5gzvflTCgaDKiwsVFNTk3r16uV1OWmBMXcfY+6+dBhz9mgAAFYRNAAA\nqzLcXmE0GtVHH32kgoKCY15wKRkFg8FO/4V9jLn7GHP3pfKYG2N08OBB9evXTz7fsfdbXD9H09Ou\nswAA6a6hoSF2yfauuL5HU1BQIEnKymqQ46TWia897x/0uoS47HX26Yz8szs9ttm8ryKlztUknUOp\nNeaSFFKz+hV8VpL00e51yjO5HlcUpzPP9LqCuIVypH67Dv/8oflIecrztqA4pdp2HntvCUoq/fv7\n+rG4HjQfHy5znF4pFzS9eqXOoT5JCjttUn7nxwpMgXopdcbd8aXWmEuSX37pb393vZoLUi9oUpDf\nkT7erHuZXqkXNCm2nR/53vJJp0GYDAAAsIqgAQBYRdAAAKwiaAAAVrk+GQDo6TKUoentX5XaO5Rh\n/F6XkxYyOqTpiyVNn64M3taSDq8IkGABBbS49f9Khw55XUraCLRJi6+VzPTFXpeCLnDoDABgFUED\nJJiRUUghhZxmGdEc3Q1GUihXCinEmCchggZIsGY1K7/gVOWfMljNTovX5aSF5lwpPyQVOPlqVrPX\n5eAIBA0AwCqCBgBgFUEDALCKoAEAWEXQAACsImgAAFbRGQBIML/8mtJ+udTRIb/hs5wb/BFpyhJJ\nU6Ycvh4QkgpBAyRYtrK1pPVRWtC4KDssLZkmmegSr0tBF7r1cWv+/PkaNGiQsrOzde655+qVV15J\ndF0AgB4i7qB58skndfPNN+v73/++1q1bpy9+8YuaMGGCtm/fbqM+AECKizto7rvvPl1//fX6xje+\noSFDhuj+++9XaWmpFixY0OXy4XBYwWCw0w3oyUIKySnoJefUfgo5tENxQyhXcozkcxyFFPK6HBwh\nrqBpa2vTG2+8ofHjx3d6fPz48Xr11Ve7fE5NTY0KCwtjt9LS0u5XCwBIOXEFzd69exWJRNS3b99O\nj/ft21e7du3q8jmzZ89WU1NT7NbQ0ND9agEAKadbs84cx+l03xhz1GMfCwQCCgQC3VkNAKAHiGuP\npqioSH6//6i9lz179hy1lwMAgBRn0GRlZencc8/VCy+80OnxF154QV/4whcSWhgAoGeI+9DZrFmz\ndM0116iyslKjR4/WwoULtX37dn3zm9+0UR8AIMXFHTRXXnml9u3bp7lz52rnzp2qqKjQihUrNHDg\nQBv1ASnHL78mdoyXOiK0oHGJPyJNXC5p4kRa0CQhxxjj6gW2g8GgCgsLFQg0yXF6ubnqE9ay56DX\nJcSl0dmrkvzTOz222+xRsYo9qih+zqHUGvNOUrUFTb9+XlfQbSbq6ttZwqTadh57bwlKKpSamprU\nq9ex38/5uAUAsIqgAQBYRdAACRZSSHn5pyiv7xm0oHFJKFfKOyTlK48WNEmIywQAFjQ7zVLX32GG\nJc15ktQspeZpmh6NPRoAgFUEDQDAKoIGAGAVQQMAsIqgAQBYxawzIMF88mlMxwVSJCKfYeqZG3xR\naUytpDFj5OPzc9IhaIAEy1GOaltWpG4LmhSU0yrVjpVMtNbrUtAFoh8AYBVBAwCwiqABEiykkIrz\nBqm4pIIWNC4J5UrFe6QSFdOCJglxjgawYK9vn9clpJ29xZK0lxY0ScizoAmHvVpz9zm9CrwuIT5F\nrVJj54caGyV3r0B0YqZMSbExlxTJ9kkfX+38jMESOzWu2rNHyvO6iDhNm5Za23lbYau07NMvz6Ez\nAIBVBA0AwCqCBgBgFUEDALCKoAESLepT/juVqqw73BoF9vmiUmWddHZ7JS1okhDTm4EE87fl6Jz/\nU6dXVtPnzC05rVLdKGn3rjqvS0EXiH4AgFUEDQDAKoIGSLBIoFl1T5WpbIvUnON1NemhOUcq2yJV\nFpWpmW/IJh3O0QCJ5hiFT92mbZK4HI07jCNtK5OkbTL0oEk67NEAAKwiaAAAVhE0AACrCBoAgFUE\nDQDAKoIGSDTjKHfLUA2tlxwmQLnCMdLQeumzHUPliKl+yYbpzUCC+cO5+tzX62lB46LcFqm+Qtq9\nq97rUtAF9mgAAFYRNAAAq+IOmlWrVmny5Mnq16+fHMfR7373OwtlAakrEmjWnx8tV/kGWtC4pTlH\nKt8gXdinnBY0SSjuoAmFQjr77LM1b948G/UAqc8xah60URvLaUHjFuNIG8ulv2RspAVNEop7MsCE\nCRM0YcIEG7UAAHog67POwuGwwuFw7H4wGLS9SgBAErE+GaCmpkaFhYWxW2lpqe1VAgCSiPWgmT17\ntpqammK3hoYG26sEACQR64fOAoGAAoGA7dUAAJIU36MBEs04CuwcqIFbaUHjFsdIA7dKAyIDaUGT\nhOLeozl06JA2b94cu79lyxatX79evXv31mmnnZbQ4oBU5A/nauS0rbSgcVFui7R1kLR711avS0EX\n4g6atWvXauzYsbH7s2bNkiRNnz5dixcvTlhhAICeIe6gqaqqkjEcDwAAfDqcowESLJLVovULR2rk\n61JLttfVpIeWbGnk69LFvUeqRS1el4MjcJkAINF8UR0aslZrJUX5KOeKqE9aO1KS1iqqqNfl4Aj8\nGQAArCJoAABWETQAAKsIGgCAVQQNAMAqggawIONAkYoava4ivRQ1Sr2jRV6XgS4QNECC+Vvz9PnJ\njWoskfK4qrAr8pqlxhJpY2Oj8pTndTk4AkEDALCKoAEAWEXQAAkWyWrRWz+vUtXLtKBxS0u2VPWy\n9E8nV9GCJgnRggZINF9UwRErtVK0oHFL1CetrJKklbSgSUL8GQAArCJoAABWETQAAKs4RxMHk2LX\nIm+UVHLEY1dfLWUe8KCYbqrLHeN1CXEL+SLK97qINDZliuRv9bqK+KTadt6otqPeW46HPRoAgFXs\n0QAW5Lb4pCizn9yUG5Jafblel4EusEcDJFheq1+hr1ygUD4taNyS1yyF8qXPXRiSv5UWNMmGoAEA\nWEXQAACsImiABGvNimrSXRs06VmpNeB1NemhNSBNelba9F+TFM1KsSlnaYDJAECCRXxGK0b/9fDP\nfo+LSRMRv7RikiStkPFFvC4HR2CPBgBgFUEDALCKoAEAWEXQAACsImgAAFYRNAAAqwgaIMHyWv0y\nYy6UcWhB45a8Zsk4UuVIQwuaJETQAACsImgAAFYRNECCtWZFNfWOjZr6FC1o3NIakKY+Jb1fM5UW\nNEmIoAESLOIzerpqr56eSgsat0T80tNTpf0XPU0LmiRE0AAArIoraGpqajRy5EgVFBSopKREl19+\nud577z1btQEAeoC4gmblypWqrq7WmjVr9MILL6ijo0Pjx49XKBSyVR8AIMXFdZmA559/vtP9RYsW\nqaSkRG+88YYuvPDCLp8TDocVDodj94PBYDfKBACkqhM6R9PU1CRJ6t279zGXqampUWFhYexWWlp6\nIqsEAKSYbgeNMUazZs3SBRdcoIqKimMuN3v2bDU1NcVuDQ0N3V0lACAFdfsKmzfeeKPeeustrV69\n+rjLBQIBBQJ8mQDpI7fVp0MXny+t/qNyaUHjitxm6VCe9MXPHZKvNdfrcnCEbgXNTTfdpGXLlmnV\nqlUaMGBAomsCUpojR3mtfomQcY2jw/3O6HOWnOIKGmOMbrrpJi1dulS1tbUaNGiQrboAAD1EXOdo\nqqur9dhjj+k3v/mNCgoKtGvXLu3atUstLS226gNSTjgzqhnfe08zFknhLK+rSQ/hLGnGImnL7TMU\nzQx/8hPgqriCZsGCBWpqalJVVZVOPfXU2O3JJ5+0VR+Qcjr8Ro9M2K1HZkgd3T4Linh0ZEiPzJD2\nXfKIjL/D63JwhLgPnQEAEA96nQEArCJoAABWETQAAKsIGgCAVQQNAMAqggZIsNxWn/Zc+nntKRYt\naFyS2yztKZbOHreHFjRJiFn+QII5clTclCXt9bqS9OFIKt4rZR4o9roUdIE9GgCAVQQNkGDhzKiq\nb96k6nm0oHFLOEuqnidt+041LWiSEEEDJFiH32j+P+3U/Gpa0LilI0OaXy01Tp1PC5okRNAAAKwi\naAAAVhE0AACrCBoAgFUEDQDAKs/mxBQUSI7j1dq76Y9ve11BfPx/lTSm00N9+khZqTQT6p4Hva4g\nfk6zpPMO/3zooKQUu469L7U/f65alXIjLtWn2HbexXvL8aT2FgUkoRyTrS3vPa8t5gPlKMfrctJC\nTou0pUz6wGxhzJNQKn22BVKCTz6VtfeXVOZ1KWnDZ6SybZJhzJMSezQAAKsIGiDB2px23dr3P3Wr\nblWb2rwuJy20ZUq33i3GPEkRNECCtatd9xYv1r3Of6pd7V6XkxbaM6V7b5X+07mXMU9CBA0AwCqC\nBgBgFUEDALCKoAEAWEXQAACsImgAAFbRGQBIsByTrQ2blkqDB9MOxSU5LdKGcsls2MCYJyGCBkgw\nn3wqDw+WVO51KWnDZ6TyjZJhzJMSh84AAFYRNECCtTntmlMyX3M0h3YoLmnLlObcLsY8SRE0QIK1\nq113lCzQHc5c2qG4pD1TumOONNe5gzFPQgQNAMAqggYAYBVBAwCwKq6gWbBggYYPH65evXqpV69e\nGj16tJ577jlbtQEAeoC4gmbAgAG66667tHbtWq1du1Zf+tKXdNlll6m+vt5WfQCAFBfXFzYnT57c\n6f5Pf/pTLViwQGvWrFF5OV+UAgAcrdudASKRiJYsWaJQKKTRo0cfc7lwOKxwOBy7HwwGu7tKICVk\nm4Bef/9x6fTTla1sr8tJC9mt0usjJfP664x5Eoo7aN5++22NHj1ara2tys/P19KlSzV06NBjLl9T\nU6M77rjjhIoEUolffo1sqRAtaNzjj0oj10pGI70uBV2Ie9bZmWeeqfXr12vNmjW64YYbNH36dG3c\nuPGYy8+ePVtNTU2xW0NDwwkVDABILXHv0WRlZWnw4MGSpMrKStXV1emBBx7QL3/5yy6XDwQCCgQC\nJ1YlkELanHY90OcxSX01UzOVpSyvS+rx2jKlB2ZKRvcw5knohL9HY4zpdA4GSHftatd3TrlP33G+\nSzsUl7RnSt+5R/qu8x3GPAnFtUdz2223acKECSotLdXBgwf1xBNPqLa2Vs8//7yt+gAAKS6uoNm9\ne7euueYa7dy5U4WFhRo+fLief/55jRs3zlZ9AIAUF1fQPPzww7bqAAD0UPQ6AwBYRdAAAKwiaAAA\nVnW7BQ2ArmWbgF7e8iuprIx2KC7JbpVerpLMyy8z5kmIoAESzC+/qkIjRQsa9/ijUtVKyajK61LQ\nBQ6dAQCsImiABGtXux7s/bge1IN8S90l7RnSg98SY56kCBogwdqcdt3Y707d6NykNrV5XU5aaMuS\nbnxQusm5kTFPQgQNAMAqggYAYBVBAwCwiqABAFhF0AAArCJoAABW0RkASLCAydKzWx+UBp6mgLiM\nuRsCYenZSZJ59lnGPAkRNECCZShDkw5dKFrQuCcjIk1aIRlN8roUdMGzoDl40Ks1d58zrMLrEuJT\n1Cg1dn4oHJZM2JtyuuOLN6TYmP+DV1Y7XpeQdsaP97qC7mlrS63tvK2wUVr26ZfnHA2QYFF/u3ZP\nWKzF0w+3RoF97RnS4unSR+MXK+qnBU2yIWiABDOZbdp027W6dvHh1iiwry1LunaxtPHWa2UyaUGT\nbAgaAIBVBA0AwCqCBgBgFUEDALCKoAEAWEXQAACsImiABPO1B3TWD5/SU1MPt0aBfYGw9NRUadiP\nn5LTRguaZMPXyYAEcyIZKqqdqqmrva4kfWREpKlPSwsPTPW6FHSBPRoAgFUEDZBgxt+hvVVLtGSK\n1OH3upr00OGXlkyRdl+4RFFfh9fl4AgcOgMSLJoZ1rs/nqZpkg7lSRnNXlfU84UD0rQlkjRNYycf\nklp5a0sm7NEAAKwiaAAAVhE0AACrCBoAgFUEDQDAKoIGAGDVCQVNTU2NHMfRzTffnKBygNTntGfp\nM3cu0qIZUhYXe3RFVpu0aIY09J5Fctq5rGmy6XbQ1NXVaeHChRo+fHgi6wFSni+Sqb7PzdCMR6RM\nvjvoiswOacYjUr8/zJAvkul1OThCt4Lm0KFDuvrqq/XQQw/p5JNPPu6y4XBYwWCw0w0AkD66FTTV\n1dWaNGmSLrrook9ctqamRoWFhbFbaWlpd1YJpAzj79BfRy/X8om0oHFLh19aPlHaO2o5LWiSUNx9\nGp544gn9+c9/Vl1d3adafvbs2Zo1a1bsfjAYJGzQo0Uzw9p49yW6RLSgcUs4IF2yXJIuoQVNEorr\n1WhoaNDMmTP1hz/8QdnZ2Z/qOYFAQIEA14cAgHQVV9C88cYb2rNnj84999zYY5FIRKtWrdK8efMU\nDofl93OsAADwd3EFzZe//GW9/fbbnR679tprddZZZ+m73/0uIQMAOEpcQVNQUKCKiopOj+Xl5alP\nnz5HPQ4AgERnAACAZSc8NaO2tjYBZQAAeir2aIAEc9qzdPp98zSvmhY0bslqk+ZVS2f+Yh4taJIQ\nQQMkmC+SqX5Lq1U9nxY0bsnskKrnS6XLqmlBk4QIGgCAVQQNkGDGF9GBc2pVO0aK8BfmiohPqh0j\n/XV4rYwv4nU5OAJ9GoAEi2a1asMvxmqsDregyaMFjXWt2dLYWkkaq7GTD8nfmudxRfhHfN4CAFhF\n0AAArCJoAABWETQAAKsIGgCAVQQNAMAqggZIMKcjU2Xz79bdt0qZ7V5Xkx4y26W7b5UGL7xbTged\nAZINQQMkmK8jSwMev1W33itlETSuyGqXbr1XKltyq3wd9DpLNgQNAMAqggZIMOOL6OBZdaqrpAWN\nWyI+qa5SavpsHS1okpBnLWh+9jMpJ8ertXfPTd92vC4hLo2SSo54bOpUqVfYi2q659pffdHrEuIW\nyooo/6E/aZRoQeOW1mxpVJ0kjdKC/zmkQEdqtaBJte280bQf9d5yPHzeAgBYRdAAAKwiaAAAVhE0\nAACrCBoAgFUEDQDAKq6wCSRYZoej239VKm1voAWNSzLbpdvnSOsuu13+KC1okg1BAyRYVodPcxYN\nlFY3eF1K2shql+bcIS0aOMfrUtAFDp0BAKwiaIAEizpG9WUh1Q+VoqnVTCJlRR2pfqj04Un1iirq\ndTk4AofOgARrCURV8T/rJNGCxi0tOVJFvSRVpGQLmp6OPRoAgFUEDQDAKoIGAGAVQQMAsIqgAQBY\nRdAAAKxiejOQYJkdjm55vL+040Na0Lgks1265R5pw1duoQVNEiJogATL6vDpnvmDpNUfel1K2shq\nl+75jrSo6B6vS0EX4jp0NmfOHDmO0+l2yimn2KoNANADxL1HU15erhdffDF23+/3J7QgINVFHaPt\nfcPSQOm07ZLPeF1Rzxd1pO2nSXvzt6r3odPk4/RzUok7aDIyMtiLAY6jJRDVoCVrJdGCxi0tOdKg\nrZI0iBY0SSju2N+0aZP69eunQYMG6V/+5V/0wQcfHHf5cDisYDDY6QYASB9xBc15552nRx99VL//\n/e/10EMPadeuXfrCF76gffv2HfM5NTU1KiwsjN1KS0tPuGgAQOqIK2gmTJigf/7nf9awYcN00UUX\nafny5ZKkRx555JjPmT17tpqammK3hgYuBgUA6eSEpjfn5eVp2LBh2rRp0zGXCQQCCgQCJ7IaAEAK\nO6GpGeFwWO+8845OPfXURNUDAOhh4gqaW265RStXrtSWLVv02muvacqUKQoGg5o+fbqt+gAAKS6u\nQ2c7duzQVVddpb1796q4uFif//zntWbNGg0cONBWfUDKyYg4+tYzp0o7dyqjw+tq0kNGh/StB6V3\nvvQt+aI0PEk2cb0iTzzxhK06gB4j0O7Tg/91hrR6p9elpI1Am/TgjdKiXz3odSnoAl+fBQBYRdAA\nCWZk1HhSuxqLJLrPuMNIaiySgoFGGUY96XAwE0iw5uyoSv7fa5JoQeOW5lyppFGSSmhBk4TYowEA\nWEXQAACsImgAAFYRNAAAqwgaAIBVBA0AwCqmNwMJlhFxNP25Emn3HlrQuCSjQ5q+WNp8/nRa0CQh\nXhEgwQLtPi2+87PS6j1el5I2Am3S4mulRb9a7HUp6AKHzgAAVhE0QIIZGYWyIwrl0oLGLUZSKFcK\nZ4RoQZOECBogwZqzo8p/4U/KDx1ujQL7mnOl/JB0wzX5asug50+yIWgAAFYRNAAAqzybdfaDH0iO\n49Xau+emF1/0uoT4ZB6QNKXTQ6efLp2cSlNu5871uoL4+VokTTr886GDklKsk7AvtT9/fu1rKTfi\nUlmKbeeZByRd8akXT+0tCgCQ9AgaAIBVBA0AwCo6AwAJ5pdfU/ZcKBWXyC+/1+WkBX9EmrJE0pQp\njHkSImiABMuOZmlJ/RypqsrrUtJGdlhaMk0y0SVel4IucOgMAGAVQQMAsIqgARIs5GuRM/ZLchyf\nQgp5XU5aCOVKjpF8jsOYJyGCBgBgFUEDALCKoAEAWEXQAACsImgAAFYRNAAAq+gMACSYX35N3Hee\n1LsP7VBc4o9IE5dLmjiRMU9CBA2QYNnRLC1/q4YWNC7KDkvLL5FMdLnXpaALHDoDAFhF0AAArCJo\ngAQL+VqUd+FE5SmfdiguCeVKeYekfOUx5kko7qD58MMP9bWvfU19+vRRbm6uzjnnHL3xxhs2agNS\nVrO/Vc1Os9dlpJXmPDHmSSquyQD79+/X+eefr7Fjx+q5555TSUmJ3n//fZ100kmWygMApLq4guZn\nP/uZSktLtWjRothjZWVlx31OOBxWOByO3Q8Gg/FVCABIaXEdOlu2bJkqKys1depUlZSUaMSIEXro\noYeO+5yamhoVFhbGbqWlpSdUMAAgtcQVNB988IEWLFigz3zmM/r973+vb37zm/r2t7+tRx999JjP\nmT17tpqammK3hoaGEy4aAJA64jp0Fo1GVVlZqTvvvFOSNGLECNXX12vBggX6+te/3uVzAoGAAoHA\niVcKAEhJce3RnHrqqRo6dGinx4YMGaLt27cntCgglfnk05j9Z2uMGSMf3yBwhS8qjakVY56k4tqj\nOf/88/Xee+91euwvf/mLBg4cmNCigFSWEw2odv1/0YLGRTmtUu1YyURrvS4FXYgr+v/93/9da9as\n0Z133qnNmzfrN7/5jRYuXKjq6mpb9QEAUlxcQTNy5EgtXbpUjz/+uCoqKvTjH/9Y999/v66++mpb\n9QEAUlzc3ZsvueQSXXLJJTZqAXqEkK9FZaO/KilTW7VFecrzuqQeL5QrlW2VpGJt0VbGPMlwmQDA\ngr1ZTYd/MN7WkU72FkvSXsY8CTE9AwBgFUEDALCKoAEAWEXQAACsImgAAFYx6wxIMJ98qgyeKRUU\n0A7FJb6oVFknqbKSMU9CBA2QYDnRgOreWEALGhfltEp1oyQTrfO6FHSB6AcAWEXQAACsImiABGv2\ntars81epTIPUrGavy0kLzTlS2RZpkMoY8yTEORogwYyMtuXsPvyzoR+KG4wjbSuTpG2MeRJijwYA\nYBVBAwCwiqABAFjl2Tma9nav1tx9zkVf9rqE+BQ1So2dH2prk8IpNPZfnDPW6xLiFskOSWP+die/\nQJybdldtrZQT9bqK+Mydm1rbeVtho3TBp1+ePRoAgFXMOgMSzTjK3TJUZc0b5TAByhWOkYbWS6GB\nQ+XI8bocHIGgARLMH87V575er1dW84bnltwWqb5CeunFeq9LQRc4dAYAsIqgAQBYRdAACRYJNOvP\nj5arfMPh1iiwrzlHKt8gXX9euVp9TPNLNpyjARLNMWoetFEbdbg1CuwzjrSxXJI2yogZGMmGPRoA\ngFUEDQDAKoIGAGAVQQMAsIqgAQBYxawzINGMo8DOgTolvI0WNC5xjDRwq9TadyAtaJIQezRAgvnD\nuRo5bau2DjrcGgX25bZIWwdJv351q7KjuV6XgyMQNAAAqwgaAIBVBA2QYJGsFq1fOFIjX5dasr2u\nJj20ZEsjX5e+NXKkwj6OVyYbJgMAieaL6tCQtVorKcpHOVdEfdLakZK0VlGl2OU10wB/BgAAq+IK\nmrKyMjmOc9SturraVn0AgBQX16Gzuro6RSKR2P0NGzZo3Lhxmjp1asILAwD0DHEFTXFxcaf7d911\nl8444wyNGTMmoUUBAHqObk8GaGtr02OPPaZZs2bJcY79TdxwOKxwOBy7HwwGu7tKAEAK6vZkgN/9\n7nc6cOCAZsyYcdzlampqVFhYGLuVlpZ2d5VAysg4UKSiRq+rSC9FjVJhW5HXZaAL3Q6ahx9+WBMm\nTFC/fv2Ou9zs2bPV1NQUuzU0NHR3lUBK8Lfm6fOTG9VYIuVxVWFX5DVLjSXSb19pVE40z+tycIRu\nHTrbtm2bXnzxRT3zzDOfuGwgEFAgEOjOagAAPUC39mgWLVqkkpISTZo0KdH1AAB6mLiDJhqNatGi\nRZo+fboyMmgsABwpktWit35epaqXaUHjlpZsqepladbnqmhBk4TiTooXX3xR27dv13XXXWejHiD1\n+aIKjliplaIFjVuiPmlllSStpAVNEoo7aMaPHy9juJoTAODT4fMWAMAqggYAYBVBAwCwiqABAFjF\n/GTAAl9LrrKjtAVwU25Iimbnel0GusAeDZBg/tY8fWF8SKF8WtC4Ja9ZCuVLz9aGaEGThAgaAIBV\nBA0AwCqCBkiwaFar6u+epEnPSq30k3VFa0Ca9Kx029mT1OZr9bocHIHJAECCGV9E+0ev0ApJEb/X\n1aSHiF9aMUmSViiiyCctDpexRwMAsIqgAQBYRdAAAKwiaAAAVhE0AACrXJ919vdr2QTdXnX6iR48\napibQweV2Z46c247OryuIH6RjlBs3INGzIFyQcgoNubNoaCi0dQa9VTbzjva//be8rcx/6RrlDnG\n5auY7dixQ6WlpW6uEgBgUUNDgwYMGHDMf3c9aKLRqD766CMVFBTIcRw3V31CgsGgSktL1dDQoF69\nenldTlpgzN3HmLsvlcfcGKODBw+qX79+8vmOfSbG9UNnPp/vuMmX7Hr16pVyG0OqY8zdx5i7L1XH\nvLCw8BOXYTIAAMAqggYAYBVB8ykFAgHdfvvtCgRSZ8ZWqmPM3ceYuy8dxtz1yQAAgPTCHg0AwCqC\nBgBgFUEDALCKoAEAWEXQAACsImgAAFYRNAAAqwgaAIBV/x8dx0MRAjecygAAAABJRU5ErkJggg==\n" } } ], "source": [ "plt.matshow(mat.data,cmap='bwr',vmin=-1,vmax=1)\n", "plt.xticks([0.5,2.5,4.5,6.5],[r'${\\bf g}_t$',r'$tanh({\\bf c}_t)$',r'${\\bf o}_t$',r'${\\bf h}_t$']);\n", "plt.axvline(x=1.5,color=\"lime\",linewidth=3)\n", "plt.axvline(x=3.5,linestyle=\"dashed\",color=\"lime\")\n", "plt.axvline(x=5.5,linestyle=\"dashed\",color=\"lime\")\n", "plt.axvline(x=7.5,color=\"lime\",linewidth=3)" ], "id": "2dc8a94f-ecfc-46ec-9874-e6ce310b78a0" }, { "cell_type": "markdown", "metadata": {}, "source": [ "`-` ${\\boldsymbol h}_t$ 특징: (1) ${\\boldsymbol c}_t$에서 원하는 것만\n", "선택적으로 특징으로 삼은 느낌. (2) $c_t$보다 훨씬 값을 다양하게 가진다.\n", "($\\odot$ 의 효과 )\n", "\n", "## D. LSTM의 알고리즘 리뷰 I (수식위주)\n", "\n", "**(step1)** calculate ${\\tt ifgo}$\n", "\n", "${\\tt ifgo} = {\\boldsymbol x}_t \\big[{\\bf W}_{ii} | {\\bf W}_{if}| {\\bf W}_{ig} |{\\bf W}_{io}\\big] + {\\boldsymbol h}_{t-1} \\big[ {\\bf W}_{hi}|{\\bf W}_{hf} |{\\bf W}_{hg} | {\\bf W}_{ho} \\big] + bias$\n", "\n", "$=\\big[{\\boldsymbol x}_t{\\bf W}_{ii} + {\\boldsymbol h}_{t-1}{\\bf W}_{hi} ~\\big|~ {\\boldsymbol x}_t{\\bf W}_{if}+ {\\boldsymbol h}_{t-1}{\\bf W}_{hf}~ \\big|~ {\\boldsymbol x}_t{\\bf W}_{ig} + {\\boldsymbol h}_{t-1}{\\bf W}_{hg} ~\\big|~ {\\boldsymbol x}_t{\\bf W}_{io} + {\\boldsymbol h}_{t-1}{\\bf W}_{ho} \\big] + bias$\n", "\n", "참고: 위의 수식은 아래코드에 해당하는 부분\n", "\n", "``` python\n", "ifgo = Xt @ lstm_cell.weight_ih.T +\\\n", " ht @ lstm_cell.weight_hh.T +\\\n", " lstm_cell.bias_ih + lstm_cell.bias_hh\n", "```\n", "\n", "**(step2)** decompose ${\\tt ifgo}$ and get ${\\boldsymbol i}_t$,\n", "${\\boldsymbol f}_t$, ${\\boldsymbol g}_t$, ${\\boldsymbol o}_t$\n", "\n", "${\\boldsymbol i}_t = \\sigma({\\boldsymbol x}_t {\\bf W}_{ii} + {\\boldsymbol h}_{t-1} {\\bf W}_{hi} +bias )$\n", "\n", "${\\boldsymbol f}_t = \\sigma({\\boldsymbol x}_t {\\bf W}_{if} + {\\boldsymbol h}_{t-1} {\\bf W}_{hf} +bias )$\n", "\n", "${\\boldsymbol g}_t = \\tanh({\\boldsymbol x}_t {\\bf W}_{ig} + {\\boldsymbol h}_{t-1} {\\bf W}_{hg} +bias )$\n", "\n", "${\\boldsymbol o}_t = \\sigma({\\boldsymbol x}_t {\\bf W}_{io} + {\\boldsymbol h}_{t-1} {\\bf W}_{ho} +bias )$\n", "\n", "**(step3)** calculate ${\\boldsymbol c}_t$ and ${\\boldsymbol h}_t$\n", "\n", "${\\boldsymbol c}_t = {\\boldsymbol i}_t \\odot {\\boldsymbol g}_t+ {\\boldsymbol f}_t \\odot {\\boldsymbol c}_{t-1}$\n", "\n", "${\\boldsymbol h}_t = {\\boldsymbol o}_t \\odot \\tanh({\\boldsymbol c}_t)$\n", "\n", "## E. LSTM의 알고리즘 리뷰 II (느낌위주)\n", "\n", "- 이해 및 암기를 돕기위해서 비유적으로 설명한 챕터입니다..\n", "\n", "`-` 느낌: RNN이 콩물에서 간장을 한번에 숙성시키는 방법이라면 LSTM은\n", "콩물에서 간장을 단계를 나누어 숙성하는 느낌이다.\n", "\n", "- RNN:\n", " ${\\boldsymbol x}_t \\overset{{\\boldsymbol h}_{t-1}}{\\longrightarrow} {\\boldsymbol h}_t$\n", "- LSTM:\n", " ${\\boldsymbol x}_t \\overset{{\\boldsymbol h}_{t-1}}{\\longrightarrow} {\\boldsymbol g}_t \\overset{{\\boldsymbol c}_{t-1}}{\\longrightarrow} \\Big({\\boldsymbol c}_t \\to {\\boldsymbol h}_t \\Big)$\n", "\n", "`-` ${\\boldsymbol g}_t$에 대하여\n", "\n", "- 과거와 현재의 결합 (선형변환): ${\\boldsymbol x}_t$와\n", " ${\\boldsymbol h}_{t-1}$를 ${\\bf W}_{ig}, {\\bf W}_{hg}$를 이용해\n", " 선형결합\n", "- 숙성(비선형변환): $\\tanh$\n", "- 느낌: RNN에서 간장을 만들던 그 수식에서 $h_t$를 $g_t$로 바꾼것\n", " 그래서 RNN의 간장과 비슷하다고 생각하면 된다.\n", "- 노트: RNN의 간장은 한계를 가지고 있는데 그 한계를 극복하기 위해 만든\n", " 것이 ${\\boldsymbol c}_t$\n", "\n", "`-` ${\\boldsymbol c}_t$에 대하여\n", "\n", "- 과거와 현재의 결합 (선형변환): ${\\boldsymbol g}_{t}$와\n", " ${\\boldsymbol c}_{t-1}$를 요소별로 선택하고 더하는 과정\n", "- 숙성 (비선형변환): 없음.\n", "- 느낌: 과거와 현재의 정보중 유리한것만 기억하여 선택적으로 결합함.\n", " 이때 결합방식에 대한 노하우는 ${\\tt input-gate}$,\n", " ${\\tt forget-gate}$ 에 있으며 그러한 결합의 결과가\n", " ${\\boldsymbol c}_t$에 있음. 이 ${\\boldsymbol c}_t$에 대한 정보는\n", " 그대로 “냉동보관(?)”되어 다음세대로 내려옴.\n", "- 비고: ${\\boldsymbol c}_t$는 사실상 LSTM 알고리즘의 꽃이라 할 수\n", " 있음. LSTM은 long short term memory의 약자임. 기존의 RNN은\n", " 장기기억을 활용함에 약점이 있는데 LSTM은 단기기억/장기기억 모두 잘\n", " 활용함. LSTM이 장기기억을 잘 활용하는 비법은 바로\n", " ${\\boldsymbol c}_t$에 있다.\n", "\n", "`-` ${\\boldsymbol h}_t$에 대하여\n", "\n", "- 과거와 현재의 결합 (선형변환): 없음\n", "- 숙성 (비선형변환): $\\tanh({\\boldsymbol c}_t)$를 요소별로 선택하여\n", " 숙성\n", "\n", "> RNN은 기억할 과거정보가 ${\\boldsymbol h}_{t-1}$ 하나이지만 LSTM은\n", "> ${\\boldsymbol c}_{t-1}$, ${\\boldsymbol h}_{t-1}$ 2개이다.\n", "\n", "## F. LSTM이 강한이유\n", "\n", "`-` 답변1: LSTM이 장기기억에 유리함. 그 이유는 input, forget, output\n", "gate 들이 장기기억을 위한 역할을 하기 때문.\n", "\n", "- 비판: 아키텍처에 대한 이론적 근거는 없음. 장기기억을 위하여 꼭\n", " LSTM같은 구조일 필요는 없음. (게이트는 꼭3개이어야 하는지?)\n", "\n", "`-` 답변2: 저는 사실 아까 살펴본 아래의 이유로 이해하고 있습니다.\n", "\n", "- 실험적으로 살펴보니 LSTM이 RNN보다 장기기억에 유리했음.\n", "- 그 이유: RNN은 ${\\boldsymbol h}_t$의 값이 -1 혹은 1로 결정되는\n", " 경우가 많았음. 그러나 경우에 따라서는 ${\\boldsymbol h}_t$이 -1~1의\n", " 값을 가지는 것이 문맥적 뉘앙스를 포착하기에는 유리한데 LSTM이 이러한\n", " 방식으로 학습되는 경우가 많았음.\n", "- 왜 LSTM의 ${\\boldsymbol h}_t$은 -1,1 이외의 값을 쉽게 가질 수\n", " 있는가? $\\odot$ 때문에.. 즉 게이트때문에..\n", "\n", "# 6. Ref\n", "\n", "`-` 참고자료들\n", "\n", "- \n", "- \n", "- " ], "id": "ede7aafe-0b33-4394-b67b-b73b00981517" } ], "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3 (ipykernel)", "language": "python" }, "language_info": { "name": "python", "codemirror_mode": { "name": "ipython", "version": "3" }, "file_extension": ".py", "mimetype": "text/x-python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.8" } } }