notebooks/train_retvec_model_tf.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using RETVec to train an emotion classifier\n",
    "\n",
    "RETVec is a state-of-the-art text vectorizer which works directly on text inputs to create resilient classification models. Models trained with RETVec achieve better classification performance with fewer parameters and exhibit stronger resilience against adversarial attacks and typos, as reported in our [paper](https://arxiv.org/abs/2302.09207).\n",
    "\n",
    "RETVec's speed and fewer parameters (~200k instead of millions) makes it a great choice to train and deploy small and efficient on-device models. It is natively supported in TensorFlow Lite via custom ops implemented in TensorFlow Text, and we provide a Javascript implementation of RETVec which allows you to deploy web models via TensorFlow.js.\n",
    "\n",
    "This notebook demonstrates how to quickly train and use a text emotion classifier in TensorFlow.\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "# installing needed dependencies\n",
    "try:\n",
    "    import retvec\n",
    "except ImportError:\n",
    "    !pip install retvec  # is retvec installed?\n",
    "\n",
    "try:\n",
    "    import datasets\n",
    "except ImportError:\n",
    "    !pip install datasets  # used to get the dataset\n",
    "\n",
    "try:\n",
    "    import matplotlib\n",
    "except ImportError:\n",
    "    !pip install matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '1'  # silence TF INFO messages\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from tensorflow.keras import layers\n",
    "from datasets import load_dataset\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we are using the `RETVecTokenizer()` layer which perform the binarization and embedding in a single step. This is the best approach for GPU training. For TPU training,\n",
    "it is more efficient to split the two steps -- see our TPU training notbook for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the RETVec tokenizer layer\n",
    "from retvec.tf import RETVecTokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create dataset\n",
    "\n",
    "We are going to use the [Go Emotion dataset](https://huggingface.co/datasets/go_emotions) to create a mulit-class emotion classifier.\n",
    "https://ai.googleblog.com/2021/10/goemotions-dataset-for-fine-grained.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# downloading dataset\n",
    "dataset = load_dataset('go_emotions')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num classes 28\n",
      "['admiration', 'amusement', 'anger', 'annoyance', 'approval', 'caring', 'confusion', 'curiosity', 'desire', 'disappointment', 'disapproval', 'disgust', 'embarrassment', 'excitement', 'fear', 'gratitude', 'grief', 'joy', 'love', 'nervousness', 'optimism', 'pride', 'realization', 'relief', 'remorse', 'sadness', 'surprise', 'neutral']\n"
     ]
    }
   ],
   "source": [
    "# get class name mapping and number of class\n",
    "CLASSES = dataset['train'].features['labels'].feature.names\n",
    "NUM_CLASSES = len(CLASSES)\n",
    "print(f\"num classes {NUM_CLASSES}\")\n",
    "print(CLASSES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# preparing data\n",
    "x_train = tf.constant(dataset['train']['text'], dtype=tf.string)\n",
    "\n",
    "# the one-hot requires a little more due to the multi-class nature of the dataset.\n",
    "y_train = np.zeros((len(x_train),NUM_CLASSES))\n",
    "for idx, ex in enumerate(dataset['train']['labels']):\n",
    "    for val in ex:\n",
    "        y_train[idx][val] = 1\n",
    "\n",
    "# test data\n",
    "x_test = tf.constant(dataset['test']['text'], dtype=tf.string)\n",
    "y_test = np.zeros((len(x_test),NUM_CLASSES))\n",
    "for idx, ex in enumerate(dataset['test']['labels']):\n",
    "    for val in ex:\n",
    "        y_test[idx][val] = 1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "A key benefit of using RETVec is that inputs to RETVec and the text model are raw strings with no pre-processing neccessary, which greatly simplifies the training and inference process, especially for on-device models.\n",
    "\n",
    "Notes:\n",
    "- Using strings directly as input requires to use a shape of `(1,)` and specify the type `tf.string`\n",
    "- We are using `RETVecTokenizer()` in its default configuration which is to truncate at `128` words and use a small pre-trained word embedding model to embed the words. You can experiment with shorter or longer length by changing the `sequence_length` parameter. The word embedding model offers significant improvements in adversarial and typo robustness. To use the RETVec character tokenizer only, set `model=None`.\n",
    "- To use native TF ops only for TF Lite compatibility, set `use_tf_lite_compatible_ops=True`. see the [TF Lite notebook](tf_lite_retvec.ipynb) for more details on how to convert a RETVec-based model to a TF Lite model which can run on-device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.\n",
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " token (InputLayer)          [(None, 1)]               0         \n",
      "                                                                 \n",
      " ret_vec_tokenizer_1 (RETVe  (None, 128, 256)          230144    \n",
      " cTokenizer)                                                     \n",
      "                                                                 \n",
      " bidirectional_2 (Bidirecti  (None, 128, 128)          164352    \n",
      " onal)                                                           \n",
      "                                                                 \n",
      " bidirectional_3 (Bidirecti  (None, 128)               98816     \n",
      " onal)                                                           \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 28)                3612      \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 496924 (1.90 MB)\n",
      "Trainable params: 266780 (1.02 MB)\n",
      "Non-trainable params: 230144 (899.00 KB)\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# using strings directly requires to put a shape of (1,) and dtype tf.string\n",
    "inputs = layers.Input(shape=(1, ), name=\"token\", dtype=tf.string)\n",
    "\n",
    "# add RETVec tokenizer layer with default settings -- this is all you have to do to build a model with RETVec!\n",
    "x = RETVecTokenizer(model='retvec-v1')(inputs)\n",
    "\n",
    "# standard two layer LSTM\n",
    "x = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(x)\n",
    "x = layers.Bidirectional(layers.LSTM(64))(x)\n",
    "outputs = layers.Dense(NUM_CLASSES, activation='sigmoid')(x)\n",
    "model = tf.keras.Model(inputs, outputs)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/25\n",
      "170/170 [==============================] - 18s 79ms/step - loss: 0.1763 - acc: 0.2889 - val_loss: 0.1476 - val_acc: 0.2959\n",
      "Epoch 2/25\n",
      "170/170 [==============================] - 10s 62ms/step - loss: 0.1480 - acc: 0.2987 - val_loss: 0.1447 - val_acc: 0.3088\n",
      "Epoch 3/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.1420 - acc: 0.3298 - val_loss: 0.1359 - val_acc: 0.3532\n",
      "Epoch 4/25\n",
      "170/170 [==============================] - 11s 66ms/step - loss: 0.1350 - acc: 0.3648 - val_loss: 0.1294 - val_acc: 0.3910\n",
      "Epoch 5/25\n",
      "170/170 [==============================] - 10s 62ms/step - loss: 0.1296 - acc: 0.3930 - val_loss: 0.1248 - val_acc: 0.4111\n",
      "Epoch 6/25\n",
      "170/170 [==============================] - 11s 62ms/step - loss: 0.1251 - acc: 0.4137 - val_loss: 0.1207 - val_acc: 0.4301\n",
      "Epoch 7/25\n",
      "170/170 [==============================] - 12s 68ms/step - loss: 0.1213 - acc: 0.4323 - val_loss: 0.1181 - val_acc: 0.4450\n",
      "Epoch 8/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.1176 - acc: 0.4504 - val_loss: 0.1161 - val_acc: 0.4470\n",
      "Epoch 9/25\n",
      "170/170 [==============================] - 12s 70ms/step - loss: 0.1150 - acc: 0.4578 - val_loss: 0.1135 - val_acc: 0.4559\n",
      "Epoch 10/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.1124 - acc: 0.4670 - val_loss: 0.1105 - val_acc: 0.4691\n",
      "Epoch 11/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.1101 - acc: 0.4776 - val_loss: 0.1092 - val_acc: 0.4765\n",
      "Epoch 12/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.1082 - acc: 0.4841 - val_loss: 0.1079 - val_acc: 0.4793\n",
      "Epoch 13/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.1064 - acc: 0.4902 - val_loss: 0.1072 - val_acc: 0.4820\n",
      "Epoch 14/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.1050 - acc: 0.4949 - val_loss: 0.1057 - val_acc: 0.4855\n",
      "Epoch 15/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.1031 - acc: 0.5037 - val_loss: 0.1053 - val_acc: 0.4841\n",
      "Epoch 16/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.1020 - acc: 0.5063 - val_loss: 0.1042 - val_acc: 0.4924\n",
      "Epoch 17/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.1005 - acc: 0.5105 - val_loss: 0.1045 - val_acc: 0.4922\n",
      "Epoch 18/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.0993 - acc: 0.5153 - val_loss: 0.1019 - val_acc: 0.5038\n",
      "Epoch 19/25\n",
      "170/170 [==============================] - 12s 70ms/step - loss: 0.0978 - acc: 0.5220 - val_loss: 0.1025 - val_acc: 0.5040\n",
      "Epoch 20/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.0970 - acc: 0.5235 - val_loss: 0.1019 - val_acc: 0.5115\n",
      "Epoch 21/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.0960 - acc: 0.5286 - val_loss: 0.1013 - val_acc: 0.5089\n",
      "Epoch 22/25\n",
      "170/170 [==============================] - 12s 70ms/step - loss: 0.0947 - acc: 0.5340 - val_loss: 0.1016 - val_acc: 0.5091\n",
      "Epoch 23/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.0937 - acc: 0.5373 - val_loss: 0.1017 - val_acc: 0.5054\n",
      "Epoch 24/25\n",
      "170/170 [==============================] - 12s 68ms/step - loss: 0.0928 - acc: 0.5426 - val_loss: 0.1010 - val_acc: 0.5106\n",
      "Epoch 25/25\n",
      "170/170 [==============================] - 12s 69ms/step - loss: 0.0915 - acc: 0.5474 - val_loss: 0.0999 - val_acc: 0.5165\n"
     ]
    }
   ],
   "source": [
    "# compile and train the model like usual\n",
    "batch_size = 256\n",
    "epochs = 25\n",
    "model.compile('adam', 'binary_crossentropy', ['acc'])\n",
    "history = model.fit(x_train, y_train, epochs=epochs, batch_size=batch_size, \n",
    "                    validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjEAAAGzCAYAAADe/0a6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABjkUlEQVR4nO3dd3hUVf7H8fdk0kMqIZUAIaEKIdQIAqKgQREBAQELiK5df2p0XdEVF9FlLYusiqKuWFFs2JUVIiAqgoJ0CBBKKKlAOmkz9/fHkGAkQAaSTCb5vJ5nnszcOffmO5OR+XjuueeYDMMwEBEREXEyLo4uQERERORsKMSIiIiIU1KIEREREaekECMiIiJOSSFGREREnJJCjIiIiDglhRgRERFxSgoxIiIi4pQUYkRERMQpKcSIiIiIU1KIEZHTeumllzCZTCQkJDi6FBGRakxaO0lETueCCy7g0KFD7N27l507dxIbG+vokkREAPXEiMhp7Nmzh59//pnZs2fTqlUrFixY4OiSalRUVOToEkTEARRiROSUFixYQGBgICNGjGDcuHE1hpjc3Fzuu+8+2rVrh4eHB61bt2by5Mnk5ORUtSkpKeEf//gHHTt2xNPTk/DwcK666ipSU1MBWL58OSaTieXLl1c79t69ezGZTLz55ptV22644QZatGhBamoql19+Ob6+vlx77bUArFy5kvHjx9OmTRs8PDyIiorivvvu49ixYyfVvX37dq6++mpatWqFl5cXnTp14pFHHgFg2bJlmEwmPv3005P2e++99zCZTKxatcru91NE6parowsQkcZrwYIFXHXVVbi7uzNp0iRefvllfv31V/r27QtAYWEhgwYNYtu2bdx444306tWLnJwcvvjiCw4cOEBwcDAWi4UrrriC5ORkJk6cyD333ENBQQFLlixh8+bNxMTE2F1XRUUFiYmJDBw4kGeffRZvb28APvroI4qLi7n99ttp2bIla9as4YUXXuDAgQN89NFHVftv3LiRQYMG4ebmxi233EK7du1ITU3lyy+/5Mknn2TIkCFERUWxYMECxowZc9J7EhMTQ//+/c/hnRWROmGIiNTgt99+MwBjyZIlhmEYhtVqNVq3bm3cc889VW2mT59uAMaiRYtO2t9qtRqGYRjz5883AGP27NmnbLNs2TIDMJYtW1bt+T179hiA8cYbb1RtmzJligEYDz300EnHKy4uPmnbrFmzDJPJZOzbt69q2+DBgw1fX99q2/5Yj2EYxrRp0wwPDw8jNze3altWVpbh6upqPPbYYyf9HhFpeDqdJCI1WrBgAaGhoVx00UUAmEwmJkyYwMKFC7FYLAB88skn9OjR46Teisr2lW2Cg4O5++67T9nmbNx+++0nbfPy8qq6X1RURE5ODgMGDMAwDH7//XcAsrOz+eGHH7jxxhtp06bNKeuZPHkypaWlfPzxx1XbPvjgAyoqKrjuuuvOum4RqTsKMSJyEovFwsKFC7nooovYs2cPu3btYteuXSQkJJCZmUlycjIAqampdOvW7bTHSk1NpVOnTri61t3Za1dXV1q3bn3S9rS0NG644QaCgoJo0aIFrVq14sILLwQgLy8PgN27dwOcse7OnTvTt2/fauOAFixYwPnnn68rtEQaCY2JEZGTfP/996Snp7Nw4UIWLlx40vMLFizg0ksvrbPfd6oemcoenz/z8PDAxcXlpLaXXHIJR44c4W9/+xudO3fGx8eHgwcPcsMNN2C1Wu2ua/Lkydxzzz0cOHCA0tJSfvnlF1588UW7jyMi9UMhRkROsmDBAkJCQpg7d+5Jzy1atIhPP/2UefPmERMTw+bNm097rJiYGFavXk15eTlubm41tgkMDARsVzr90b59+2pd86ZNm9ixYwdvvfUWkydPrtq+ZMmSau3at28PcMa6ASZOnEhSUhLvv/8+x44dw83NjQkTJtS6JhGpXzqdJCLVHDt2jEWLFnHFFVcwbty4k2533XUXBQUFfPHFF4wdO5YNGzbUeCmycXwezbFjx5KTk1NjD0Zlm7Zt22I2m/nhhx+qPf/SSy/Vum6z2VztmJX3//Of/1Rr16pVKwYPHsz8+fNJS0ursZ5KwcHBXHbZZbz77rssWLCA4cOHExwcXOuaRKR+qSdGRKr54osvKCgo4Morr6zx+fPPP79q4rv33nuPjz/+mPHjx3PjjTfSu3dvjhw5whdffMG8efPo0aMHkydP5u233yYpKYk1a9YwaNAgioqKWLp0KXfccQejRo3C39+f8ePH88ILL2AymYiJieGrr74iKyur1nV37tyZmJgYHnjgAQ4ePIifnx+ffPIJR48ePant888/z8CBA+nVqxe33HIL0dHR7N27l6+//pr169dXazt58mTGjRsHwMyZM2v/RopI/XPkpVEi0viMHDnS8PT0NIqKik7Z5oYbbjDc3NyMnJwc4/Dhw8Zdd91lREZGGu7u7kbr1q2NKVOmGDk5OVXti4uLjUceecSIjo423NzcjLCwMGPcuHFGampqVZvs7Gxj7Nixhre3txEYGGjceuutxubNm2u8xNrHx6fGurZu3WoMGzbMaNGihREcHGzcfPPNxoYNG046hmEYxubNm40xY8YYAQEBhqenp9GpUyfj0UcfPemYpaWlRmBgoOHv728cO3aslu+iiDQErZ0kInIaFRUVREREMHLkSF5//XVHlyMif6AxMSIip/HZZ5+RnZ1dbbCwiDQO6okREanB6tWr2bhxIzNnziQ4OJh169Y5uiQR+RP1xIiI1ODll1/m9ttvJyQkhLffftvR5YhIDdQTIyIiIk5JPTEiIiLilBRiRERExCk1icnurFYrhw4dwtfX95xWxRUREZGGYxgGBQUFREREnLQeWm00iRBz6NAhoqKiHF2GiIiInIX9+/fXuDL9mTSJEOPr6wvY3gQ/Pz8HVyMiIiK1kZ+fT1RUVNX3uL2aRIipPIXk5+enECMiIuJkznYoiAb2ioiIiFNSiBERERGnpBAjIiIiTqlJjImpDcMwqKiowGKxOLqUZs9sNuPq6qrL4UVE5Jw0ixBTVlZGeno6xcXFji5FjvP29iY8PBx3d3dHlyIiIk6qyYcYq9XKnj17MJvNRERE4O7urh4ABzIMg7KyMrKzs9mzZw8dOnQ4qwmOREREzirEzJ07l2eeeYaMjAx69OjBCy+8QL9+/Wps++abbzJ16tRq2zw8PCgpKal6fMMNN/DWW29Va5OYmMjixYvPprxqysrKsFqtREVF4e3tfc7Hk3Pn5eWFm5sb+/bto6ysDE9PT0eXJCIiTsjuEPPBBx+QlJTEvHnzSEhIYM6cOSQmJpKSkkJISEiN+/j5+ZGSklL1uKaekOHDh/PGG29UPfbw8LC3tNPS/+03Lvp7iIjIubL7m2T27NncfPPNTJ06la5duzJv3jy8vb2ZP3/+KfcxmUyEhYVV3UJDQ09q4+HhUa1NYGCgvaWJiIhIM2JXiCkrK2Pt2rUMGzbsxAFcXBg2bBirVq065X6FhYW0bduWqKgoRo0axZYtW05qs3z5ckJCQujUqRO33347hw8fPuXxSktLyc/Pr3YTERGR5sWuEJOTk4PFYjmpJyU0NJSMjIwa9+nUqRPz58/n888/591338VqtTJgwAAOHDhQ1Wb48OG8/fbbJCcn89RTT7FixQouu+yyU14OPWvWLPz9/atuWvxRRESk+an3q5P69+9P//79qx4PGDCALl268MorrzBz5kwAJk6cWPV89+7diYuLIyYmhuXLlzN06NCTjjlt2jSSkpKqHlcuICUiIiLNh10hJjg4GLPZTGZmZrXtmZmZhIWF1eoYbm5u9OzZk127dp2yTfv27QkODmbXrl01hhgPD486H/grIiIi1RmGwZZD+XyzKZ0Wnq7cMSTW0SVVY9fpJHd3d3r37k1ycnLVNqvVSnJycrXeltOxWCxs2rSJ8PDwU7Y5cOAAhw8fPm2bc2EYBsVlFQ65GYZhV62LFy9m4MCBBAQE0LJlS6644gpSU1Ornj9w4ACTJk0iKCgIHx8f+vTpw+rVq6ue//LLL+nbty+enp4EBwczZsyYOnsfRUSk6TEMg00H8vjXt9sZ8uxyrnjhR15ansqbP+3FarXvO6y+2X06KSkpiSlTptCnTx/69evHnDlzKCoqqpoLZvLkyURGRjJr1iwAHn/8cc4//3xiY2PJzc3lmWeeYd++ffzlL38BbIN+Z8yYwdixYwkLCyM1NZUHH3yQ2NhYEhMT6/ClnnCs3ELX6f+rl2OfydbHE/F2r/3bXlRURFJSEnFxcRQWFjJ9+nTGjBnD+vXrKS4u5sILLyQyMpIvvviCsLAw1q1bh9VqBeDrr79mzJgxPPLII7z99tuUlZXxzTff1NdLExERJ2UYBpsO5vH1pnS+3ZRB2pETM9x7uLpwUacQLo8Lx2oYuNB4Joy1O8RMmDCB7Oxspk+fTkZGBvHx8SxevLhqsG9aWlq1OUCOHj3KzTffTEZGBoGBgfTu3Zuff/6Zrl27ArZ1dDZu3Mhbb71Fbm4uERERXHrppcycOVOnjICxY8dWezx//nxatWrF1q1b+fnnn8nOzubXX38lKCgIgNjYE119Tz75JBMnTmTGjBlV23r06NEwhYuISKNmGAYbDuTxzaZ0vtmUzoGjx6qe83Rz4eLOIVzePZyLOoXg49E4J/g3Gfae32iE8vPz8ff3Jy8vDz8/v2rPlZSUsGfPHqKjo6tmhjUMg2PljlkI0svNbNeyBzt37mT69OmsXr2anJwcrFYrRUVFfP3113z11Vds2bKFFStW1Livt7c3c+fOPWnG5Magpr+LiIjUL8MwWL8/93hwyeBg7ong4uVmPhFcOrey66zB2Trd93dtNM5oVc9MJlOD/HHqwsiRI2nbti2vvfYaERERWK1WunXrRllZGV5eXqfd90zPi4hI02cYBr/vz+Wbjel8u7l6cPF2twWXEd3DGdIpBC93swMrtZ9zfJM3U4cPHyYlJYXXXnuNQYMGAfDjjz9WPR8XF8d///tfjhw5UnU66Y/i4uJITk5ulD0xIiJSfyp7XL7amM63m9I5lHdivUIfdzNDu4RyefcwLuzofMHljxRiGrHAwEBatmzJq6++Snh4OGlpaTz00ENVz0+aNIl//vOfjB49mlmzZhEeHs7vv/9OREQE/fv357HHHmPo0KHExMQwceJEKioq+Oabb/jb3/7mwFclIiL1Zf+RYj79/SCf/n6QPTlFVdt93M0M6xrK5d3DubBjKzzdnDe4/JFCTCPm4uLCwoUL+b//+z+6detGp06deP755xkyZAhgu+T9u+++4/777+fyyy+noqKCrl27MnfuXACGDBnCRx99xMyZM/nXv/6Fn58fgwcPduArEhGRupZ3rJxvNqXz6bqDrNl7pGq7l5uZS88LZUT3cAY3oeDyR81yYK84nv4uIiJnr9xiZUVKNp/+fpAl2zIpq7BNrWEywcDYYMb0jCTxvLBGe1VRJQ3sFRERaQYMw2DjgTw+/f0gX2w4xJGisqrnOoX6clWvSEbFRxLm33z+x1AhRkREpBE7cLSYz9cf4pN1B9idfWKcS3ALD0bHRzCmVyRdw/3smr6jqVCIERERaWQKSsr5dlMGi34/wC+7T4xz8XRz4dKuYVzVK5KBscG4mu1aPajJUYgRERFxMMMwOJh7jLX7jrJ0Wxbfbcmg9Pg4F4D+7Vsyplckl3ULw9fTzYGVNi4KMSIiIg2srMLKlkN5rN13lHVpR1m77yiZ+aXV2sS08uGqXq0Z3TOSyABNXloThRgREZF6llNYyrp9R1mbdpR1+46y4UBe1RVFlcwuJs6L8KNfuyCujI+ge6R/sxznYg+FGBERkTpksRrszCpg7T5bD8u6fUfZe7j4pHaB3m70bhtIzzaB9G4bSFxrf6dZEqex0LslIiJyDkorLKzZc6QqtKxPy6WgtOKkdh1DW9C7bSC9joeW6GAf9bScI4UYEREROxmGwaaDeXy89gCfrz9E3rHyas/7uJuJbxNA7zaB9GobSM+oQPy9NSC3rinENGHt2rXj3nvv5d5773V0KSIiTUJWfgmfrT/Ix2sPsCOzsGp7iK8HF8QG06ttIL3aBNAp1LfZX/7cEBRiRERETqOk3ELytiw+XrufH3bmYLHaVuvxcHUh8bwwxvVuzQWxwZhddGqooSnEiIiI/EnlFP8frz3AFxuqny7q1SaAcb2jGBEXjr+XThE5UvPs6zIMKCtyzK2W622++uqrREREYLVWvwRv1KhR3HjjjaSmpjJq1ChCQ0Np0aIFffv2ZenSpWf9lsyePZvu3bvj4+NDVFQUd9xxB4WFhdXa/PTTTwwZMgRvb28CAwNJTEzk6NGjAFitVp5++mliY2Px8PCgTZs2PPnkk2ddj4iII2TmlzBvRSqXPPcDo+b+xDu/7CPvWDnh/p7ceVEMyfdfyKI7LuCahDYKMI1A8+yJKS+Gf0Y45nc/fAjcfc7YbPz48dx9990sW7aMoUOHAnDkyBEWL17MN998Q2FhIZdffjlPPvkkHh4evP3224wcOZKUlBTatGljd1kuLi48//zzREdHs3v3bu644w4efPBBXnrpJQDWr1/P0KFDufHGG/nPf/6Dq6sry5Ytw2KxADBt2jRee+01nnvuOQYOHEh6ejrbt2+3uw4RkYZWUm5h6bZMPl57gB92ZHP8bBEeri4M72Y7XTQgRqeLGiOTYdSya6ARO91S3iUlJezZs4fo6Gg8PY+v7FlW1OhDDMDo0aNp2bIlr7/+OmDrnZkxYwb79+/HxeXkTrRu3bpx2223cddddwHnNrD3448/5rbbbiMnJweAa665hrS0NH788ceT2hYUFNCqVStefPFF/vKXv9Tq+DX+XUREGohhGKzfn8sn6w7wxfpD5JecuCS6d9tAxvVuzYi4cPw0xX+9Ot33d200z54YN29bmHDU766la6+9lptvvpmXXnoJDw8PFixYwMSJE3FxcaGwsJB//OMffP3116Snp1NRUcGxY8dIS0s7q7KWLl3KrFmz2L59O/n5+VRUVFBSUkJxcTHe3t6sX7+e8ePH17jvtm3bKC0treoxEhFprFKzC/l8/SG+WH+w2gR04f6eXNUrkrG9WtO+VQsHVij2aJ4hxmSqdW+II40cORLDMPj666/p27cvK1eu5LnnngPggQceYMmSJTz77LPExsbi5eXFuHHjKCsrs/v37N27lyuuuILbb7+dJ598kqCgIH788UduuukmysrK8Pb2xsvr1Ot2nO45ERFHy8gr4csNh/h8w0E2H8yv2u7pZru6aHzvKPrHtNTpIifUPEOMk/D09OSqq65iwYIF7Nq1i06dOtGrVy/ANsj2hhtuYMyYMQAUFhayd+/es/o9a9euxWq18u9//7vqNNWHH35YrU1cXBzJycnMmDHjpP07dOiAl5cXycnJtT6dJCJSn/KKy/l2czqfrz/EL3sOV11TYXYxMbhDMKPiI7mkayg+HvoadGb66zVy1157LVdccQVbtmzhuuuuq9reoUMHFi1axMiRIzGZTDz66KMnXclUW7GxsZSXl/PCCy8wcuRIfvrpJ+bNm1etzbRp0+jevTt33HEHt912G+7u7ixbtozx48cTHBzM3/72Nx588EHc3d254IILyM7OZsuWLdx0003n9PpFRGqrcj6Xz9YfZHlKFuWWE0M++7QNZFR8BJd3D6dlCw8HVil1SSGmkbv44osJCgoiJSWFa665pmr77NmzufHGGxkwYEBViMjPzz/NkU6tR48ezJ49m6eeeopp06YxePBgZs2axeTJk6vadOzYke+++46HH36Yfv364eXlRUJCApMmTQLg0UcfxdXVlenTp3Po0CHCw8O57bbbzu3Fi4icQYXFyk+ph/l8/UH+tzmDojJL1XOdw3y5Mj6CkXERRAXVfjyiOI/meXWSOJz+LiJytgzD4Pf9uXyx/hBfbTxETuGJsYCRAV6Mio/gyvgIOofZf7WLNCxdnSQiIk1SaYWFrPxSMvJLyMgrITO/hIO5x1i6LZP9R45VtQvycWdE93BG94ygV5tArQzdjCjENAMLFizg1ltvrfG5tm3bsmXLlgauSESaM8MwOFpcXhVMKkNKVoHtZ0Z+KZn5JRwpOvXVlt7uZhLPC+PK+AgGxgbjpsUWmyWFmGbgyiuvJCEhocbn3Nw0kZOI1J/C0go++m0/a/cdrQosmfmllFXU7kIEd1cXwvw8CfPzJMTPgzA/T3pEBTCsSyhe7uZ6rl4aO4WYZsDX1xdfX19HlyEizUh2QSlv/ryHd1btqzYb7h8F+bgT6udJmJ8HYf6ex+97EurvWRVcArzddHpITqnZhJgmMH65SdHfQ6Rp2ne4iFd/2M1Haw9U9ba0D/ZhfJ8oooK8bCHleK+Kh6t6UuTcNPkQU3m6pLi4WDPLNiLFxbbpvnU6S6Rp2HQgj3krUvl2c3rVAorxUQHcdmEMl3YNxUWz4Uo9aPIhxmw2ExAQQFZWFgDe3t7qmnQgwzAoLi4mKyuLgIAAzGb9n5iIszIMgx935TBvRSo/7Tpctf2iTq247cIY+kUH6d9bqVdNPsQAhIWFAVQFGXG8gICAqr+LiDiXCouVbzZn8MqKVLYcsk2yaXYxcWWPCG69sL3mZ5EG0yxCjMlkIjw8nJCQEMrLyx1dTrPn5uamHhgRJ3SszMJHa/fz2srdVfO0eLmZmdgvipsGRtM6ULPiSsNqFiGmktls1peniIidjhaV8faqfby1am/V3C1BPu5M6d+Oyf3bEujj7uAKpV6VFcOWRWB2h7irHV1NNc0qxIiISO0dzD3Gf1fuZuGa/Rwrt61J1DrQi1sGt2d87yjN09LUZafAb2/AhvegJA8C2kK3ceDSeCYWVIgREZEqhaUVJG/L5KuN6Xy/PQvL8UuNuob7cduQGC7vFoarZsdtuirKYPuXtvCyd+WJ7QFtoPcNYCkDl8az3p1CjIhIM1dUWsH327P4emM6y1KyKP3DbLoDYlpy24UxDOoQrCuNmrKj+2Dtm/D7O1CUbdtmcoGOw6HPTRBzcaPqgamkECMi0gwVl1WwbHs2X286xPfbsygpPxFcooN9GNE9nCt6hOtKo6bMaoGd38Fv82HnEuD4BD8twqDXZOg9BfxbO7TEM1GIERFpJo6VWVieksVXm9L5fltW1TgXgLYtvRnRPZwRceF0DfdTr0tTVpAB696x9bzkHzixvf0QW69Lp8vA7BwTkSrEiIg0YSXlFpanZPP1pnSSt2VSXHYiuEQFeTGiewRXxIVzXoSCS5NmGLBnha3XZfvXYD2+npVXIMRfC31uhJYxjq3xLCjEiIg0MSXlFlbuzOHrjYdYsjWToj8El8gAL66Is/W4dI/0d57gYrXAxg/g5xegohRCz6t+C2jXKMdsOFzxEVj/Hqx9Aw7vOrE9KsEWXLqOBrfGM1DXXgoxIiJNQF5xOSt3ZZO8LYulWzMpKD2xcnSEvyeXdw/nih4R9GjtRMEFwGqFbV/Asn9CTsqJ7UdSbdsruflASJfqwSakK3gHNXzNjmapgN3LbaFv6+dgKbVtd28BcRNs4SWsm0NLrCsKMSIiTsgwDLYcymd5ShbLU7L5fX9u1eXQAGF+tuAyIi6cnlEBzrcAo2HArmT4/nFI32Db5hUIF9wL4XGQuRWytkLmZsjaDuVFcPA32+2PfCOOh5quENrNdr9lB3BtYhP0GQYcWgcbP4TNn5y4wgggtDv0vRG6jwcPX8fVWA8UYkREnERlb8vylGxW7Mgmu6C02vOxIS0Y0rEVw7uF0atNoPMFl0r7fobkmZD2s+2xewvof6ft5ulv2xZz8Yn2lgpbz0zmlhO3rC2QmwYFh2y3XUtOtHdxg+COtp4bNy/bqSrD8oefFbYeoMpt1orj961/uP+HfawWMLtCZG9oNwiiB0OLkIZ5r47sho0f2XpdjqSe2O4VBN2ugh6TbHU5U++bHUyGYRhnbta45efn4+/vT15eHn5+uhxQRJoGq9Vga/qJ3pZ1aUf5Q2cL3u5mBsQEM6RTKy7s2IqoICdfu+jQ7/D9E7Brqe2x2QP63QwD7wOfYPuPV5IHWdv+FG62Qml+3dZdk+BOtjATPcgWbOrytFZRDmxeZAsuf+x5cvWCzpfbThnFXOwUVxid6/e3QoyISCOSV1zODztP9LbkFFbvbekQ0oIhnVoxpFMIfdoF4uHaBKb+z06BZU/axm8AuLhCz+th8F/BP7Juf5dhQN5+2+mo7O22nhUXM5jMtt/rYrZN8uZy/LHJ/If7LqduW1oIaatsVwBlbKZqzpVKod1PBJq2A8ArwL66y4pg+zew6UPbaTbj+GBtk4vt0ui4CdB5hNOdLlKIQSFGRJxXSbmF7RkFrNyRzfId2fz+p94WH3czA2JP9LY0qZWij+6DFU/BhvfBsAIm27iNIQ855eW+VYqPwL6fYM8PsGclZG+r/rzJBcJ7nDj11Ob8msNH5QDdTR/Ctq9s434qRfS0BZfzrgLf0Hp9OfVJIQaFGBFp/AzDID2vhO0Z+WxLL2B7RgHb0vPZk1NUbUAuQMfQFgzpFMKQjq3o0y4Id9cmdulwQQb88KxtsjVruW1b5yvgokdsA3CbmsIs2zpEe1bafv7xUmew9exE9j7RU+PeAjZ/fPIA3cB2tuDSfTwEd2jQl1BfFGJQiBGRxqW4rIKUDFtQ2Z6ez7bjP/NLKmpsH+DtRr92QQzpFMKFnVoRGeDVwBU3kOIj8NN/YPUrUHHMtq39ELh4OrTu7dDSGlT+oeOB5nhPTe6+U7f1DrYN0O1+NbTu0+QG6J7r97euThIROUtWq8GBo8fYlpHP9vQCtmfksz2jgL2Hi6jpfw9dXUzEhrSgc5gvncP96BzmS5dwP0J8PZxr7pZKhmE7DVTtSp2KP22rAEs5bPrINlFd5aDa1v1g6KO20ynNjV8E9Jhgu4HttNofe2qO5doG6Ha/GmIucooBuo6iECMiYqcdmQX869vtrN59uNpsuH/UytejKqR0DvOlc5gfsSEtnOPUkKUClj1hu3TXUnry5cR/vMTYXqHd4OJHoWNik+tVOGuBbW23ntdRlX713tSKQoyISC0VlVbw/Pc7eX3lHiqOj2NxN7vQIbQFncP86BJuCy2dwnwJbuHh4GrPUtFh+PgG26DUc1V5ZY/JDEHRMPgB6DpGywOcjsKLXc4qxMydO5dnnnmGjIwMevTowQsvvEC/fv1qbPvmm28yderUats8PDwoKSmpemwYBo899hivvfYaubm5XHDBBbz88st06NA0Bi6JiHMzDIP/bclgxpdbSc+z/dt1SddQ7h3WgY6hvriZm8iXcvoGWHgd5KXZpvG/YjaEdf/DZcbm6sHkdJckm1z0hSz1zu4Q88EHH5CUlMS8efNISEhgzpw5JCYmkpKSQkhIzTMU+vn5kZJyYs2LP5/7ffrpp3n++ed56623iI6O5tFHHyUxMZGtW7fi6em8C1OJiPPbd7iIx77YwvIU21UirQO9mHHleQzt4ryXtdZo08fw+V22AbdB7WHie7YZbUUaMbuvTkpISKBv3768+OKLAFitVqKiorj77rt56KGHTmr/5ptvcu+995Kbm1vj8QzDICIigvvvv58HHngAgLy8PEJDQ3nzzTeZOHHiGWvS1UkiUtdKyi3MW5HKS8tTKauw4m524dYL23PHkFi83JvABHOVLBWQ/A/boFuA2GEw9r+2dYpE6tm5fn/b1QdaVlbG2rVrGTZs2IkDuLgwbNgwVq1adcr9CgsLadu2LVFRUYwaNYotW7ZUPbdnzx4yMjKqHdPf35+EhIRTHrO0tJT8/PxqNxGRurJiRzaJc35gztKdlFVYGRgbzLf3DuL+Szs1rQBTfAQWjDsRYAbeB9d8qAAjTsOu00k5OTlYLBZCQ6t3o4aGhrJ9+/Ya9+nUqRPz588nLi6OvLw8nn32WQYMGMCWLVto3bo1GRkZVcf48zErn/uzWbNmMWPGDHtKFxE5o/S8Y8z8aivfbLL92xPi68GjV3Tlirhw57wE+nQyNsPCa2xzlLh5w+iX4Lwxjq5KxC71fnVS//796d+/f9XjAQMG0KVLF1555RVmzpx5VsecNm0aSUlJVY/z8/OJioo651pFpHkqt1h586e9PLd0B8VlFswuJm4Y0I57h3XA17MJztGxeRF8fieUF0NAW9v4l7Bujq5KxG52hZjg4GDMZjOZmZnVtmdmZhIWFlarY7i5udGzZ0927bJNu1y5X2ZmJuHh4dWOGR8fX+MxPDw88PBw0ssXRaRRWbPnCI9+tpmUzAIAercNZOaobnSNaILj66wWSH4cfppje9z+Ihg3v25XWBZpQHaNiXF3d6d3794kJydXbbNarSQnJ1frbTkdi8XCpk2bqgJLdHQ0YWFh1Y6Zn5/P6tWra31MERF75RSWcv+HG7j6lVWkZBYQ6O3G02Pj+OjW/k0zwBw7CgvGnwgwA+6Gaz9WgBGnZvfppKSkJKZMmUKfPn3o168fc+bMoaioqGoumMmTJxMZGcmsWbMAePzxxzn//POJjY0lNzeXZ555hn379vGXv/wFsF1ufe+99/LEE0/QoUOHqkusIyIiGD16dN29UhERwGI1eH9NGk8v3l61ltGkflE8mNiZQB93B1dXT7K2wfuT4OgecPWCUS9C93GOrkrknNkdYiZMmEB2djbTp08nIyOD+Ph4Fi9eXDUwNy0tDZc/zMZ49OhRbr75ZjIyMggMDKR37978/PPPdO16YqXSBx98kKKiIm655RZyc3MZOHAgixcv1hwxIlJnLFaD1bsP89Ti7Ww4kAdA13A/nhjTjV5tmvDVOFu/gE9vg/Ii8G8DExdAeJyjqxKpE1rFWkSarAqLlTV7jvDN5nQWb84kp7AUAF8PV+6/tCPXnd8W16Yy2+6fWa2w7ElY+aztcbtBMP4t8Gnp2LpE/kCrWIuI/EG5xcqq1MN8uzmd/23J5EhRWdVzfp6ujIiL4L5hHQjxa8I9vcdyYdEtsPN/tsfn3wGXzASz/smXpkWfaBFxemUVVn7alcM3m9L5bmsmecfKq54L9HYj8bwwLuseTv/2LRvnKtKWCrCW1826Q9kptvlfDu8CV08Y+R/oceaZz0WckUKMiDilknILK3fm8O2mdJZsy6Tg+CBdgOAW7iSeF8bl3cNJiA5qvKeMyorgxzmwaq5tzMofmf604GK1+5ULLbr8aSFGMxzdazuWX2uY+C5E9HTEKxNpEAoxIuI0jpVZWLEji282ZZC8LZOiMkvVc618PbisWxiXdQunX3QQZpdGPMOu1QqbPoSl/4CC9JrbGBawWGp+7kzaXmAb/9Ki1VmXKOIMFGJEpFErKbeQvC2Lbzans2x7FsV/CC5hfp5c1t3W49K7TSAujTm4VNr/Kyz+Gxxca3sc2M42XiXmIttkdIbV9tNaYQsyVfdr2G5YbY+tluPbKmyXULfuq/Ev0izoUy4ijdL+I8W8+8s+PvhtP7nFJ8a4RAZ4cXl32xiX+NYBzhFcAPIOwtLHYNNHtsfuLWDwA7ZBt66agVzkbCjEiEijYbUa/JSaw1s/7yV5exaVE0BEBnhxRY9wRnQPp3ukv3MtxlhWDD8/bxv7UnEMMEHPa+Hi6eAbeqa9ReQ0FGJExOEKSsr5ZO0B3v5lH7uzTwxwHdQhmMn923Fx55DGPcalJoYBmz+BJY9B/gHbtjb9Yfi/ICLeoaWJNBUKMSLiMDszC3h71T4WrTtQNUi3hYcr43q35rrz2xIb0sLBFZ6lg2th8TTYv9r22L8NXPo4dB199pdOi8hJFGJEpEFVWKws3ZbF26v28nPq4artsSEtmNK/LWN6taaFh5P+05SfDskzYMP7tsduPjDoPuh/F7h5ObY2kSbISf+lEBFnc7iwlIW/7mfBL/s4lFcCgIsJLukaypT+7egf07L+x7ocOwpFh21jUTx86+645cdg1Yuw8rkT8730mARDHwO/8Lr7PSJSjUKMiNSrDftzeWvVXr7akE6ZxQpAkI87E/tGce35bYkMqMceioJM2PcT7PvZdsvacuI59xbgGw6+Yaf/6Xaa5QkMA7Z+Bt9Nh7w027bW/WzjXlr3rr/XJSKAQoyI1IOyCitfbTzEW6v2sWF/btX2uNb+TOnfjhFx4Xi6mev2lxoG5KYdDyzHg8uR1JPbufnYekvKCuHwTtvtdLwCaw43XoHw6+uQ9rOtnV8kDJsB3cdp3ItIA1GIEZE6k1dczoI1+3jzp71kFdhWjHY3uzAiLpwpA9oRHxVQd7/MMCBnZ/WelsqrgKqYIKybbQbbtgNsVwe1CIHSQijMhPxDUJBhmzX3pJ/pUFFiOwV17Chkba25DlcvGHgvDPg/cPeuu9cnImekECMi52z/kWJe/3EPH/62v2pG3RBfDyb3b8uEvm1o5VsHk7lZLZC5pXpPS3FO9TYurra1gtoOsAWXqATwCjj5WB4tbLeWMaf+fYYBJXknAk1NYadVJxgyDfxbn/vrExG7KcSIyFn7Pe0o/125h283p2M9PjFd5zBfbh7UnpE9Is5+xWjDsPWSZGyE9I22S5bTfoHSvOrtXD1tU+y3HWC7te4L7j7n9qIqmUy2AOQVACFd6uaYIlKnFGJExC4Wq8HSbZm89sNuftt3tGr74I6tuHlQNANjg+27yshqgcOpxwPLBtvPjE1QfPjktu4tbL0r7S6w9bRE9NSU/SLNmEKMiNTKsTILH687wOsrd7P3cDEAbmYTo+Ij+cugaDqH+Z35IBWltrEl6RtP9LJkbjlxWfIfmcy20zVhcRDeA9qcb7uvhQ1F5Dj9ayAip5VdUMo7q/byzi/7OHp8IUY/T1euO78tUwa0I9TvFJcgl+TbelQqw0rGRsjebltp+c9cvSD0PAiPOx5a4iCkqyaIE5HTUogRkRrtyirgvyv3sOj3g5RV2OZ3iQry4qYLohnfJwqfP86qW5BZ/XRQ+kY4uqfmA3sG/CGs9LD9DO4ALnV8ybWINHkKMSJSxTAMftl9hNdW7ub77VlV2+OjArhlcHsSu4Zgzt0Lu7609bJU9rAUZtZ8QL/WJwJLWHfbff8ozaMiInVCIUZEANiekc/fPt7IhgO2K4DcTBVMjinhurZ5tCtfgenXTfDVZijNr2Fvk603pfJUUNjxm0/Lhn0RItKsKMSINHOGYbBgdRpPf7WeRONHrnPfyQCfQ0SU7sF0oAz+PH+c2d02XuWPp4RCz6u7S5tFRGpJIUakGcsrLufvn/xGwPaFfOf6OWGm45dMHzvewMP/xGmgyl6W4I5gdnNYzSIilRRiRJqpdbszWPLeHB4q/5BIN9ucLIZ/a0xxE04MuA1sp/ErItJoKcSINDOWinKWf/QCHbe/xN9M2WCCMu8w3C/6K6aek8HV3dEliojUikKMSHNhtZC35n2OLX2SoRWHwAT55kDcL/orngk3gdsp5nsREWmkFGJEmjqrFbZ+StF3T+Cfvxt/4Ijhy/6utxI3JgmTBuSKiJNSiBFpqqxW2P4V1mX/xCV7Gz7AUaMFn3qP5cLrHqFHZKijKxQROScKMSJNjWHAjsWw7EnI2IQLkG9481rF5ZT2voWkkX3wdNPsuCLi/BRiRJoKw4BdybbwcmgdAIWGF69bhvOR6ygevaY/ieeFObhIEZG6oxAj4uwMA/asgGX/hP2rASgzefLf8kt5tWIEHdq14cOJPYkI0GKKItK0KMSIOLNjR+HjmyA1GQCr2YOPXS7jqYJEjpj8uXtoB/7v4lhczS4OLlREpO4pxIg4q/x0eHcsZG3BMHuwPXIsf9k9iIMV/oT6ebBgQjwDYoIdXaWISL1RiBFxRodT4Z3RkJuG1SeUJwJnMn9HCwAu7hzCM+PiaNnCw7E1iojUM4UYEWeTvsHWA1OUTXGLtlxX9hDrdrXAzWziocu6cOMF7TBpqQARaQYUYkScyd4f4f1JUJrPfo9YxuTcTw7+tGvpzQuTetG9tb+jKxQRaTAKMSLOYttXGB/fiMlSyq905ca8JIpdfLh1UDT3Du2Il7vmfhGR5kUhRsQZrHsH48v/w2RY+c7Sm7vL76ZjZCtmXdWdbpHqfRGR5kkhRqSRq/jhOVy//wcm4IOKIcw03cJfR3ThhgHtdOm0iDRrCjEijZVhkPHJg4RtfhWAeRUjWRV9N9+O6U5UkLeDixMRcTyFGJFGqKD4GLv+O5WeR74FYI7peqLHTePNHhG68khE5DiFGJFGZunGvXh8ehODjN+oMFz4OPJvTLk2iUAfd0eXJiLSqCjEiDQSWfkl/OvT1UxIfZAEl+2U4s7uIS8w8aKJji5NRKRRUogRcTCr1WDhr/t59dtVvGR9kq4u+yg1+8A1H9AlZpCjyxMRabQUYkQcaFdWIQ8v2kT6vm284/Yv2rlkUuHVCo8pn0JYd0eXJyLSqCnEiDhAWYWVl5enMnfZLtpb97LI/V+0MuViBLbD9fpPIai9o0sUEWn0FGJEGpDFavDZ7wd5/vud7DtcTF/Tdt7y/DfeRhGEdsN03SfgG+boMkVEnIJCjEgDsFoNvtqUzpylO9idXQTAGO+NPMtzmK2l0KY/TFoIXgGOLVRExIkoxIjUI8Mw+N+WDJ5bspOUzAIAAr3M/Cf2dwbtegaTYYGOw2HcG+CuCexEROyhECNSDwzD4PvtWcxesoMth/IB8PV05aFeFiZm/hvzzjW2hj0mwZUvgNnNgdWKiDgnhRiROmQYBit35jB7yQ7W788FwMfdzK39Q7jV8iEea18FwwJuPnDRw3D+HeCi9Y9ERM6GQoxIHfk5NYfnluzg171HAfByMzO5fxvuCtuK77L7oOCQrWGXK2H4LPBv7cBqRUScn0KMyDn6be8R/v3dDlbtPgyAu6sL15/fljt6uNByxSOwZqmtYWA7uPxZ6HCJ44oVEWlCzqofe+7cubRr1w5PT08SEhJYs2ZNrfZbuHAhJpOJ0aNHV9t+ww03YDKZqt2GDx9+NqWJNJj1+3OZPH8N4+atYtXuw7iZTUzu35aVSQN41PcrWr51IexaCmZ3GPwg3PGLAoyISB2yuyfmgw8+ICkpiXnz5pGQkMCcOXNITEwkJSWFkJCQU+63d+9eHnjgAQYNqnka9eHDh/PGG29UPfbw8LC3NJEGseVQHs8t2cHSbVkAuLqYGN+nNXdd3IHIw7/Au5PhSKqtcfSFMOLfENzBgRWLiDRNdoeY2bNnc/PNNzN16lQA5s2bx9dff838+fN56KGHatzHYrFw7bXXMmPGDFauXElubu5JbTw8PAgL0yRf0ngdyj3GzK+28u3mDABcTDCmZ2vuGdqBNm558N2dsPkTW+MWoZD4T+g2FkwmB1YtItJ02RViysrKWLt2LdOmTava5uLiwrBhw1i1atUp93v88ccJCQnhpptuYuXKlTW2Wb58OSEhIQQGBnLxxRfzxBNP0LJlyxrblpaWUlpaWvU4Pz/fnpchYreCknKuf301qdlFmEwwMi6Ce4Z1ICbIE379L3z/BJQVgMkF+t4MFz8Cnv6OLltEpEmzK8Tk5ORgsVgIDQ2ttj00NJTt27fXuM+PP/7I66+/zvr160953OHDh3PVVVcRHR1NamoqDz/8MJdddhmrVq3CbDaf1H7WrFnMmDHDntJFzprVanDfBxtIzS4izM+TN2/sS+cwPzjwG7x2H2RstDWM7A0jZkNEvEPrFRFpLur16qSCggKuv/56XnvtNYKDg0/ZbuLEiVX3u3fvTlxcHDExMSxfvpyhQ4ee1H7atGkkJSVVPc7PzycqKqpuixc57vnvd7J0Wyburi7Mu743nf0t8OW9sPZNwLD1uAz7B/SaAi4nh24REakfdoWY4OBgzGYzmZmZ1bZnZmbWOJ4lNTWVvXv3MnLkyKptVqvV9otdXUlJSSEmJuak/dq3b09wcDC7du2qMcR4eHho4K80iCVbM5mzdCcA/xzZkfjD38L7f4fiHFuDHpPgkpnQopUDqxQRaZ7sCjHu7u707t2b5OTkqsukrVYrycnJ3HXXXSe179y5M5s2baq27e9//zsFBQX85z//OWXvyYEDBzh8+DDh4eH2lCdy7sqK4egeOLKbnLQUjv68infd0jnP6zCB32YChq1dq862q47aDXRouSIizZndp5OSkpKYMmUKffr0oV+/fsyZM4eioqKqq5UmT55MZGQks2bNwtPTk27dulXbPyAgAKBqe2FhITNmzGDs2LGEhYWRmprKgw8+SGxsLImJief48kRqUJIPR3bbbscDC0eO/yxIr2oWDFxtAsxA2fGNngEw8F44/05wdW/w0kVE5AS7Q8yECRPIzs5m+vTpZGRkEB8fz+LFi6sG+6alpeFix1owZrOZjRs38tZbb5Gbm0tERASXXnopM2fO1CkjOXelhfDbfMjcciK4VJ4KOgXD05891jA2HWvJYfdIxl0yGL+IThDUHnyCdcm0iEgjYTIMw3B0EecqPz8ff39/8vLy8PPzc3Q50liUFcOCcbDvp5Of82llCyV/vAVGQ1A0s3/K4fnknbi7uvDxbf2Jax3Q4KWLiDQH5/r9rbWTpGkqL4GFk2wBxsMPLvg/aBl7Iqx41vwfy/+2ZPB88vGBvGO6K8CIiDRiCjHS9FSUwYfXw+7l4OYD130CUf3OuNuurAKSPlgPwA0D2jGut1aZFhFpzM5qAUiRRstSDh9PhZ3fgasXXPthrQJM3rFybn57LUVlFhKig3hkRJcGKFZERM6FQow0HVYLLLoFtn8FZg+Y9F6tLoG2zci7nj05RUT4ezL32l64mfWfhohIY6d/qaVpsFrh8zthyyJwcYOr34aYi2u163NLd/D99iw8XF145fo+BLfQVXEiIs5AIUacn2HAV/fChvfBZIZx86HT8FrtunhzOi98vwuAWVd1p3trLdooIuIsFGLEuRkGfPs3WPeWbQXpq16FrlfWatedmQXc/+EGAKZe0I6remkgr4iIM1GIEedlGLDkUVjziu3xqLnQfVytdrUN5P2NojIL57cP4uHLNZBXRMTZKMSI81r2T/j5Bdv9K+ZA/DW12s1iNbhn4e/sPVxMZIAXc6/RQF4REWekf7nFOf3wDPzwtO3+8Kegz9Ra7zp7SQrLU7KPD+TtTUsN5BURcUoKMeJ8fn4Bvn/Cdv+Sx+H822q967eb0pm7LBWAp8bG0S1SA3lFRJyVQow4lzWvwXd/t92/6BG44J5a75qSUcD9H9kG8t40MJrRPSPro0IREWkgCjHiPNa+Bd88YLs/6H4Y/Nda75pXXM4t7/xGcZmFATEtmXZZ53oqUkREGopCjDiHDR/Al8d7XfrfBRc/CiZTrXa1WA3uXvg7+44P5H3xml64aiCviIjT07/k0vht+RQ+uw0woO9f4NInah1gAJ79LoUfdmTj6WYbyBvk415/tYqISINRiJHGbfvX8MlfwLBCz+vhsmfsCjDfbkrn5eUayCsi0hQpxEjjtXMJfDgFrBUQNwFG/gdcav+R3ZVVyAPHB/L+ZWA0o+I1kFdEpClRiJHGafdy+OA6sJZD19Ew6iVwMdd696LSCm57dy1FZRYSooN4SAN5RUSaHIUYaXwyNsP7k6CiBDqNgLH/BbNrrXc3DIOHFm1iV1YhIb4evHBNTw3kFRFpgvQvuzQuJfnw4WQoL4b2Q2D8G2B2s+sQb/68ly83HMLVxcTca3sR4utZP7WKiIhDKcRI42EY8MVdcCQV/FrDuDfA1b4lAX7be4Qnv94GwLTLu9C3XVB9VCoiIo2AQow0Hqtfga2fg4sbjH8TvO0LINkFpdz53joqrAZXxIVz4wXt6qVMERFpHBRipHE48NuJ5QQunQlRfe3avcJi5e7315GZX0psSAueGhuHyY5LsUVExPkoxIjjFR85fil1OXS5EhJqv6BjpWe+S+GX3UfwcTcz77re+HjUfiCwiIg4J4UYcSyrFRbdAvkHIKg9jHrRrsnsABZvzuCVFbsBeHpcD2JDWtRHpSIi0sgoxIhj/Tgbdi0BV0+4+m3wtG9G3d3ZJya0u2lgNCPiwuujShERaYQUYsRx9vwAy5603b/8GQjrbtfuxWUV3P7uOgpLK+jbLlAT2omINDMKMeIYBRnw8U22NZF6XGNbF8kOhmEwbdEmUjILaOXrwdxreuGmCe1ERJoV/asvDc9SYQswRVkQ0hVG/NvucTBvr9rH5+sPYXYx8eKknoT4aUI7EZHmRiFGGt6yJ2Hfj+DewjYOxt3brt3X7jvKE19vBeCh4Z1JaN+yPqoUEZFGTiFGGtaO/9kG8wJc+TwEd7Br95zCUu5csI5yi8Hl3cP4y6DoeihSREScgUKMNJzcNNvl1AB9b4ZuY+3avcJi5f/e/52M/BLat/Lh6XE9NKGdiEgzphAjDaOiDD66AUpyIaIXJD5p9yH+vWQHP6cextvdzCvX9aaFJrQTEWnWFGKkYXz3dzi4FjwDbOsi2bmw43dbMnh5eSoA/xobR4dQ37qvUUREnIpCjNS/LZ/Cmlds98e8AoFt7dp9b04R939om9DuhgHtuLJHRF1XKCIiTkghRupXzi74/G7b/QvuhU7D7dr9WJmF295dS0FpBb3bBvLw5V3qvkYREXFKCjFSf8qK4cPJUFYAbS+Aix+1a3fDMHjk001szygguIU7c6/phburPrIiImKjbwSpP9/8FbK2gE8rGDcfzPYNxH13dRqLfj+I2cXEC5N6EeavCe1EROQEhRipH7+/C+vfBZMLjH0dfMPs2z3tKI9/uQWABxM70T9GE9qJiEh1CjFS9zI2w9f32+4PeRjaX2jX7keLyqomtEs8L5RbBrevhyJFRMTZKcRI3SrJt42DqSiB2GEw6H67djcMg0c+28ShvBKig314ZrwmtBMRkZopxEjdMQz44m44kgp+kTDmVXCx7yP26e8H+WZTBq4uJv4zMR4/T7d6KlZERJydQozUnTWvwtbPwMXVNqGdj33jWA4cLeaxz23jYO4Z2oG41gF1XqKIiDQdCjFSN9I3wP8esd2/ZCZE9bNrd4vVIOnDDRSUVtCrTQC3D4mphyJFRKQpUYiRc1dWBB/fBNZy6DQCzr/d7kP8d+Vu1uw5gre7mecmxONq1kdTREROT98Ucu4WT4PDO8E3HEa9CHYOxN16KJ9nv0sBYPoVXWnb0qc+qhQRkSZGIUbOzZbPYN1bgAmuehW8g+zavaTcwn0frKfcYjCsSygT+kbVS5kiItL0KMTI2cs7AF/+n+3+wPsgerDdh3j2fymkZNqWFfjX2O66nFpERGpNIUbOjtUCi26BkjyI6AUXPWz3IX7elcN/f9wDwFNj4whu4VHXVYqISBOmECNnZ+Vs2PcTuLeAsf8Fs33zueQdK+f+jzYAMKlfG4Z2Ca2PKkVEpAlTiBH77V8Dy2fZ7l/+LLS0/3Lo6Z9vJj2vhHYtvfn7iC51XKCIiDQHCjFin5I8+OQmMCzQfTz0mGj3Ib7YcIjP1x/C7GLiuQnx+HjYt7q1iIgIKMSIvb5+AHLTIKANjPi33ZdTp+cd4++fbgLgzoti6dkmsD6qFBGRZkAhRmpvw0LY9CGYzDD2dfD0t2t3q9XggY82kF9SQY/W/tx9cWw9FSoiIs3BWYWYuXPn0q5dOzw9PUlISGDNmjW12m/hwoWYTCZGjx5dbbthGEyfPp3w8HC8vLwYNmwYO3fuPJvSpL4c2Q1fH1+ReshDdi8rAPDGz3v5addhPN1ceG5CPG6alVdERM6B3d8iH3zwAUlJSTz22GOsW7eOHj16kJiYSFZW1mn327t3Lw888ACDBg066bmnn36a559/nnnz5rF69Wp8fHxITEykpKTE3vKkPljK4ZO/QFkhtBkAg+63+xA7Mgt4avF2AB4Z0ZX2rVrUdZUiItLM2B1iZs+ezc0338zUqVPp2rUr8+bNw9vbm/nz559yH4vFwrXXXsuMGTNo3759tecMw2DOnDn8/e9/Z9SoUcTFxfH2229z6NAhPvvsM7tfkNSD5bPg4Frb6aOrXgUXs127l1VYuXfhesoqrFzUqRXXJbSpp0JFRKQ5sSvElJWVsXbtWoYNG3biAC4uDBs2jFWrVp1yv8cff5yQkBBuuummk57bs2cPGRkZ1Y7p7+9PQkLCKY9ZWlpKfn5+tZvUkz0/2OaEARj5PATYvyzA7CU72JqeT5CPO0+Ni9OsvCIiUifsCjE5OTlYLBZCQ6tPTBYaGkpGRkaN+/z444+8/vrrvPbaazU+X7mfPcecNWsW/v7+VbeoKK23Uy+Kj8CiWwEDel4P5422+xBr9hzhlR9SAfjnmO6E+HrWbY0iItJs1evIyoKCAq6//npee+01goOD6+y406ZNIy8vr+q2f//+Oju2HGcY8MXdUHAIWsbCZU/ZfYiCknLu+2A9hgHje7dmeLeweihURESaK7tmGQsODsZsNpOZmVlte2ZmJmFhJ39BpaamsnfvXkaOHFm1zWq12n6xqyspKSlV+2VmZhIeHl7tmPHx8TXW4eHhgYeH1tmpV2vfgO1fgYub7XJqdx+7D/GPL7ZyMPcYUUFePHblefVQpIiINGd29cS4u7vTu3dvkpOTq7ZZrVaSk5Pp37//Se07d+7Mpk2bWL9+fdXtyiuv5KKLLmL9+vVERUURHR1NWFhYtWPm5+ezevXqGo8pDSBrOyw+vqDjsMcgIt7uQ3y7KZ1P1h3AxQTPXR1PC83KKyIidczub5akpCSmTJlCnz596NevH3PmzKGoqIipU6cCMHnyZCIjI5k1axaenp5069at2v4BAQEA1bbfe++9PPHEE3To0IHo6GgeffRRIiIiTppPRhpAeYntcuqKYxBzMZx/p92HyMov4eHjs/LePiSGPu2C6rpKERER+0PMhAkTyM7OZvr06WRkZBAfH8/ixYurBuampaXh4mLfUJsHH3yQoqIibrnlFnJzcxk4cCCLFy/G01ODQBtc8gzI3ATewTB6Htj5tzQMg79+vJGjxeV0i/TjnqEd66lQERFp7kyGYRiOLuJc5efn4+/vT15eHn5+fo4ux3ntXAILxtnuX/MhdEy0+xDvrNrLo59vwcPVha//byCxIb51XKSIiDQV5/r9rXnfxaYwCz673Xa/361nFWBSswt58pttAEy7rLMCjIiI1CuFGAGr1RZgirIh5Dy45PGzOIRB0gfrKSm3MqhDMJP7t6v7OkVERP5AIUZg9cuwaym4esK418HN/rFI32xOZ8OBPHw9XHlmXA9cXDQrr4iI1C+FmOYufQMsecx2P/FJCOli9yEsVoPnluwA4C+D2hPmrwHZIiJS/xRimrtv/grWcug0AvqcvLZVbXy+/iCp2UUEeLtx48B2dVufiIjIKSjENGdZ22H/anBxhStmw1kszFhusfKf5J0A3DK4Pb6ebnVdpYiISI0UYpqz9e/afnYcDr5nt67RonUH2He4mOAW7twwoF3d1SYiInIGCjHNlaUcNiy03e953VkdorTCwvPJuwC47cIYvN21tICIiDQchZjmaud3tkuqW4RC7CVndYgPf93PwdxjhPp5cN35beu4QBERkdNTiGmufj9+KqnHRDDb34NSUm7hxWW2Xpg7L4rF081cl9WJiIickUJMc1SQCTv+Z7sff3ankhasTiMzv5TIAC8m9I2qw+JERERqRyGmOdq4EAwLRCVAK/sXaCwuq+Dl5bZemLsvjsXDVb0wIiLS8BRimhvDOHEq6SwH9L718z5yCstoE+TN2N6t67A4ERGR2lOIaW4O/Ao5O8DNG84bY/fuBSXlvPJDKgD3DO2Am1kfIRERcQx9AzU3v79j+3neGPCwf5XpN37aS25xOTGtfBjdM7KOixMREak9hZjmpKwINi+y3T+LU0l5xeW8tnI3APcO64hZizyKiIgDKcQ0J1s/h7JCCGoPbfrbvftrK3dTUFJB5zBfRnQPr4cCRUREak8hpjmpHNAbf63d6yQdKSrjjZ/2ALZeGBf1woiIiIMpxDQXh1Nh309gcoEek+ze/ZUVqRSVWegW6UfieaH1UKCIiIh9FGKai/ULbD9jhoK/fQNyswpKeGvVXgDuv6QTprNY7VpERKSuKcQ0B1YLrH/Pdv8sBvS+tCyVknIrPdsEMKRTqzouTkRE5OwoxDQHqd9DQTp4BUGny+zaNT3vGO+tTgPUCyMiIo2LQkxzUDk3TNwEcPWwa9cXv99FmcVKQnQQF8S2rIfiREREzo5CTFNXdBi2f2O73/Nau3bdf6SYD37dD8D9l6oXRkREGheFmKZu04dgLYfweAjrbteuzyfvpMJqMKhDMP2ig+qnPhERkbOkENOUGQasO34qyc4BvXtyilj0+0EAki6xf6VrERGR+qYQ05Slr4esLWD2gO7j7Nr1P0t3YLEaDO0cQs82gfVTn4iIyDlQiGnKKmfo7TISvGofRHZkFvD5hkMA3KdeGBERaaQUYpqq8mOw6SPbfTtPJc1ZugPDgOHnhdEt0r8eihMRETl3CjFN1favoSQP/KMg+sJa77blUB7fbMrAZFIvjIiING4KMU1V5dww8deCS+3/zM8t2QnAFXERdArzrY/KRERE6oRCTFN0dB/sXmG7H39NrXdbvz+XpdsycTHBvcM61FNxIiIidUMhpina8D5g2E4jBbat9W6zl+wAYEzP1sS0alFPxYmIiNQNhZimxmqF34+vWN3z+lrv9tveI/ywIxtXFxP3DFUvjIiINH4KMU3N3h8gLw08/KHLFbXe7d/f2XphxvdpTZuW3vVVnYiISJ1RiGlqKueG6T4O3LxqtcvPu3JYtfsw7mYX7rpYvTAiIuIcFGKakmNHYesXtvu1nBvGMAz+fXwszKR+UUQG1C74iIiIOJpCTFOy+ROwlELIeRDRs1a7rNiRzdp9R/FwdeHOi2LruUAREZG6oxDTlFSeSup5HZhMtdrlhe93AXD9+W0J8fOsr8pERETqnEJMU5GxGQ79Di5uEHd1rXbZfDCPtfuO4mY2ccuF7eu5QBERkbqlENNUrD9+WXWny8AnuFa7LFi9D4DE88II8VUvjIiIOBeFmKagogw2LLTdr+XcMPkl5Xz2u22l6uvPr/2EeCIiIo2FQkxTsONbOHYEfMMh5uJa7bJo7QGOlVvoGNqCftFB9VygiIhI3VOIaQoqB/T2mARm1zM2NwyDd1enAXDd+W0x1XIQsIiISGOiEOPs8g/BrqW2+7WcG+aX3UfYlVWIt7uZMT0j67E4ERGR+qMQ4+w2vA+GFdoMgJYxtdrl3V9sA3rH9IzE19OtPqsTERGpNwoxzswwqs8NUwtZ+SX8b0sGYDuVJCIi4qwUYpxZ2io4shvcW0DXUbXaZeGv+6mwGvRpG0iXcL96LlBERKT+KMQ4s8pemPPGgEeLMzavsFh57/iA3uv7qxdGREScm0KMsyotgC2f2u7Xcm6YpduyyMgvoaWPO8O7hdVjcSIiIvVPIcZZbfkUyouhZQeI6lerXSpn6L26bxQerub6rE5ERKTeKcQ4KzsXe9ydXcjKnTmYTHBNvzb1XJyIiEj9U4hxRtk7YP9qMJmhx8Ra7bLg+FiYizuFEBXkXZ/ViYiINAiFGGe04T3bzw6Xgu+Zx7YcK7Pw0W/7AV1WLSIiTcdZhZi5c+fSrl07PD09SUhIYM2aNadsu2jRIvr06UNAQAA+Pj7Ex8fzzjvvVGtzww03YDKZqt2GDx9+NqU1fYYBWz6z3Y+7ula7fLnxEPklFUQFeTG4Y6v6q01ERKQBnXmhnT/54IMPSEpKYt68eSQkJDBnzhwSExNJSUkhJCTkpPZBQUE88sgjdO7cGXd3d7766iumTp1KSEgIiYmJVe2GDx/OG2+8UfXYw8PjLF9SE5e5GY7uAVdPW09MLVTO0HttQlvMLlonSUREmga7e2Jmz57NzTffzNSpU+natSvz5s3D29ub+fPn19h+yJAhjBkzhi5duhATE8M999xDXFwcP/74Y7V2Hh4ehIWFVd0CAwPP7hU1dVu/sP2MHVaruWE27M9l44E83M0ujO/dup6LExERaTh2hZiysjLWrl3LsGHDThzAxYVhw4axatWqM+5vGAbJycmkpKQwePDgas8tX76ckJAQOnXqxO23387hw4dPeZzS0lLy8/Or3ZqNrZ/bfna5slbNK3thRsSF07KFerdERKTpsOt0Uk5ODhaLhdDQ0GrbQ0ND2b59+yn3y8vLIzIyktLSUsxmMy+99BKXXHJJ1fPDhw/nqquuIjo6mtTUVB5++GEuu+wyVq1ahdl88nwms2bNYsaMGfaU3jRkp0BOCri4QcfEMzbPLS7jiw2HAA3oFRGRpsfuMTFnw9fXl/Xr11NYWEhycjJJSUm0b9+eIUOGADBx4onLhLt3705cXBwxMTEsX76coUOHnnS8adOmkZSUVPU4Pz+fqKioen8dDld5KinmIvAKOGPzj9ceoLTCStdwP3q1OXN7ERERZ2JXiAkODsZsNpOZmVlte2ZmJmFhp77U18XFhdjYWADi4+PZtm0bs2bNqgoxf9a+fXuCg4PZtWtXjSHGw8OjeQ78teNUktVqVM0Nc935bTHVYkI8ERERZ2LXmBh3d3d69+5NcnJy1Tar1UpycjL9+/ev9XGsViulpaWnfP7AgQMcPnyY8PBwe8pr2o7shsxNtgnuOo84Y/OfUnPYk1OEr4cro+IjGqBAERGRhmX36aSkpCSmTJlCnz596NevH3PmzKGoqIipU6cCMHnyZCIjI5k1axZgG7/Sp08fYmJiKC0t5ZtvvuGdd97h5ZdfBqCwsJAZM2YwduxYwsLCSE1N5cEHHyQ2NrbaJdjNXuWppOhB4B10xubvrLIN6B3buzU+Hg1y1lBERKRB2f3tNmHCBLKzs5k+fToZGRnEx8ezePHiqsG+aWlpuLic6OApKirijjvu4MCBA3h5edG5c2feffddJkyYAIDZbGbjxo289dZb5ObmEhERwaWXXsrMmTOb5ymjU7HjVFJ63jGWbrOd8rs2QeskiYhI02QyDMNwdBHnKj8/H39/f/Ly8vDz83N0OXUvdz/M6QaY4P4U8A09bfPZ36Xw/Pe7OL99EAtvqf1pPhERkYZ0rt/fWjvJGWz70vaz7YAzBphyi5X3f7Wtk3T9+e3quTARERHHUYhxBnacSvpuSybZBaW08vXg0vNOH3hEREScmUJMY1eQAftX2+53GXnG5u/8sheASX2jcDPrzysiIk2XvuUau21fAga07gv+kadtujOzgF92H8HsYmKSBvSKiEgTpxDT2NlxKqlycrthXUII9/eqz6pEREQcTiGmMSvKgX0/2e53PX2IKSqt4JO1BwCtkyQiIs2DQkxjtv1rMKwQ3gMC25226efrD1FQWkF0sA8XxAQ3TH0iIiIOpBDTmNXyVJJhGLz7i22G3msT2uDionWSRESk6VOIaayOHYU9K2z3u446bdN1ablsTc/Hw9WFcb1bN0BxIiIijqcQ01ilLAZrBYR0heAOp21a2QtzZY8IArzdG6I6ERERh1OIaaxqeSrpcGEpX29MB+D6/hrQKyIizYdCTGNUWgCp39vun+GqpI/WHqDMYiWutT9xrQPqvzYREZFGQiGmMdrxP7CUQstY2+mkU7BYDRastp1K0mXVIiLS3CjENEZ/PJVkOvWVRj/syGb/kWP4e7kxMi6igYoTERFpHBRiGpuyYti11Hb/DKeSKgf0ju/dGi93c31XJiIi0qgoxDQ2u5ZCeTEEtIHw+FM223+kmO9TsgC4VqeSRESkGVKIaWxqeSrpvTVpGAYM6hBMdLBPAxUnIiLSeCjENCYVpbZBvXDaCe5KKyx8+Ot+QAN6RUSk+VKIaUxSl0FZAfhGQGSfUzZbvDmDw0VlhPt7MrRzSAMWKCIi0ngoxDQmVaeSRoJLzX8awzB4beVuACb1a4OrWX9CERFpnvQN2FhYyiHlG9v901yV9NOuw2w+mI+nm4tOJYmISLOmENNY7PkBSnLBpxW06X/KZvNWpAIwsW8bgny0TpKIiDRfCjGNReWppM5XgEvNc75sOpDHj7tyMLuYuGlgdAMWJyIi0vgoxDQGVgts/9p2/zSnkub9YOuFGRkXTlSQd0NUJiIi0mgpxDQG+36G4hzwCoR2g2pucriIbzfZVqu+9cKYhqxORESkUVKIaQwqTyV1GgFmtxqbvPrDbqwGDOnUii7hfg1YnIiISOOkEONoVits+9J2/xSnkrILSvlo7QEAblMvjIiICKAQ43gHfoXCDPDwg/ZDamzy5s97KKuwEh8VQEJ0UMPWJyIi0kgpxDha5amkjsPB1eOkpwtLK3hnlW216tsujMF0mvWUREREmhOFGEcyjDOeSnp/dRr5JRW0b+XDpV1DG7A4ERGRxk0hxpEO/Q55aeDmA7HDTnq6rMLK6z/uAeDWwe1xcVEvjIiISCWFGEeqPJXU4RJw8zrp6c/WHyQjv4RQPw9G94xs4OJEREQaN4UYRzEM2PaF7X4Np5KsVoNXji8xcOMF0Xi41jyLr4iISHOlEOMomVvgyG5w9YQOl5709NJtmaRmF+Hr6co1CW0cUKCIiEjjphDjKJWnkmKGgodvtacMw6ha6PG689vi61nzBHgiIiLNmUKMo5zmVNKve4+yLi0Xd1cXpl7QrmHrEhERcRIKMY6QnQLZ28HFzTY/zJ9U9sKM7dWaEF/Phq5ORETEKSjEOMLW470w7YeAV0C1p1IyCvh+exYmE9wyuH2DlyYiIuIsFGIcYdvx8TA1nEqqvCLpsm5hRAf7NGRVIiIiTkUhpqEd2Q0Zm8Bktq1a/QcHc4/xxYZDgBZ6FBEROROFmIZWeSqp3UDwaVntqf+u3E2F1WBATEviWgc0fG0iIiJORCGmoZ3iqqSjRWUsXLMfUC+MiIhIbSjENKTc/XBwLWCCziOrPfX2qn0cK7dwXoQfgzoEO6Y+ERERJ6IQ05AqV6xu0x98T6xIfazMwlur9gJw64UxmExa6FFERORMFGIa0ilOJX34236OFJURFeTF5d3CHFCYiIiI81GIaSgFGZD2i+1+lxOnkiosVl5buRuAWwa1x9WsP4mIiEht6BuzIRRmw/uTAANa9wX/1lVPfb0pnQNHj9HSx53xfaIcV6OIiIiTcXV0AU3ekd3w7ljbT68guOzpqqdsCz3aemFuGNAOTzezo6oUERFxOgox9engOlgwHopzIKANXLcIgjtUPb1iRzbb0vPxdjdzff+2DixURETE+SjE1JedS+HDyVBeBGHd4dqPwbf6oN2Xl9uWGJjUrw0B3u6OqFJERMRpKcTUh/XvwRd3g7XCtsjj1e+Ap1+1Jr+nHWX1niO4upi4aWC0Y+oUERFxYgoxdckwYOW/4fuZtsdxE+DKF8H15F6WeccXehwVH0lEgFdDVikiItIkKMTUFasFvn0Qfv2v7fEF98LQx8Dl5AvAUrML+W5rJgC3Xdi+AYsUERFpOhRi6kL5MfjkL7D9K8AElz0FCbeesvmrK3ZjGDCsSwgdQn0brk4REZEm5KzmiZk7dy7t2rXD09OThIQE1qxZc8q2ixYtok+fPgQEBODj40N8fDzvvPNOtTaGYTB9+nTCw8Px8vJi2LBh7Ny582xKa3jFR+DtUbYAY/aA8W+eNsBk5pfw6e8HAS30KCIici7sDjEffPABSUlJPPbYY6xbt44ePXqQmJhIVlZWje2DgoJ45JFHWLVqFRs3bmTq1KlMnTqV//3vf1Vtnn76aZ5//nnmzZvH6tWr8fHxITExkZKSkrN/ZQ0hNw3mJ8L+1eDpD9d/CueNPu0u83/cQ5nFSp+2gfRpF9QwdYqIiDRBJsMwDHt2SEhIoG/fvrz44osAWK1WoqKiuPvuu3nooYdqdYxevXoxYsQIZs6ciWEYREREcP/99/PAAw8AkJeXR2hoKG+++SYTJ0484/Hy8/Px9/cnLy8PPz+/M7avExmb4N1xUJgBfpFw3ScQ0uW0u+QdK+eCf31PYWkF/53ch2FdQ0/bXkREpCk71+9vu3piysrKWLt2LcOGDTtxABcXhg0bxqpVq864v2EYJCcnk5KSwuDBgwHYs2cPGRkZ1Y7p7+9PQkLCKY9ZWlpKfn5+tVuD2r0C5l9mCzAhXeGmJWcMMAALVu+jsLSCDiEtuLhzSAMUKiIi0nTZFWJycnKwWCyEhlbvQQgNDSUjI+OU++Xl5dGiRQvc3d0ZMWIEL7zwApdccglA1X72HHPWrFn4+/tX3aKiGnDNoU0f25YRKCuAtgNh6rfgH3nG3UrKLcz/cS8At14Yg4uLqZ4LFRERadoaZAFIX19f1q9fz6+//sqTTz5JUlISy5cvP+vjTZs2jby8vKrb/v37667Y0/n5BfjkJrCWQ9fRtlNIXgFn3C01u5AJr/5CTmEp4f6eXNkjot5LFRERaersusQ6ODgYs9lMZmZmte2ZmZmEhYWdYi/bKafY2FgA4uPj2bZtG7NmzWLIkCFV+2VmZhIeHl7tmPHx8TUez8PDAw8PD3tKPzdWK3z3d/hlru1xwu2Q+M8a54CpvpvBmz/v5anF2ymtsOLr6cqsq7rj7qrFw0VERM6VXd+m7u7u9O7dm+Tk5KptVquV5ORk+vfvX+vjWK1WSktLAYiOjiYsLKzaMfPz81m9erVdx6w3FaXwyY0nAswlM2H4rDMGmP1Hirnmv7/w+FdbKa2wMqhDMN/dN5ghnTQWRkREpC7YPdldUlISU6ZMoU+fPvTr1485c+ZQVFTE1KlTAZg8eTKRkZHMmjULsI1f6dOnDzExMZSWlvLNN9/wzjvv8PLLLwNgMpm49957eeKJJ+jQoQPR0dE8+uijREREMHr06Lp7pWejJA8WXgt7V4KLG4x+CeKuPu0uhmHw4W/7mfnVNgpLK/ByM/PIiC5cm9AGk0njYEREROqK3SFmwoQJZGdnM336dDIyMoiPj2fx4sVVA3PT0tJw+UMvRVFREXfccQcHDhzAy8uLzp078+677zJhwoSqNg8++CBFRUXccsst5ObmMnDgQBYvXoynp2cdvMRzsP9X2PsjuPvChHcg5qLTNs/KL+GhRZv4frttzpw+bQP599U9aNvSpyGqFRERaVbsniemMarXeWLWvQPhPSA87rTNvtxwiEc/30xucTnuZhceSOzITQPbY9ZVSCIiIjU61+9vrZ10Jr2uP+3TR4vK+Pvnm/l6YzoA3SL9mH11PB21JpKIiEi9Uog5B8nbMnlo0SayC0oxu5i466JY7ro4Fjezrj4SERGpbwoxZ6GgpJyZX23lw98OABAb0oLZV/cgrnWAYwsTERFpRhRi7PRzag5//WgjB3OPYTLBXwZGc/+lnfB0Mzu6NBERkWZFIaaWjpVZeGrxdt78eS8AUUFePDuuBwntWzq2MBERkWZKIaYW1qUd5YEPN7A7pwiAaxLa8MjlXfDx0NsnIiLiKPoWPo2yCiv/Sd7By8tTsRoQ6ufBU2PjNOuuiIhII6AQcxrLUrKYuywVgNHxEcy4shv+3m4OrkpERERAIea0Lu0ayqR+bRjcIZjLuoefeQcRERFpMAoxp2EymZh1VXdHlyEiIiI10KxsIiIi4pQUYkRERMQpKcSIiIiIU1KIEREREaekECMiIiJOSSFGREREnJJCjIiIiDglhRgRERFxSgoxIiIi4pQUYkRERMQpKcSIiIiIU1KIEREREaekECMiIiJOqUmsYm0YBgD5+fkOrkRERERqq/J7u/J73F5NIsQUFBQAEBUV5eBKRERExF4FBQX4+/vbvZ/JONv404hYrVYOHTqEr68vJpOpTo+dn59PVFQU+/fvx8/Pr06PLaem990x9L47ht53x9D77hh/fN99fX0pKCggIiICFxf7R7g0iZ4YFxcXWrduXa+/w8/PTx9yB9D77hh63x1D77tj6H13jMr3/Wx6YCppYK+IiIg4JYUYERERcUoKMWfg4eHBY489hoeHh6NLaVb0vjuG3nfH0PvuGHrfHaMu3/cmMbBXREREmh/1xIiIiIhTUogRERERp6QQIyIiIk5JIUZERESckkKMiIiIOCWFmNOYO3cu7dq1w9PTk4SEBNasWePokpq8f/zjH5hMpmq3zp07O7qsJueHH35g5MiRREREYDKZ+Oyzz6o9bxgG06dPJzw8HC8vL4YNG8bOnTsdU2wTcab3/IYbbjjpsz98+HDHFNuEzJo1i759++Lr60tISAijR48mJSWlWpuSkhLuvPNOWrZsSYsWLRg7diyZmZkOqrhpqM37PmTIkJM+87fddptdv0ch5hQ++OADkpKSeOyxx1i3bh09evQgMTGRrKwsR5fW5J133nmkp6dX3X788UdHl9TkFBUV0aNHD+bOnVvj808//TTPP/888+bNY/Xq1fj4+JCYmEhJSUkDV9p0nOk9Bxg+fHi1z/7777/fgBU2TStWrODOO+/kl19+YcmSJZSXl3PppZdSVFRU1ea+++7jyy+/5KOPPmLFihUcOnSIq666yoFVO7/avO8AN998c7XP/NNPP23fLzKkRv369TPuvPPOqscWi8WIiIgwZs2a5cCqmr7HHnvM6NGjh6PLaFYA49NPP616bLVajbCwMOOZZ56p2pabm2t4eHgY77//vgMqbHr+/J4bhmFMmTLFGDVqlEPqaU6ysrIMwFixYoVhGLbPtpubm/HRRx9Vtdm2bZsBGKtWrXJUmU3On993wzCMCy+80LjnnnvO6bjqialBWVkZa9euZdiwYVXbXFxcGDZsGKtWrXJgZc3Dzp07iYiIoH379lx77bWkpaU5uqRmZc+ePWRkZFT7/Pv7+5OQkKDPfz1bvnw5ISEhdOrUidtvv53Dhw87uqQmJy8vD4CgoCAA1q5dS3l5ebXPe+fOnWnTpo0+73Xoz+97pQULFhAcHEy3bt2YNm0axcXFdh23SaxiXddycnKwWCyEhoZW2x4aGsr27dsdVFXzkJCQwJtvvkmnTp1IT09nxowZDBo0iM2bN+Pr6+vo8pqFjIwMgBo//5XPSd0bPnw4V111FdHR0aSmpvLwww9z2WWXsWrVKsxms6PLaxKsViv33nsvF1xwAd26dQNsn3d3d3cCAgKqtdXnve7U9L4DXHPNNbRt25aIiAg2btzI3/72N1JSUli0aFGtj60QI43KZZddVnU/Li6OhIQE2rZty4cffshNN93kwMpE6tfEiROr7nfv3p24uDhiYmJYvnw5Q4cOdWBlTcedd97J5s2bNc6ugZ3qfb/llluq7nfv3p3w8HCGDh1KamoqMTExtTq2TifVIDg4GLPZfNLo9MzMTMLCwhxUVfMUEBBAx44d2bVrl6NLaTYqP+P6/DtW+/btCQ4O1me/jtx111189dVXLFu2jNatW1dtDwsLo6ysjNzc3Grt9XmvG6d632uSkJAAYNdnXiGmBu7u7vTu3Zvk5OSqbVarleTkZPr37+/AypqfwsJCUlNTCQ8Pd3QpzUZ0dDRhYWHVPv/5+fmsXr1an/8GdODAAQ4fPqzP/jkyDIO77rqLTz/9lO+//57o6Ohqz/fu3Rs3N7dqn/eUlBTS0tL0eT8HZ3rfa7J+/XoAuz7zOp10CklJSUyZMoU+ffrQr18/5syZQ1FREVOnTnV0aU3aAw88wMiRI2nbti2HDh3isccew2w2M2nSJEeX1qQUFhZW+7+dPXv2sH79eoKCgmjTpg333nsvTzzxBB06dCA6OppHH32UiIgIRo8e7biindzp3vOgoCBmzJjB2LFjCQsLIzU1lQcffJDY2FgSExMdWLXzu/POO3nvvff4/PPP8fX1rRrn4u/vj5eXF/7+/tx0000kJSURFBSEn58fd999N/379+f88893cPXO60zve2pqKu+99x6XX345LVu2ZOPGjdx3330MHjyYuLi42v+ic7q2qYl74YUXjDZt2hju7u5Gv379jF9++cXRJTV5EyZMMMLDww13d3cjMjLSmDBhgrFr1y5Hl9XkLFu2zABOuk2ZMsUwDNtl1o8++qgRGhpqeHh4GEOHDjVSUlIcW7STO917XlxcbFx66aVGq1atDDc3N6Nt27bGzTffbGRkZDi6bKdX03sOGG+88UZVm2PHjhl33HGHERgYaHh7extjxowx0tPTHVd0E3Cm9z0tLc0YPHiwERQUZHh4eBixsbHGX//6VyMvL8+u32M6/stEREREnIrGxIiIiIhTUogRERERp6QQIyIiIk5JIUZERESckkKMiIiIOCWFGBEREXFKCjEiIiLilBRiRERExCkpxIiIiIhTUogRERERp6QQIyIiIk7p/wHZEIjh6cYETgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualize the training curves\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['acc', 'val_acc'])\n",
    "plt.title(f'Accuracy')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save & Reload Keras Model\n",
    "\n",
    "Let's save our model, then test it on some examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Skipping full serialization of Keras layer <keras.src.layers.regularization.dropout.Dropout object at 0x7f9bc036a7d0>, because it is not built.\n",
      "WARNING:tensorflow:Skipping full serialization of Keras layer <keras.src.layers.regularization.spatial_dropout1d.SpatialDropout1D object at 0x7f9bc02cd1e0>, because it is not built.\n",
      "INFO:tensorflow:Assets written to: demo_models/emotion_model/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: demo_models/emotion_model/assets\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.\n"
     ]
    }
   ],
   "source": [
    "# saving the model\n",
    "save_path = 'demo_models/emotion_model'\n",
    "model.save(save_path)\n",
    "\n",
    "model = tf.keras.models.load_model(save_path, compile=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_emotions(txt, threshold=0.5):\n",
    "    # recall it is multi-class so we need to get all prediction above a threshold (0.5)\n",
    "    preds = model(tf.constant([txt]))[0]\n",
    "    out = 0\n",
    "    for i in range(NUM_CLASSES):\n",
    "        if preds[i] > threshold:\n",
    "            emotion_name = CLASSES[i]\n",
    "            emotion_prob = round(float(preds[i]) * 100, 1)\n",
    "            print(f\"{emotion_name} ({emotion_prob})%\")\n",
    "            out += 1\n",
    "    if not out:\n",
    "        print(\"neutral\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "joy (92.2)%\n"
     ]
    }
   ],
   "source": [
    "txt = \"I enjoy having a good icecream.\"\n",
    "predict_emotions(txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "joy (91.9)%\n"
     ]
    }
   ],
   "source": [
    "# the model works even with typos, substitutions, and emojis!\n",
    "txt = \"I enjoy hving a g00d ic3cream!!! 🍦\"\n",
    "predict_emotions(txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save model for TFJS\n",
    "\n",
    "We have a RETVec implementation in TFJS, so to save the model, we can remove the RETVecTokenizer layer and save the rest of the model. You can then convert the model to tfjs using the [TensorFlow JS converter](https://www.tensorflow.org/js/tutorials/conversion/import_saved_model). Then, head over to the TFJS colab to see how to use retvec js and deploy your model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_2\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_1 (InputLayer)        [(None, 128, 256)]        0         \n",
      "                                                                 \n",
      " bidirectional_2 (Bidirecti  (None, 128, 128)          164352    \n",
      " onal)                                                           \n",
      "                                                                 \n",
      " bidirectional_3 (Bidirecti  (None, 128)               98816     \n",
      " onal)                                                           \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 28)                3612      \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 266780 (1.02 MB)\n",
      "Trainable params: 266780 (1.02 MB)\n",
      "Non-trainable params: 0 (0.00 Byte)\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "tfjs_model = tf.keras.Model(\n",
    "    model.layers[2].input,\n",
    "    model.output,\n",
    ")\n",
    "tfjs_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: demo_models/emotion_model_tfjs/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: demo_models/emotion_model_tfjs/assets\n"
     ]
    }
   ],
   "source": [
    "save_path = 'demo_models/emotion_model_tfjs'\n",
    "tfjs_model.save(save_path, include_optimizer=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert model to tfjs using tensorflowjs_converter\n",
    "! tensorflowjs_converter demo_models/emotion_model_tfjs  demo_models/emotion_model_tfjs_converted"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.12 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "e7370f93d1d0cde622a1f8e1c04877d8463912d04d973331ad4851f04de6915a"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}