notebooks/bqml-model-monitoring-tutorial.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cW621uZrxLVM"
      },
      "source": [
        "\u003c!--- header table ---\u003e\n",
        "\u003ctable align=\"left\"\u003e\n",
        "  \u003ctd style=\"text-align: center\"\u003e\n",
        "    \u003ca href=\"https://colab.research.google.com/github/GoogleCloudPlatform/bigquery-ml-utils/blob/master/notebooks/bqml-model-monitoring-tutorial.ipynb\"\u003e\n",
        "      \u003cimg src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"\u003e\n",
        "      \u003cbr\u003eRun in\u003cbr\u003eColab\n",
        "    \u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd style=\"text-align: center\"\u003e\n",
        "    \u003ca href=\"https://console.cloud.google.com/vertex-ai/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fbigquery-ml-utils%2Fmaster%2Fnotebooks%2Fbqml-model-monitoring-tutorial.ipynb\"\u003e\n",
        "      \u003cimg width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"\u003e\n",
        "      \u003cbr\u003eRun in\u003cbr\u003eColab Enterprise\n",
        "    \u003c/a\u003e\n",
        "  \u003c/td\u003e      \n",
        "  \u003ctd style=\"text-align: center\"\u003e\n",
        "    \u003ca href=\"https://github.com/GoogleCloudPlatform/bigquery-ml-utils/blob/master/notebooks/bqml-model-monitoring-tutorial.ipynb\"\u003e\n",
        "      \u003cimg src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"\u003e\n",
        "      \u003cbr\u003eView on\u003cbr\u003eGitHub\n",
        "    \u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd style=\"text-align: center\"\u003e\n",
        "    \u003ca href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/bigquery-ml-utils/master/notebooks/bqml-model-monitoring-tutorial.ipynb\"\u003e\n",
        "      \u003cimg src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"\u003e\n",
        "      \u003cbr\u003eOpen in\u003cbr\u003eVertex AI Workbench\n",
        "    \u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jwgBsT4ZeFWn"
      },
      "source": [
        "# BigQuery ML (BQML) - Model Monitoring Tutorial \n",
        "\n",
        "This notebook provides an in-depth look at model monitoring with BigQuery ML that accompanies the [blog announcing these features](link here). It complements the introductory notebook, ['BigQuery ML (BQML) - Model Monitoring Introduction'](./bqml-model-monitoring-introduction.ipynb), and delves deeper into various monitoring techniques and tools.\n",
        "\n",
        "**Key Areas Covered:**\n",
        "\n",
        "* **Data Preparation and Understanding:**\n",
        "    * Employing `ML.DESCRIBE_DATA` to analyze feature distributions and identify potential data quality issues.\n",
        "    * Splitting data into training, evaluation, and test sets while preserving data balance.\n",
        "* **Model Training and Evaluation:**\n",
        "    * Training a random forest model using `CREATE MODEL` with the `TRANSFORM` clause for incorporating feature engineering.\n",
        "    * Evaluating model performance using `ML.EVALUATE` and `ML.CONFUSION_MATRIX` on different data splits.\n",
        "* **Model Serving and Data Drift Simulation:**\n",
        "    * Generating predictions with `ML.PREDICT` and simulating data drift over time to represent real-world scenarios.\n",
        "* **Model Monitoring Techniques:**\n",
        "    * Implementing `ML.VALIDATE_DATA_SKEW` to detect changes between training and serving data distributions.\n",
        "    * Using `ML.VALIDATE_DATA_DRIFT` to identify gradual shifts in the serving data distribution over time.\n",
        "    * Understanding the metrics used for monitoring: L-infinity distance and Jensen-Shannon divergence.\n",
        "* **TensorFlow Data Validation (TFDV) Integration:**\n",
        "    * Leveraging `ML.TFDV_DESCRIBE` and `ML.TFDV_VALIDATE` to generate descriptive statistics and detect anomalies.\n",
        "    * Visualizing feature distributions using `tfdv.visualize_statistics` for comparative analysis.\n",
        "* **Monitoring Feature Attributions and Drift:**\n",
        "    * Employing `ML.EXPLAIN_PREDICT` to obtain feature attributions and understand feature importance for predictions.\n",
        "    * Monitoring drift in feature attributions over time to gain insights into model behavior changes. \n",
        "* **Continuous Monitoring and Automation:**\n",
        "    * Building a BigQuery SQL job using procedural language to automate skew and drift detection.\n",
        "    * Incorporating alerting mechanisms and model retraining when anomalies are detected.\n",
        "    * Discussing options for automating monitoring jobs:\n",
        "        * BigQuery Scheduled Queries\n",
        "        * Dataform\n",
        "        * Workflows\n",
        "        * Cloud Composer\n",
        "        * Vertex AI Pipelines\n",
        "* **Model Monitoring for Online Inference with Vertex AI:**\n",
        "    * Deploying the trained BigQuery ML model to Vertex AI for online predictions. \n",
        "    * Enabling request-response logging to BigQuery for continuous monitoring of online predictions.\n",
        "\n",
        "**Target Audience:**\n",
        "\n",
        "This tutorial is designed for data scientists, machine learning engineers, and individuals with a basic understanding of BigQuery, SQL, and machine learning concepts who want to delve deeper into model monitoring with BigQuery ML. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J_-1oCZ7oQe_"
      },
      "source": [
        "---\n",
        "## Colab Setup\n",
        "\n",
        "When running this notebook in [Colab](https://colab.google/) or [Colab Enterprise](https://cloud.google.com/colab/docs/introduction), this section will authenticate to GCP (follow prompts in the popup) and set the current project for the session."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "executionInfo": {
          "elapsed": 178,
          "status": "ok",
          "timestamp": 1710716382256,
          "user": {
            "displayName": "Mike Henderson",
            "userId": "07691629187611687318"
          },
          "user_tz": 240
        },
        "id": "bUeb3Vz9c-cz",
        "tags": []
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = 'statmike-mlops-349915' # replace with project ID"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "executionInfo": {
          "elapsed": 1769,
          "status": "ok",
          "timestamp": 1710716384851,
          "user": {
            "displayName": "Mike Henderson",
            "userId": "07691629187611687318"
          },
          "user_tz": 240
        },
        "id": "ztC0h14fc-fr",
        "outputId": "16d1144f-1671-4919-edba-31bc197ffc7b",
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Not a Colab Environment\n"
          ]
        }
      ],
      "source": [
        "try:\n",
        "    from google.colab import auth\n",
        "    auth.authenticate_user(project_id = PROJECT_ID)\n",
        "    print('Colab authorized to GCP')\n",
        "except Exception:\n",
        "    print('Not a Colab Environment')\n",
        "    pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5vf7Uf0HxLVM"
      },
      "source": [
        "---\n",
        "## Installs\n",
        "\n",
        "The list `packages` contains tuples of package import names and install names.  If the import name is not found then the install name is used to install quitely for the current user."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "tags": [],
        "id": "iB13Nen6xLVM"
      },
      "outputs": [],
      "source": [
        "# tuples of (import name, install name, min_version)\n",
        "packages = [\n",
        "    ('tensorflow_data_validation', 'tensorflow-data-validation'),\n",
        "    ('tensorflow_metadata', 'tensorflow-metadata'),\n",
        "    ('google.cloud.aiplatform', 'google-cloud-aiplatform'),\n",
        "    ('google.cloud.bigquery', 'google-cloud-bigquery'),\n",
        "    ('google.cloud.bigquery_datatransfer', 'google-cloud-bigquery-datatransfer'),\n",
        "    ('numpy', 'numpy'),\n",
        "    ('pandas', 'pandas')\n",
        "]\n",
        "\n",
        "import importlib\n",
        "install = False\n",
        "for package in packages:\n",
        "    if not importlib.util.find_spec(package[0]):\n",
        "        print(f'installing package {package[1]}')\n",
        "        install = True\n",
        "        !pip install {package[1]} -U -q --user\n",
        "    elif len(package) == 3:\n",
        "        if importlib.metadata.version(package[0]) \u003c package[2]:\n",
        "            print(f'updating package {package[1]}')\n",
        "            install = True\n",
        "            !pip install {package[1]} -U -q --user"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hiR5YHsYxLVM"
      },
      "source": [
        "### Restart Kernel (If Installs Occured)\n",
        "After a kernel restart the code submission can start with the next cell after this one."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "tags": [],
        "id": "gLvha18ExLVM"
      },
      "outputs": [],
      "source": [
        "if install:\n",
        "    import IPython\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yt5uZFvRoWLp"
      },
      "source": [
        "---\n",
        "## Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sssVoM75xLVM"
      },
      "source": [
        "Packages:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "tags": [],
        "id": "tC6iY-CBxLVM"
      },
      "outputs": [],
      "source": [
        "import json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y1HhwZO6xLVM"
      },
      "source": [
        "Clients:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "executionInfo": {
          "elapsed": 159,
          "status": "ok",
          "timestamp": 1710716582376,
          "user": {
            "displayName": "Mike Henderson",
            "userId": "07691629187611687318"
          },
          "user_tz": 240
        },
        "id": "14nUE43Oc-ip",
        "outputId": "55b2c2b1-b6ad-42cc-b3cc-d6e3afcaa071",
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The google.cloud.bigquery extension is already loaded. To reload it, use:\n",
            "  %reload_ext google.cloud.bigquery\n"
          ]
        }
      ],
      "source": [
        "%load_ext google.cloud.bigquery"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VE0zy4C_xLVM"
      },
      "source": [
        "Prepare the code below for your environment.\n",
        "\n",
        "This notebook takes advantage of the [BigQuery IPython magic](https://cloud.google.com/python/docs/reference/bigquery/latest/magics) for legibility and ease of copy/pasting to BigQuery SQL editor.  If this notebook is being used from an environment that can run notebooks it needs further preparation: Colab, Colab Enterprise, Vertex AI Workbench Instances, or BigQuery Studio with a Python Notebook.  The SQL code in these cells uses the fully qualified [BigQuery table](https://cloud.google.com/bigquery/docs/tables-intro) names in the form `projectname.datasetname.tablename`.  Prepare for your environment by:\n",
        "- Edit \u003e Find\n",
        "    - Find: `statmike-mlops-349915`\n",
        "    - Replace: `\u003cyour project id\u003e`\n",
        "    - Replace All"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "by8oK6c0xLVM"
      },
      "source": [
        "---\n",
        "## BigQuery Source Data\n",
        "\n",
        "This project will uses the following data source as a tutorial.  It is good for examples like this because it is small, has few columns, but contains categorial and numerical data types.\n",
        "\n",
        "The source table is a BigQuery Public Dataset table.  The following cell uses the [BigQuery IPython magic](https://cloud.google.com/python/docs/reference/bigquery/latest/magics) to retrieve 5 rows of the table for review.  This data is known as [Palmer Penguins](https://allisonhorst.github.io/palmerpenguins/) data: \n",
        "\n",
        "```\n",
        "@Manual{,\n",
        "  title = {palmerpenguins: Palmer Archipelago (Antarctica) penguin data},\n",
        "  author = {Allison Marie Horst and Alison Presmanes Hill and Kristen B Gorman},\n",
        "  year = {2020},\n",
        "  note = {R package version 0.1.0},\n",
        "  doi = {10.5281/zenodo.3960218},\n",
        "  url = {https://allisonhorst.github.io/palmerpenguins/},\n",
        "}\n",
        "```\n",
        "\n",
        "\n",
        "There are 334 observations of 4 numerical features (culman length, culmen depth, flipper length, body mass) and 2  categorical features (island, sex) that represent 3 species of penguins."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "tags": [],
        "id": "-VC85MI2xLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "11ae24b896a74d1fbf5d1c451abf9618",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6cd18bd2de3842e7b3d48ace3c05cb26",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003especies\u003c/th\u003e\n",
              "      \u003cth\u003eisland\u003c/th\u003e\n",
              "      \u003cth\u003eculmen_length_mm\u003c/th\u003e\n",
              "      \u003cth\u003eculmen_depth_mm\u003c/th\u003e\n",
              "      \u003cth\u003eflipper_length_mm\u003c/th\u003e\n",
              "      \u003cth\u003ebody_mass_g\u003c/th\u003e\n",
              "      \u003cth\u003esex\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003eDream\u003c/td\u003e\n",
              "      \u003ctd\u003e36.6\u003c/td\u003e\n",
              "      \u003ctd\u003e18.4\u003c/td\u003e\n",
              "      \u003ctd\u003e184.0\u003c/td\u003e\n",
              "      \u003ctd\u003e3475.0\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003eDream\u003c/td\u003e\n",
              "      \u003ctd\u003e39.8\u003c/td\u003e\n",
              "      \u003ctd\u003e19.1\u003c/td\u003e\n",
              "      \u003ctd\u003e184.0\u003c/td\u003e\n",
              "      \u003ctd\u003e4650.0\u003c/td\u003e\n",
              "      \u003ctd\u003eMALE\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003eDream\u003c/td\u003e\n",
              "      \u003ctd\u003e40.9\u003c/td\u003e\n",
              "      \u003ctd\u003e18.9\u003c/td\u003e\n",
              "      \u003ctd\u003e184.0\u003c/td\u003e\n",
              "      \u003ctd\u003e3900.0\u003c/td\u003e\n",
              "      \u003ctd\u003eMALE\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eChinstrap penguin (Pygoscelis antarctica)\u003c/td\u003e\n",
              "      \u003ctd\u003eDream\u003c/td\u003e\n",
              "      \u003ctd\u003e46.5\u003c/td\u003e\n",
              "      \u003ctd\u003e17.9\u003c/td\u003e\n",
              "      \u003ctd\u003e192.0\u003c/td\u003e\n",
              "      \u003ctd\u003e3500.0\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003eDream\u003c/td\u003e\n",
              "      \u003ctd\u003e37.3\u003c/td\u003e\n",
              "      \u003ctd\u003e16.8\u003c/td\u003e\n",
              "      \u003ctd\u003e192.0\u003c/td\u003e\n",
              "      \u003ctd\u003e3000.0\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                                     species island  culmen_length_mm  \\\n",
              "0        Adelie Penguin (Pygoscelis adeliae)  Dream              36.6   \n",
              "1        Adelie Penguin (Pygoscelis adeliae)  Dream              39.8   \n",
              "2        Adelie Penguin (Pygoscelis adeliae)  Dream              40.9   \n",
              "3  Chinstrap penguin (Pygoscelis antarctica)  Dream              46.5   \n",
              "4        Adelie Penguin (Pygoscelis adeliae)  Dream              37.3   \n",
              "\n",
              "   culmen_depth_mm  flipper_length_mm  body_mass_g     sex  \n",
              "0             18.4              184.0       3475.0  FEMALE  \n",
              "1             19.1              184.0       4650.0    MALE  \n",
              "2             18.9              184.0       3900.0    MALE  \n",
              "3             17.9              192.0       3500.0  FEMALE  \n",
              "4             16.8              192.0       3000.0  FEMALE  "
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM `bigquery-public-data.ml_datasets.penguins`\n",
        "LIMIT 5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rEON8kYExLVM"
      },
      "source": [
        "---\n",
        "## **Goal: Train, Serve, And Monitor A Model**\n",
        "\n",
        "**Train A Model**\n",
        "\n",
        "In this workflow our goal is to train and operationalize a machine leanring model that can classify unidentified penguins into the correct `species` using only measurements (`culmen_length_mm`, `culmen_depth_mm`, `flipper_length_mm`, `body_mass_g`), location (`island`) and gender (`sex`).  That means serving predictions, monitoring features, and potentially retraining the model.\n",
        "\n",
        "**Operationalize The Model**\n",
        "\n",
        "Taking a trained model to production introduces challenges. The model needs to be regularly evaluated to understand it's performance once put into production. This can be challenging as actual values used to assess the accuracy of predictions may arrive later than the predictions. Continuous monitoring bridges this gap with early signals in two key ways. First, monitoring the distributions of each of the models features for shift (called **skew**) from the training data to the current data. Then, also monitoring for any gradual **drift** in the distribution over time. Setting thresholds and getting alerts for features on each of this distribution changes gives early signals that the model needs to be carefully considered for evaluation and re-training to ensure effectiveness. If the evaluation can be automated then this can even extend to **continuous training**. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "paWh6AySxLVM"
      },
      "source": [
        "---\n",
        "## Data Preparation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "sb4RKEHbxLVM"
      },
      "source": [
        "### Understand The Data Source With `ML.DESCRIBE_DATA`\n",
        "\n",
        "Reviewing a few records, like above, gives a good sense of how the data is arranged. Before proceeding with machine learning techniques it is important to understand more about these raw columns.  Are they ready to use a features in a model or is some form of feature engineering needed first?  For this, the distribution of values is an important starting point.  \n",
        "\n",
        "While SQL could be used to look at the distribution, it would be a time consuming process and requires different techniques for different data types like numerical, string, boolean, dates, times, array and struct version of these, and arrays of structs.\n",
        "\n",
        "To make this process fast and simple, the new [`ML.DESCRIBE_DATA`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-describe-data)\n",
        "\n",
        "function is used to get a single row for each column the describes the data distribution:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "tags": [],
        "id": "h3uGI-ppxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "890fe470362344238cde45efa1dc74d6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8ad547d0cd604066b8f299cc23818f3e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003ename\u003c/th\u003e\n",
              "      \u003cth\u003enum_rows\u003c/th\u003e\n",
              "      \u003cth\u003enum_nulls\u003c/th\u003e\n",
              "      \u003cth\u003enum_zeros\u003c/th\u003e\n",
              "      \u003cth\u003emin\u003c/th\u003e\n",
              "      \u003cth\u003emax\u003c/th\u003e\n",
              "      \u003cth\u003emean\u003c/th\u003e\n",
              "      \u003cth\u003estddev\u003c/th\u003e\n",
              "      \u003cth\u003emedian\u003c/th\u003e\n",
              "      \u003cth\u003equantiles\u003c/th\u003e\n",
              "      \u003cth\u003eunique\u003c/th\u003e\n",
              "      \u003cth\u003eavg_string_length\u003c/th\u003e\n",
              "      \u003cth\u003enum_values\u003c/th\u003e\n",
              "      \u003cth\u003etop_values\u003c/th\u003e\n",
              "      \u003cth\u003emin_array_length\u003c/th\u003e\n",
              "      \u003cth\u003emax_array_length\u003c/th\u003e\n",
              "      \u003cth\u003eavg_array_length\u003c/th\u003e\n",
              "      \u003cth\u003etotal_array_length\u003c/th\u003e\n",
              "      \u003cth\u003earray_length_quantiles\u003c/th\u003e\n",
              "      \u003cth\u003edimension\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003ebody_mass_g\u003c/td\u003e\n",
              "      \u003ctd\u003e344\u003c/td\u003e\n",
              "      \u003ctd\u003e2\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e2700\u003c/td\u003e\n",
              "      \u003ctd\u003e6300\u003c/td\u003e\n",
              "      \u003ctd\u003e4201.754386\u003c/td\u003e\n",
              "      \u003ctd\u003e800.781229\u003c/td\u003e\n",
              "      \u003ctd\u003e4000.0\u003c/td\u003e\n",
              "      \u003ctd\u003e[2700.0, 4000.0, 6300.0]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e342\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_depth_mm\u003c/td\u003e\n",
              "      \u003ctd\u003e344\u003c/td\u003e\n",
              "      \u003ctd\u003e2\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e13.1\u003c/td\u003e\n",
              "      \u003ctd\u003e21.5\u003c/td\u003e\n",
              "      \u003ctd\u003e17.151170\u003c/td\u003e\n",
              "      \u003ctd\u003e1.971904\u003c/td\u003e\n",
              "      \u003ctd\u003e17.3\u003c/td\u003e\n",
              "      \u003ctd\u003e[13.1, 17.3, 21.5]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e342\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003e344\u003c/td\u003e\n",
              "      \u003ctd\u003e2\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e32.1\u003c/td\u003e\n",
              "      \u003ctd\u003e59.6\u003c/td\u003e\n",
              "      \u003ctd\u003e43.921930\u003c/td\u003e\n",
              "      \u003ctd\u003e5.451596\u003c/td\u003e\n",
              "      \u003ctd\u003e44.1\u003c/td\u003e\n",
              "      \u003ctd\u003e[32.1, 44.1, 59.6]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e342\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eflipper_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003e344\u003c/td\u003e\n",
              "      \u003ctd\u003e2\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e172\u003c/td\u003e\n",
              "      \u003ctd\u003e231\u003c/td\u003e\n",
              "      \u003ctd\u003e200.915205\u003c/td\u003e\n",
              "      \u003ctd\u003e14.041141\u003c/td\u003e\n",
              "      \u003ctd\u003e197.0\u003c/td\u003e\n",
              "      \u003ctd\u003e[172.0, 197.0, 231.0]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e342\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003e344\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003eTorgersen\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e3\u003c/td\u003e\n",
              "      \u003ctd\u003e6.093023\u003c/td\u003e\n",
              "      \u003ctd\u003e344\u003c/td\u003e\n",
              "      \u003ctd\u003e[{'value': 'Biscoe', 'count': 168}]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e5\u003c/th\u003e\n",
              "      \u003ctd\u003esex\u003c/td\u003e\n",
              "      \u003ctd\u003e344\u003c/td\u003e\n",
              "      \u003ctd\u003e10\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e.\u003c/td\u003e\n",
              "      \u003ctd\u003eMALE\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e3\u003c/td\u003e\n",
              "      \u003ctd\u003e4.979042\u003c/td\u003e\n",
              "      \u003ctd\u003e334\u003c/td\u003e\n",
              "      \u003ctd\u003e[{'value': 'MALE', 'count': 168}]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e6\u003c/th\u003e\n",
              "      \u003ctd\u003especies\u003c/td\u003e\n",
              "      \u003ctd\u003e344\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e3\u003c/td\u003e\n",
              "      \u003ctd\u003e35.465116\u003c/td\u003e\n",
              "      \u003ctd\u003e344\u003c/td\u003e\n",
              "      \u003ctd\u003e[{'value': 'Adelie Penguin (Pygoscelis adeliae...\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                name  num_rows  num_nulls  num_zeros  \\\n",
              "0        body_mass_g       344          2          0   \n",
              "1    culmen_depth_mm       344          2          0   \n",
              "2   culmen_length_mm       344          2          0   \n",
              "3  flipper_length_mm       344          2          0   \n",
              "4             island       344          0       \u003cNA\u003e   \n",
              "5                sex       344         10       \u003cNA\u003e   \n",
              "6            species       344          0       \u003cNA\u003e   \n",
              "\n",
              "                                   min                                max  \\\n",
              "0                                 2700                               6300   \n",
              "1                                 13.1                               21.5   \n",
              "2                                 32.1                               59.6   \n",
              "3                                  172                                231   \n",
              "4                               Biscoe                          Torgersen   \n",
              "5                                    .                               MALE   \n",
              "6  Adelie Penguin (Pygoscelis adeliae)  Gentoo penguin (Pygoscelis papua)   \n",
              "\n",
              "          mean      stddev  median                 quantiles  unique  \\\n",
              "0  4201.754386  800.781229  4000.0  [2700.0, 4000.0, 6300.0]    \u003cNA\u003e   \n",
              "1    17.151170    1.971904    17.3        [13.1, 17.3, 21.5]    \u003cNA\u003e   \n",
              "2    43.921930    5.451596    44.1        [32.1, 44.1, 59.6]    \u003cNA\u003e   \n",
              "3   200.915205   14.041141   197.0     [172.0, 197.0, 231.0]    \u003cNA\u003e   \n",
              "4          NaN         NaN     NaN                        []       3   \n",
              "5          NaN         NaN     NaN                        []       3   \n",
              "6          NaN         NaN     NaN                        []       3   \n",
              "\n",
              "   avg_string_length  num_values  \\\n",
              "0                NaN         342   \n",
              "1                NaN         342   \n",
              "2                NaN         342   \n",
              "3                NaN         342   \n",
              "4           6.093023         344   \n",
              "5           4.979042         334   \n",
              "6          35.465116         344   \n",
              "\n",
              "                                          top_values  min_array_length  \\\n",
              "0                                                 []              \u003cNA\u003e   \n",
              "1                                                 []              \u003cNA\u003e   \n",
              "2                                                 []              \u003cNA\u003e   \n",
              "3                                                 []              \u003cNA\u003e   \n",
              "4                [{'value': 'Biscoe', 'count': 168}]              \u003cNA\u003e   \n",
              "5                  [{'value': 'MALE', 'count': 168}]              \u003cNA\u003e   \n",
              "6  [{'value': 'Adelie Penguin (Pygoscelis adeliae...              \u003cNA\u003e   \n",
              "\n",
              "   max_array_length  avg_array_length  total_array_length  \\\n",
              "0              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "1              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "2              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "3              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "4              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "5              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "6              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "\n",
              "  array_length_quantiles  dimension  \n",
              "0                     []       \u003cNA\u003e  \n",
              "1                     []       \u003cNA\u003e  \n",
              "2                     []       \u003cNA\u003e  \n",
              "3                     []       \u003cNA\u003e  \n",
              "4                     []       \u003cNA\u003e  \n",
              "5                     []       \u003cNA\u003e  \n",
              "6                     []       \u003cNA\u003e  "
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.DESCRIBE_DATA(\n",
        "  TABLE `bigquery-public-data.ml_datasets.penguins`\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_1uEtv59xLVM"
      },
      "source": [
        "Some observations:\n",
        "- All columns have `num_rows` and `num_nulls`\n",
        "- Numerical columns have `min`, `max`, `stddev`, `median`, and `quantiles` showing the distribution of the data in the column\n",
        "- Categorical (string) columns have `min`, `max` values representing the class levels and a `unique` column show how many class levels\n",
        "\n",
        "Data quality observation:\n",
        "- The column `sex` has both missing values (10 indicated in `num_nulls`) as well as a `min` values of `.`.  This appears to be a data quality issue that needs fixing."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "Pa16jIwvxLVM"
      },
      "source": [
        "### Create A BigQuery Dataset\n",
        "\n",
        "Create a new [BigQuery Dataset](https://cloud.google.com/bigquery/docs/datasets) as a working location for this workflow:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "tags": [],
        "id": "pY6ROCpTxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "98e6634ab9bc49558c992df12f42eb6a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: []\n",
              "Index: []"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "CREATE SCHEMA IF NOT EXISTS `statmike-mlops-349915.bqml_model_monitoring`\n",
        "    OPTIONS(\n",
        "        location = 'US'\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ps7SPdc9xLVM"
      },
      "source": [
        "### Prepare The Source Data\n",
        "\n",
        "Make a copy of the table source in the new BigQuery dataset with fixes applied to the data quality issue identified for the `sex` column with values of `.`.\n",
        "\n",
        "\u003e Note: A copy is being made in this case because the source project is `bigquery-public` which is not editable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "tags": [],
        "id": "mQPXcAEPxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ff1cbc3a9d754a0b843587da95212af8",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: []\n",
              "Index: []"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "CREATE OR REPLACE TABLE `statmike-mlops-349915.bqml_model_monitoring.training` AS\n",
        "    SELECT * EXCEPT(sex),\n",
        "        CASE WHEN sex = '.' THEN NULL ELSE sex END AS sex\n",
        "    FROM `bigquery-public-data.ml_datasets.penguins`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idsnMeXMxLVM"
      },
      "source": [
        "### Split The Data\n",
        "\n",
        "Depending on the ML technique, it may be desired to split the data into partitions for training, evaluation, and testing (in this case monitoring examples). \n",
        "\n",
        "The following cell creates a version with a new column column named `splits` with values [`TRAIN`, `EVAL`, `TEST`].  The data is first grouped (stratified) by the values of `species` and `island` to preserve any imbalance across the groups. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "tags": [],
        "id": "3sJUHvFmxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b0ed862680434424a60e0bff6ce2e8db",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: []\n",
              "Index: []"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "CREATE OR REPLACE TABLE `statmike-mlops-349915.bqml_model_monitoring.training_split` AS\n",
        "    WITH\n",
        "        # randomized numbering within groups (species, island)\n",
        "        RANDOM AS (\n",
        "            SELECT *,\n",
        "                ROW_NUMBER() OVER (PARTITION BY species, island ORDER BY RAND()) AS sequence\n",
        "            FROM `statmike-mlops-349915.bqml_model_monitoring.training`\n",
        "        ),\n",
        "        # get group sizes\n",
        "        GROUP_SIZES AS (\n",
        "            SELECT species, island, COUNT(*) AS count\n",
        "            FROM `statmike-mlops-349915.bqml_model_monitoring.training`\n",
        "            GROUP BY species, island\n",
        "        )\n",
        "    SELECT\n",
        "        * EXCEPT(sequence, count),\n",
        "        CASE\n",
        "            WHEN sequence \u003c= CEIL(.2 * count) AND species is not Null THEN 'TEST'\n",
        "            WHEN sequence \u003c= CEIL(.3 * count) THEN 'EVAL'\n",
        "            ELSE 'TRAIN'\n",
        "        END AS splits\n",
        "    FROM RANDOM\n",
        "    LEFT OUTER JOIN GROUP_SIZES USING(species, island)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lh7iL1hzxLVM"
      },
      "source": [
        "Review the data by `splits`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "tags": [],
        "id": "PtHxdKHVxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d121ea34a0314a988fe8ff6607f1cd43",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2e10e8f9e5654630bb224d5fd1ad20ea",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003especies\u003c/th\u003e\n",
              "      \u003cth\u003eisland\u003c/th\u003e\n",
              "      \u003cth\u003ecount\u003c/th\u003e\n",
              "      \u003cth\u003eTRAIN_PCT\u003c/th\u003e\n",
              "      \u003cth\u003eEVAL_PCT\u003c/th\u003e\n",
              "      \u003cth\u003eTEST_PCT\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003e124\u003c/td\u003e\n",
              "      \u003ctd\u003e69.354839\u003c/td\u003e\n",
              "      \u003ctd\u003e10.483871\u003c/td\u003e\n",
              "      \u003ctd\u003e20.161290\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003eTorgersen\u003c/td\u003e\n",
              "      \u003ctd\u003e52\u003c/td\u003e\n",
              "      \u003ctd\u003e69.230769\u003c/td\u003e\n",
              "      \u003ctd\u003e9.615385\u003c/td\u003e\n",
              "      \u003ctd\u003e21.153846\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003eDream\u003c/td\u003e\n",
              "      \u003ctd\u003e56\u003c/td\u003e\n",
              "      \u003ctd\u003e69.642857\u003c/td\u003e\n",
              "      \u003ctd\u003e8.928571\u003c/td\u003e\n",
              "      \u003ctd\u003e21.428571\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003e44\u003c/td\u003e\n",
              "      \u003ctd\u003e68.181818\u003c/td\u003e\n",
              "      \u003ctd\u003e11.363636\u003c/td\u003e\n",
              "      \u003ctd\u003e20.454545\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eChinstrap penguin (Pygoscelis antarctica)\u003c/td\u003e\n",
              "      \u003ctd\u003eDream\u003c/td\u003e\n",
              "      \u003ctd\u003e68\u003c/td\u003e\n",
              "      \u003ctd\u003e69.117647\u003c/td\u003e\n",
              "      \u003ctd\u003e10.294118\u003c/td\u003e\n",
              "      \u003ctd\u003e20.588235\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                                     species     island  count  TRAIN_PCT  \\\n",
              "0          Gentoo penguin (Pygoscelis papua)     Biscoe    124  69.354839   \n",
              "1        Adelie Penguin (Pygoscelis adeliae)  Torgersen     52  69.230769   \n",
              "2        Adelie Penguin (Pygoscelis adeliae)      Dream     56  69.642857   \n",
              "3        Adelie Penguin (Pygoscelis adeliae)     Biscoe     44  68.181818   \n",
              "4  Chinstrap penguin (Pygoscelis antarctica)      Dream     68  69.117647   \n",
              "\n",
              "    EVAL_PCT   TEST_PCT  \n",
              "0  10.483871  20.161290  \n",
              "1   9.615385  21.153846  \n",
              "2   8.928571  21.428571  \n",
              "3  11.363636  20.454545  \n",
              "4  10.294118  20.588235  "
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT species, island,\n",
        "    COUNT(*) as count,\n",
        "    100 * COUNTIF(splits = 'TRAIN')/COUNT(*) AS TRAIN_PCT,\n",
        "    100 * COUNTIF(splits = 'EVAL')/COUNT(*) AS EVAL_PCT,\n",
        "    100 * COUNTIF(splits = 'TEST')/COUNT(*) AS TEST_PCT\n",
        "FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "GROUP BY species, island"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pzmN3a6ixLVM"
      },
      "source": [
        "### Review the Training Split With: `ML.DESCRIBE_DATA`\n",
        "\n",
        "Use [`ML.DESCRIBE_DATA`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-describe-data) to first review the training split of the source data.  This time, some of the additional parameters of the function are useful:\n",
        "- `top_k`: get the top 3 most frequent categories for string columns (default = 1)\n",
        "- `num_quantiles`: get 4 quantiles for numerical columns (default = 2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "tags": [],
        "id": "6BPzZK8yxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "223d32a801e443108c9b40f81a19403e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b5fc95abdeae433c9f7da0c086d53803",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003ename\u003c/th\u003e\n",
              "      \u003cth\u003enum_rows\u003c/th\u003e\n",
              "      \u003cth\u003enum_nulls\u003c/th\u003e\n",
              "      \u003cth\u003enum_zeros\u003c/th\u003e\n",
              "      \u003cth\u003emin\u003c/th\u003e\n",
              "      \u003cth\u003emax\u003c/th\u003e\n",
              "      \u003cth\u003emean\u003c/th\u003e\n",
              "      \u003cth\u003estddev\u003c/th\u003e\n",
              "      \u003cth\u003emedian\u003c/th\u003e\n",
              "      \u003cth\u003equantiles\u003c/th\u003e\n",
              "      \u003cth\u003eunique\u003c/th\u003e\n",
              "      \u003cth\u003eavg_string_length\u003c/th\u003e\n",
              "      \u003cth\u003enum_values\u003c/th\u003e\n",
              "      \u003cth\u003etop_values\u003c/th\u003e\n",
              "      \u003cth\u003emin_array_length\u003c/th\u003e\n",
              "      \u003cth\u003emax_array_length\u003c/th\u003e\n",
              "      \u003cth\u003eavg_array_length\u003c/th\u003e\n",
              "      \u003cth\u003etotal_array_length\u003c/th\u003e\n",
              "      \u003cth\u003earray_length_quantiles\u003c/th\u003e\n",
              "      \u003cth\u003edimension\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003ebody_mass_g\u003c/td\u003e\n",
              "      \u003ctd\u003e238\u003c/td\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e2850\u003c/td\u003e\n",
              "      \u003ctd\u003e6300\u003c/td\u003e\n",
              "      \u003ctd\u003e4197.784810\u003c/td\u003e\n",
              "      \u003ctd\u003e794.428017\u003c/td\u003e\n",
              "      \u003ctd\u003e4050.0\u003c/td\u003e\n",
              "      \u003ctd\u003e[2850.0, 3550.0, 4050.0, 4750.0, 6300.0]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e237\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_depth_mm\u003c/td\u003e\n",
              "      \u003ctd\u003e238\u003c/td\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e13.1\u003c/td\u003e\n",
              "      \u003ctd\u003e21.2\u003c/td\u003e\n",
              "      \u003ctd\u003e17.135443\u003c/td\u003e\n",
              "      \u003ctd\u003e1.979603\u003c/td\u003e\n",
              "      \u003ctd\u003e17.3\u003c/td\u003e\n",
              "      \u003ctd\u003e[13.1, 15.6, 17.3, 18.6, 21.2]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e237\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003e238\u003c/td\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e32.1\u003c/td\u003e\n",
              "      \u003ctd\u003e58\u003c/td\u003e\n",
              "      \u003ctd\u003e43.796203\u003c/td\u003e\n",
              "      \u003ctd\u003e5.300938\u003c/td\u003e\n",
              "      \u003ctd\u003e43.8\u003c/td\u003e\n",
              "      \u003ctd\u003e[32.1, 39.6, 44.0, 48.2, 58.0]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e237\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eflipper_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003e238\u003c/td\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e172\u003c/td\u003e\n",
              "      \u003ctd\u003e231\u003c/td\u003e\n",
              "      \u003ctd\u003e200.565401\u003c/td\u003e\n",
              "      \u003ctd\u003e13.765706\u003c/td\u003e\n",
              "      \u003ctd\u003e197.0\u003c/td\u003e\n",
              "      \u003ctd\u003e[172.0, 190.0, 197.0, 213.0, 231.0]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e237\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003e238\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003eTorgersen\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e3\u003c/td\u003e\n",
              "      \u003ctd\u003e6.092437\u003c/td\u003e\n",
              "      \u003ctd\u003e238\u003c/td\u003e\n",
              "      \u003ctd\u003e[{'value': 'Biscoe', 'count': 116}, {'value': ...\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e5\u003c/th\u003e\n",
              "      \u003ctd\u003esex\u003c/td\u003e\n",
              "      \u003ctd\u003e238\u003c/td\u003e\n",
              "      \u003ctd\u003e5\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "      \u003ctd\u003eMALE\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e2\u003c/td\u003e\n",
              "      \u003ctd\u003e5.055794\u003c/td\u003e\n",
              "      \u003ctd\u003e233\u003c/td\u003e\n",
              "      \u003ctd\u003e[{'value': 'FEMALE', 'count': 123}, {'value': ...\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e6\u003c/th\u003e\n",
              "      \u003ctd\u003especies\u003c/td\u003e\n",
              "      \u003ctd\u003e238\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e3\u003c/td\u003e\n",
              "      \u003ctd\u003e35.462185\u003c/td\u003e\n",
              "      \u003ctd\u003e238\u003c/td\u003e\n",
              "      \u003ctd\u003e[{'value': 'Adelie Penguin (Pygoscelis adeliae...\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "      \u003ctd\u003e[]\u003c/td\u003e\n",
              "      \u003ctd\u003e\u0026lt;NA\u0026gt;\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                name  num_rows  num_nulls  num_zeros  \\\n",
              "0        body_mass_g       238          1          0   \n",
              "1    culmen_depth_mm       238          1          0   \n",
              "2   culmen_length_mm       238          1          0   \n",
              "3  flipper_length_mm       238          1          0   \n",
              "4             island       238          0       \u003cNA\u003e   \n",
              "5                sex       238          5       \u003cNA\u003e   \n",
              "6            species       238          0       \u003cNA\u003e   \n",
              "\n",
              "                                   min                                max  \\\n",
              "0                                 2850                               6300   \n",
              "1                                 13.1                               21.2   \n",
              "2                                 32.1                                 58   \n",
              "3                                  172                                231   \n",
              "4                               Biscoe                          Torgersen   \n",
              "5                               FEMALE                               MALE   \n",
              "6  Adelie Penguin (Pygoscelis adeliae)  Gentoo penguin (Pygoscelis papua)   \n",
              "\n",
              "          mean      stddev  median                                 quantiles  \\\n",
              "0  4197.784810  794.428017  4050.0  [2850.0, 3550.0, 4050.0, 4750.0, 6300.0]   \n",
              "1    17.135443    1.979603    17.3            [13.1, 15.6, 17.3, 18.6, 21.2]   \n",
              "2    43.796203    5.300938    43.8            [32.1, 39.6, 44.0, 48.2, 58.0]   \n",
              "3   200.565401   13.765706   197.0       [172.0, 190.0, 197.0, 213.0, 231.0]   \n",
              "4          NaN         NaN     NaN                                        []   \n",
              "5          NaN         NaN     NaN                                        []   \n",
              "6          NaN         NaN     NaN                                        []   \n",
              "\n",
              "   unique  avg_string_length  num_values  \\\n",
              "0    \u003cNA\u003e                NaN         237   \n",
              "1    \u003cNA\u003e                NaN         237   \n",
              "2    \u003cNA\u003e                NaN         237   \n",
              "3    \u003cNA\u003e                NaN         237   \n",
              "4       3           6.092437         238   \n",
              "5       2           5.055794         233   \n",
              "6       3          35.462185         238   \n",
              "\n",
              "                                          top_values  min_array_length  \\\n",
              "0                                                 []              \u003cNA\u003e   \n",
              "1                                                 []              \u003cNA\u003e   \n",
              "2                                                 []              \u003cNA\u003e   \n",
              "3                                                 []              \u003cNA\u003e   \n",
              "4  [{'value': 'Biscoe', 'count': 116}, {'value': ...              \u003cNA\u003e   \n",
              "5  [{'value': 'FEMALE', 'count': 123}, {'value': ...              \u003cNA\u003e   \n",
              "6  [{'value': 'Adelie Penguin (Pygoscelis adeliae...              \u003cNA\u003e   \n",
              "\n",
              "   max_array_length  avg_array_length  total_array_length  \\\n",
              "0              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "1              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "2              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "3              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "4              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "5              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "6              \u003cNA\u003e               NaN                \u003cNA\u003e   \n",
              "\n",
              "  array_length_quantiles  dimension  \n",
              "0                     []       \u003cNA\u003e  \n",
              "1                     []       \u003cNA\u003e  \n",
              "2                     []       \u003cNA\u003e  \n",
              "3                     []       \u003cNA\u003e  \n",
              "4                     []       \u003cNA\u003e  \n",
              "5                     []       \u003cNA\u003e  \n",
              "6                     []       \u003cNA\u003e  "
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.DESCRIBE_DATA(\n",
        "    (\n",
        "        SELECT * EXCEPT(splits)\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "        WHERE splits = 'TRAIN'\n",
        "    ),\n",
        "    STRUCT(3 AS top_k, 4 AS num_quantiles)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "iztbfD-zxLVM"
      },
      "source": [
        "---\n",
        "## Model Training\n",
        "\n",
        "Create a model trained to classify `species` for the traning records.  Here, directly in BigQuery, the [`CREATE MODEL` statement for random forest](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create-random-forest) is used.  This uses the `TRANSFORM` clause within the model to embed user specified transfomations within the model for use during serving and model monitoring which will both be show later in this introduction.\n",
        "\n",
        "From reviewing the training data above, the following transformations will be tried:\n",
        "- Apply [`ML.ROBUST_SCALER`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-robust-scaler) to `body_mass_g`\n",
        "- Apply [`ML.STANDARD_SCALER`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-standard-scaler) to `culmen_depth_mm` and `culmen_length_mm`\n",
        "- Apply [`ML.QUANTILE_BUCKATIZE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-quantile-bucketize) to `flipper_length_mm`\n",
        "- Apply [`ML.IMPUTER`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-imputer) to `island`, and `sex`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "tags": [],
        "id": "Q0qzamrSxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6c23e91d8a93409bb313d64cc3fd6dec",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: []\n",
              "Index: []"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "CREATE OR REPLACE MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`\n",
        "    TRANSFORM(\n",
        "        ML.ROBUST_SCALER(body_mass_g) OVER() AS body_mass_g,\n",
        "        ML.STANDARD_SCALER(culmen_length_mm) OVER() AS culmen_length_mm,\n",
        "        ML.STANDARD_SCALER(culmen_depth_mm) OVER() AS culmen_depth_mm,\n",
        "        ML.QUANTILE_BUCKETIZE(flipper_length_mm, 3) OVER() AS flipper_length_mm,\n",
        "        ML.IMPUTER(sex, 'most_frequent') OVER() AS sex,\n",
        "        ML.IMPUTER(island, 'most_frequent') OVER() AS island,\n",
        "        species, split\n",
        "    )\n",
        "    OPTIONS(\n",
        "        MODEL_TYPE = 'RANDOM_FOREST_CLASSIFIER',\n",
        "        INPUT_LABEL_COLS = ['species'],\n",
        "        \n",
        "        # data specifics\n",
        "        DATA_SPLIT_METHOD = 'CUSTOM',\n",
        "        DATA_SPLIT_COL = 'split',\n",
        "        \n",
        "        # model specifics\n",
        "        AUTO_CLASS_WEIGHTS = FALSE,\n",
        "        NUM_PARALLEL_TREE= 150,\n",
        "        TREE_METHOD = 'HIST',\n",
        "        SUBSAMPLE = 0.85,\n",
        "        COLSAMPLE_BYTREE = 0.9,\n",
        "        ENABLE_GLOBAL_EXPLAIN = TRUE,\n",
        "        \n",
        "        # register model in Vertex AI For Online Serving\n",
        "        MODEL_REGISTRY = 'VERTEX_AI'\n",
        "    )\n",
        "AS\n",
        "    SELECT * EXCEPT(splits),\n",
        "        CASE WHEN splits = 'TRAIN' THEN FALSE\n",
        "        ELSE TRUE END AS split\n",
        "    FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "    WHERE splits != 'TEST'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "tags": [],
        "id": "pdxu3114xLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "78e39cb76e0f4bae8afb23e2baee6d75",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6c9ca06fce294a81b3b8b7c97714753b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003etraining_run\u003c/th\u003e\n",
              "      \u003cth\u003eiteration\u003c/th\u003e\n",
              "      \u003cth\u003eloss\u003c/th\u003e\n",
              "      \u003cth\u003eeval_loss\u003c/th\u003e\n",
              "      \u003cth\u003elearning_rate\u003c/th\u003e\n",
              "      \u003cth\u003eduration_ms\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e0.256961\u003c/td\u003e\n",
              "      \u003ctd\u003e0.262248\u003c/td\u003e\n",
              "      \u003ctd\u003e1.0\u003c/td\u003e\n",
              "      \u003ctd\u003e94391\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "   training_run  iteration      loss  eval_loss  learning_rate  duration_ms\n",
              "0             0          1  0.256961   0.262248            1.0        94391"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.TRAINING_INFO(MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`)\n",
        "ORDER BY iteration"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4YgK4WBSxLVM"
      },
      "source": [
        "### Evaluation With `ML.EVALUATE`\n",
        "\n",
        "BigQuery ML models can be directly evaluated with the [`ML.EVALUATE` function](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-evaluate).  The evaluation of the model on evaluation data specified during training can be done by just specifying the model.  To evaluate the model on new data, like a `TEST` split or even the `TRAIN` split, the records can be specified with a query statement as seen in the following evaluations.\n",
        "\n",
        "\u003e An evaluation split with 10% of records was created above.  This set would normally be used for tuning model hyperparameters, like `BATCH_SIZE`, `DROPOUT`, `HIDDEN_UNITS`, `LEARN_RATE`, and/or `OPTIMIZER` in the case of the DNN used here.  This could also be done directly in the BigQuery `CREATE MODEL` statement with [hyperparameter tuning](https://cloud.google.com/bigquery/docs/hp-tuning-overview)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "tags": [],
        "id": "2_szmP7kxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7be41f867fb442c798b22b41df058ae6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d56257fde8624b0c9bf747e7bf488e9b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eprecision\u003c/th\u003e\n",
              "      \u003cth\u003erecall\u003c/th\u003e\n",
              "      \u003cth\u003eaccuracy\u003c/th\u003e\n",
              "      \u003cth\u003ef1_score\u003c/th\u003e\n",
              "      \u003cth\u003elog_loss\u003c/th\u003e\n",
              "      \u003cth\u003eroc_auc\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003e1.0\u003c/td\u003e\n",
              "      \u003ctd\u003e1.0\u003c/td\u003e\n",
              "      \u003ctd\u003e1.0\u003c/td\u003e\n",
              "      \u003ctd\u003e1.0\u003c/td\u003e\n",
              "      \u003ctd\u003e0.278005\u003c/td\u003e\n",
              "      \u003ctd\u003e1.0\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "   precision  recall  accuracy  f1_score  log_loss  roc_auc\n",
              "0        1.0     1.0       1.0       1.0  0.278005      1.0"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.EVALUATE (\n",
        "    MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XPF4TtuhxLVM"
      },
      "source": [
        "### Evaluate The Model: Training Data Split (`TRAIN`)\n",
        "\n",
        "Evaluating the model on the training data is good for setting a baseline for the likely best possible performance of the model.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "tags": [],
        "id": "620FggwHxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b1042532d6ec4686acc1787b083e78ae",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "94ce8eb82d514374bdcfd05f3d6c3fee",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eprecision\u003c/th\u003e\n",
              "      \u003cth\u003erecall\u003c/th\u003e\n",
              "      \u003cth\u003eaccuracy\u003c/th\u003e\n",
              "      \u003cth\u003ef1_score\u003c/th\u003e\n",
              "      \u003cth\u003elog_loss\u003c/th\u003e\n",
              "      \u003cth\u003eroc_auc\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003e0.993769\u003c/td\u003e\n",
              "      \u003ctd\u003e0.989032\u003c/td\u003e\n",
              "      \u003ctd\u003e0.991597\u003c/td\u003e\n",
              "      \u003ctd\u003e0.991322\u003c/td\u003e\n",
              "      \u003ctd\u003e0.266515\u003c/td\u003e\n",
              "      \u003ctd\u003e1.0\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "   precision    recall  accuracy  f1_score  log_loss  roc_auc\n",
              "0   0.993769  0.989032  0.991597  0.991322  0.266515      1.0"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.EVALUATE (\n",
        "    MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "        WHERE splits = 'TRAIN'\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WpxAwgrSxLVM"
      },
      "source": [
        "### Evaluate The Model: `TEST` Data Split\n",
        "\n",
        "Evaluating the model on the `TEST` split is a good gauge of performance in general as it represents data the model was not exposed to during training: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "tags": [],
        "id": "xF9s8JNSxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "938c0b8940b942c4bd6d9afbec2d13f4",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "fe4c70a1c49a47fa93c640df97b5f40d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eprecision\u003c/th\u003e\n",
              "      \u003cth\u003erecall\u003c/th\u003e\n",
              "      \u003cth\u003eaccuracy\u003c/th\u003e\n",
              "      \u003cth\u003ef1_score\u003c/th\u003e\n",
              "      \u003cth\u003elog_loss\u003c/th\u003e\n",
              "      \u003cth\u003eroc_auc\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003e0.958333\u003c/td\u003e\n",
              "      \u003ctd\u003e0.979167\u003c/td\u003e\n",
              "      \u003ctd\u003e0.971831\u003c/td\u003e\n",
              "      \u003ctd\u003e0.967025\u003c/td\u003e\n",
              "      \u003ctd\u003e0.324351\u003c/td\u003e\n",
              "      \u003ctd\u003e0.99813\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "   precision    recall  accuracy  f1_score  log_loss  roc_auc\n",
              "0   0.958333  0.979167  0.971831  0.967025  0.324351  0.99813"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.EVALUATE (\n",
        "    MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "        WHERE splits = 'TEST'\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iFyNGkghxLVM"
      },
      "source": [
        "### Evaluate The Classification Performance with `ML.CONFUSION_MATRIX`\n",
        "\n",
        "This model is a classification model and reviewing the true versus predicted classification categories can be done with the [`ML.CONFUSION_MATRIX` function](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-confusion)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "tags": [],
        "id": "fxxIUmPtxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "87150f14f9a04f3092a728bc5b6a6099",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7052d893088043d898b543ee755e14ff",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eexpected_label\u003c/th\u003e\n",
              "      \u003cth\u003eAdelie_Penguin__Pygoscelis_adeliae_\u003c/th\u003e\n",
              "      \u003cth\u003eChinstrap_penguin__Pygoscelis_antarctica_\u003c/th\u003e\n",
              "      \u003cth\u003eGentoo_penguin__Pygoscelis_papua_\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003e15\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eChinstrap penguin (Pygoscelis antarctica)\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e7\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e13\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                              expected_label  \\\n",
              "0        Adelie Penguin (Pygoscelis adeliae)   \n",
              "1  Chinstrap penguin (Pygoscelis antarctica)   \n",
              "2          Gentoo penguin (Pygoscelis papua)   \n",
              "\n",
              "   Adelie_Penguin__Pygoscelis_adeliae_  \\\n",
              "0                                   15   \n",
              "1                                    0   \n",
              "2                                    0   \n",
              "\n",
              "   Chinstrap_penguin__Pygoscelis_antarctica_  \\\n",
              "0                                          0   \n",
              "1                                          7   \n",
              "2                                          0   \n",
              "\n",
              "   Gentoo_penguin__Pygoscelis_papua_  \n",
              "0                                  0  \n",
              "1                                  0  \n",
              "2                                 13  "
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.CONFUSION_MATRIX (\n",
        "    MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "tags": [],
        "id": "bcpiQk5jxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "83fb50cd023049a99c2ff9a61efb7790",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "da1f023981954558aaab8e1565294698",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eexpected_label\u003c/th\u003e\n",
              "      \u003cth\u003eAdelie_Penguin__Pygoscelis_adeliae_\u003c/th\u003e\n",
              "      \u003cth\u003eChinstrap_penguin__Pygoscelis_antarctica_\u003c/th\u003e\n",
              "      \u003cth\u003eGentoo_penguin__Pygoscelis_papua_\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003e105\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eChinstrap penguin (Pygoscelis antarctica)\u003c/td\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e46\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e85\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                              expected_label  \\\n",
              "0        Adelie Penguin (Pygoscelis adeliae)   \n",
              "1  Chinstrap penguin (Pygoscelis antarctica)   \n",
              "2          Gentoo penguin (Pygoscelis papua)   \n",
              "\n",
              "   Adelie_Penguin__Pygoscelis_adeliae_  \\\n",
              "0                                  105   \n",
              "1                                    1   \n",
              "2                                    1   \n",
              "\n",
              "   Chinstrap_penguin__Pygoscelis_antarctica_  \\\n",
              "0                                          0   \n",
              "1                                         46   \n",
              "2                                          0   \n",
              "\n",
              "   Gentoo_penguin__Pygoscelis_papua_  \n",
              "0                                  0  \n",
              "1                                  0  \n",
              "2                                 85  "
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.CONFUSION_MATRIX (\n",
        "    MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "        WHERE splits = 'TRAIN'\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "tags": [],
        "id": "iG6msG8AxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "356d091d363540de990b59cd7a3bd6c0",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d80397bb089e4350b566b10bf2c3689b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eexpected_label\u003c/th\u003e\n",
              "      \u003cth\u003eAdelie_Penguin__Pygoscelis_adeliae_\u003c/th\u003e\n",
              "      \u003cth\u003eChinstrap_penguin__Pygoscelis_antarctica_\u003c/th\u003e\n",
              "      \u003cth\u003eGentoo_penguin__Pygoscelis_papua_\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003e30\u003c/td\u003e\n",
              "      \u003ctd\u003e2\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eChinstrap penguin (Pygoscelis antarctica)\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e14\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e25\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                              expected_label  \\\n",
              "0        Adelie Penguin (Pygoscelis adeliae)   \n",
              "1  Chinstrap penguin (Pygoscelis antarctica)   \n",
              "2          Gentoo penguin (Pygoscelis papua)   \n",
              "\n",
              "   Adelie_Penguin__Pygoscelis_adeliae_  \\\n",
              "0                                   30   \n",
              "1                                    0   \n",
              "2                                    0   \n",
              "\n",
              "   Chinstrap_penguin__Pygoscelis_antarctica_  \\\n",
              "0                                          2   \n",
              "1                                         14   \n",
              "2                                          0   \n",
              "\n",
              "   Gentoo_penguin__Pygoscelis_papua_  \n",
              "0                                  0  \n",
              "1                                  0  \n",
              "2                                 25  "
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.CONFUSION_MATRIX (\n",
        "    MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "        WHERE splits = 'TEST'\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "VxqLgK_0xLVM"
      },
      "source": [
        "---\n",
        "## Model Serving\n",
        "\n",
        "Once a version of the model is trusted to serve predictions we can put it into use.  At first, this means creating and using predictions from the model.  This is quickly followed by important considerations:\n",
        "\n",
        "\u003eHas the system that is being measured, that creates the data, or that records the data changed in a way that could impact the models ability to correctly predict/classify?\n",
        "\n",
        "It seems like a good idea to continously run evaluations to make sure the model accuracy does not degrade - and it is.  There can be a lag between new data and knowing the actual outcome which makes using using evaluations to monitor models a potentially reactive, or late signal.  To overcome this we can also monitor the model.  More on this later on below."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BC2rgYkkxLVM"
      },
      "source": [
        "### Predictions With `ML.PREDICT`\n",
        "\n",
        "Use [`ML.PREDICT`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-predict) to serve predictions with the model directly in BigQuery."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "tags": [],
        "id": "-d2sgVkZxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9a04c20d2138402382bd0541239e9d25",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "599234341bbd4e2a85bb5012d006b779",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003epredicted_species\u003c/th\u003e\n",
              "      \u003cth\u003epredicted_species_probs\u003c/th\u003e\n",
              "      \u003cth\u003especies\u003c/th\u003e\n",
              "      \u003cth\u003eisland\u003c/th\u003e\n",
              "      \u003cth\u003eculmen_length_mm\u003c/th\u003e\n",
              "      \u003cth\u003eculmen_depth_mm\u003c/th\u003e\n",
              "      \u003cth\u003eflipper_length_mm\u003c/th\u003e\n",
              "      \u003cth\u003ebody_mass_g\u003c/th\u003e\n",
              "      \u003cth\u003esex\u003c/th\u003e\n",
              "      \u003cth\u003esplits\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003e[{'label': 'Gentoo penguin (Pygoscelis papua)'...\u003c/td\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNone\u003c/td\u003e\n",
              "      \u003ctd\u003eTRAIN\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e[{'label': 'Gentoo penguin (Pygoscelis papua)'...\u003c/td\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003e46.5\u003c/td\u003e\n",
              "      \u003ctd\u003e13.5\u003c/td\u003e\n",
              "      \u003ctd\u003e210.0\u003c/td\u003e\n",
              "      \u003ctd\u003e4550.0\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "      \u003ctd\u003eTRAIN\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e[{'label': 'Gentoo penguin (Pygoscelis papua)'...\u003c/td\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003e42.0\u003c/td\u003e\n",
              "      \u003ctd\u003e13.5\u003c/td\u003e\n",
              "      \u003ctd\u003e210.0\u003c/td\u003e\n",
              "      \u003ctd\u003e4150.0\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "      \u003ctd\u003eTRAIN\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e[{'label': 'Gentoo penguin (Pygoscelis papua)'...\u003c/td\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003e47.5\u003c/td\u003e\n",
              "      \u003ctd\u003e14.0\u003c/td\u003e\n",
              "      \u003ctd\u003e212.0\u003c/td\u003e\n",
              "      \u003ctd\u003e4875.0\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "      \u003ctd\u003eTRAIN\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e[{'label': 'Gentoo penguin (Pygoscelis papua)'...\u003c/td\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003e43.3\u003c/td\u003e\n",
              "      \u003ctd\u003e14.0\u003c/td\u003e\n",
              "      \u003ctd\u003e208.0\u003c/td\u003e\n",
              "      \u003ctd\u003e4575.0\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "      \u003ctd\u003eTRAIN\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                     predicted_species  \\\n",
              "0  Adelie Penguin (Pygoscelis adeliae)   \n",
              "1    Gentoo penguin (Pygoscelis papua)   \n",
              "2    Gentoo penguin (Pygoscelis papua)   \n",
              "3    Gentoo penguin (Pygoscelis papua)   \n",
              "4    Gentoo penguin (Pygoscelis papua)   \n",
              "\n",
              "                             predicted_species_probs  \\\n",
              "0  [{'label': 'Gentoo penguin (Pygoscelis papua)'...   \n",
              "1  [{'label': 'Gentoo penguin (Pygoscelis papua)'...   \n",
              "2  [{'label': 'Gentoo penguin (Pygoscelis papua)'...   \n",
              "3  [{'label': 'Gentoo penguin (Pygoscelis papua)'...   \n",
              "4  [{'label': 'Gentoo penguin (Pygoscelis papua)'...   \n",
              "\n",
              "                             species  island  culmen_length_mm  \\\n",
              "0  Gentoo penguin (Pygoscelis papua)  Biscoe               NaN   \n",
              "1  Gentoo penguin (Pygoscelis papua)  Biscoe              46.5   \n",
              "2  Gentoo penguin (Pygoscelis papua)  Biscoe              42.0   \n",
              "3  Gentoo penguin (Pygoscelis papua)  Biscoe              47.5   \n",
              "4  Gentoo penguin (Pygoscelis papua)  Biscoe              43.3   \n",
              "\n",
              "   culmen_depth_mm  flipper_length_mm  body_mass_g     sex splits  \n",
              "0              NaN                NaN          NaN    None  TRAIN  \n",
              "1             13.5              210.0       4550.0  FEMALE  TRAIN  \n",
              "2             13.5              210.0       4150.0  FEMALE  TRAIN  \n",
              "3             14.0              212.0       4875.0  FEMALE  TRAIN  \n",
              "4             14.0              208.0       4575.0  FEMALE  TRAIN  "
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.PREDICT(\n",
        "    MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "        LIMIT 5\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "FAezHP47xLVM"
      },
      "source": [
        "---\n",
        "## Serving Data\n",
        "\n",
        "The data source for training and evaluation represents what is known about a system at a point-in-time.  In real world systems measurements can change for many reasons, for example:\n",
        "- training/serving skew due to measurement and data systems:\n",
        "    - the tool used for measurement may change resulting is slight shifts and variations\n",
        "    - the system reporting results might be updated and introduce errors like rearranging the order or parameters and format of the results\n",
        "    - the data processing systems might have changes that alter the calculations of measurements\n",
        "- the system, environment, or subjects being measured may:\n",
        "    - change behaviors\n",
        "    - short-term even forces change\n",
        "    - have environmental changes that impact them\n",
        "    - have seasonal changes that were not represented in the training data\n",
        "    - change in general due to upstream causes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ULeZZi6LxLVM"
      },
      "source": [
        "### Introduce Changes Over Time\n",
        "\n",
        "To illustrate the impact of change over time, the following code creates a new set of data from the test split.  It creates a series of changes to measurements for each of the past 30 days.  In each case, the values of the parameters `flipper_length_mm` and `body_mass_g` are allowed to slightly increase which can accumulate over the timespan. This might mimic a potential growth season for the penguins that might not have been represented in the training data.\n",
        "\n",
        "This table of serving data could be sourced form where your features are coming from.  In the case of an online model, like [Vertex AI Prediction Endpoints](https://cloud.google.com/vertex-ai/docs/predictions/overview), then [prediction request could be logged to BigQuery](https://cloud.google.com/vertex-ai/docs/predictions/online-prediction-logging#enabling-and-disabling) for use in the model monitoring tasks below.  This online serving pattern is covered later in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "tags": [],
        "id": "UkiHOHGtxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2dff9057d2fa44fc896acf5b031b1a38",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: []\n",
              "Index: []"
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "DECLARE counter INT64 DEFAULT 30;\n",
        "\n",
        "# actual data as starting point - add entity_id, and feature_timestamp:\n",
        "CREATE OR REPLACE TABLE `statmike-mlops-349915.bqml_model_monitoring.serving` AS\n",
        "    SELECT * EXCEPT(flipper_length_mm, body_mass_g),\n",
        "    TIMESTAMP_SUB(TIMESTAMP(CURRENT_DATE()), INTERVAL counter DAY) AS instance_timestamp,\n",
        "    flipper_length_mm as flipper_length_mm,\n",
        "    body_mass_g as body_mass_g\n",
        "    FROM `statmike-mlops-349915.bqml_model_monitoring.training`\n",
        "    WHERE RAND() \u003c= 0.20;\n",
        "\n",
        "LOOP\n",
        "    SET counter = counter - 1;\n",
        "    IF counter \u003c 0 THEN LEAVE;\n",
        "    END IF;\n",
        "    INSERT `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "        SELECT * EXCEPT(instance_timestamp, flipper_length_mm, body_mass_g),\n",
        "            TIMESTAMP_ADD(instance_timestamp, INTERVAL 1 DAY) AS instance_timestamp,\n",
        "            CASE\n",
        "                WHEN RAND() \u003e= 0.75 THEN flipper_length_mm\n",
        "                ELSE flipper_length_mm + 6*(RAND()-0.1)\n",
        "            END AS flipper_length_mm,\n",
        "            CASE\n",
        "                WHEN RAND() \u003e= 0.75 THEN body_mass_g\n",
        "                ELSE body_mass_g + 120*(RAND()-0.1)\n",
        "            END AS body_mass_g,\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "        WHERE instance_timestamp \u003e= (SELECT MAX(instance_timestamp) FROM `statmike-mlops-349915.bqml_model_monitoring.serving`);\n",
        "END LOOP;"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "tags": [],
        "id": "eEdammxNxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c0868110a4b6408e98179f2cc3b659bb",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "22b71a7f94c04e529dbbb6ca701026de",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "%%bigquery serve\n",
        "SELECT\n",
        "    DATE(instance_timestamp) AS DATE,\n",
        "    AVG(flipper_length_mm) AS flipper_length_mm,\n",
        "    AVG(body_mass_g) AS body_mass_g\n",
        "FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "GROUP BY DATE\n",
        "ORDER BY DATE"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "tags": [],
        "id": "-232Ad9AxLVM"
      },
      "outputs": [],
      "source": [
        "serve.set_index('DATE', inplace = True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "tags": [],
        "id": "Px-AmCowxLVM"
      },
      "outputs": [],
      "source": [
        "#serve"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "tags": [],
        "id": "D2rgx4g7xLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "\u003cAxes: xlabel='DATE'\u003e"
            ]
          },
          "execution_count": 27,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAq8AAAFzCAYAAAAQdeBUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABVvElEQVR4nO3deVzVVf7H8ddlB9lEZVF2XFGzXEq0cMdSK5saW11aR8PKnJ+WjjPmNKbTNG1WNk1ly2jraIulZSq4UZq5LyiLKyCaCiiy3Xt+f5B3IjdA9LK8n48Hj0d8v+ee7+ceCN5+Od9zLMYYg4iIiIhIHeDk6AJERERERCpL4VVERERE6gyFVxERERGpMxReRURERKTOUHgVERERkTpD4VVERERE6gyFVxERERGpMxReRURERKTOcHF0AdVhs9nIysrCx8cHi8Xi6HJERERE5DeMMRQUFNC8eXOcnGrufmmdDK9ZWVmEhYU5ugwRERERuYD9+/cTGhpaY/3VyfDq4+MDlA+Gr6+vg6sRERERkd/Kz88nLCzMnttqSp0Mr6enCvj6+iq8ioiIiNRiNT3FUw9siYiIiEidofAqIiIiInWGwquIiIiI1Bl1cs5rZRhjKCsrw2q1OroUkWpxdnbGxcVFy8GJiIj8Sr0MryUlJWRnZ1NYWOjoUkQuipeXFyEhIbi5uTm6FBERkVqh3oVXm81GZmYmzs7ONG/eHDc3N925kjrHGENJSQmHDx8mMzOTVq1a1egCzyIiInVVvQuvJSUl2Gw2wsLC8PLycnQ5ItXm6emJq6sre/fupaSkBA8PD0eXJCIi4nD19laO7lJJfaDvYxERkYr0m1FERERE6gyFVxEREZEGYG3mUb7anO3oMi6awmstYozhoYceIiAgAIvFgr+/P+PGjbOfj4yM5MUXX3RYfVVhsVj47LPPHF0GAE899RRXXnmlo8sQERFxiG1ZeYyas5Zh/0phymdbKCgqdXRJF6XePbBVly1evJh33nmHpKQkoqOjue222yqcX7duHY0aNXJQdXWDxWJhwYIFDB061NGliIiIOFTmkZP889tUFv5yt9XFycKgjiGUWY2DK7s4Cq+1SHp6OiEhIfTo0QMAF5eKX55mzZo5oqwzlJSUaN1RERGRWio77xQvL93Nxz8ewGozWCxwU6fmPN6/NZFN6/5NsAYxbcAYQ2FJ2WX/MKby/7IZNWoUjzzyCPv27cNisRAZGXlGm99OG7BYLMyePZsbbrgBT09PoqOj+fTTT+3n9+zZg8Vi4cMPP6RHjx54eHjQoUMHkpOTK/S7detWbrjhBry9vQkKCmL48OEcOXLEfr53796MHTuWcePG0bRpUwYOHFj5wf/F/v37GTZsGP7+/gQEBHDzzTezZ8+eCu9/6NChPPfcc4SEhNCkSRMSExMpLf3fnzays7MZPHgwnp6eREVFMW/evApjcnrMbrnllrOO4fvvv09kZCR+fn7ccccdFBQUVKr23r1788gjjzBu3DgaN25MUFAQ//73vzl58iT33nsvPj4+tGzZkkWLFtlfk5SUhMVi4ZtvvuGqq67C09OTvn37kpuby6JFi2jXrh2+vr7cdddd2kxDRERqxLGTJUz/aju9/pHEB2v3Y7UZ+rUN5OtHr+OlO66qF8EVGsid11OlVmL/8s1lv+72vw7Ey61yQ/zSSy8RExPDG2+8wbp163B2dub3v//9BV/35z//mZkzZ/LSSy/x/vvvc8cdd7BlyxbatWtnbzNhwgRefPFFYmNjef7557nxxhvJzMykSZMmHD9+nL59+/LAAw/wwgsvcOrUKZ544gmGDRvGsmXL7H28++67jBkzhtWrV1d5HEpLSxk4cCBxcXGsXLkSFxcX/va3v3H99dezefNm+13c5cuXExISwvLly0lLS+P222/nyiuv5MEHHwRgxIgRHDlyhKSkJFxdXRk/fjy5ubn266xbt47AwEDmzJnD9ddfj7Ozs/1ceno6n332GQsXLuTYsWMMGzaMmTNnMn369Eq9h3fffZeJEyeydu1aPvroI8aMGcOCBQu45ZZbmDx5Mi+88ALDhw9n3759FdYXfuqpp3jllVfw8vJi2LBhDBs2DHd3d+bNm8eJEye45ZZbmDVrFk888USVx1VERATgRHEZb63M5N8rMzhRXAbA1ZEBTLy+DV0jAxxcXc1rEOG1LvDz88PHxwdnZ2eCg4Mr/brf//73PPDAAwA8/fTTLFmyhFmzZvHaa6/Z24wdO5Zbb70VgNmzZ7N48WLeeustJk6cyCuvvMJVV13FM888Y2//9ttvExYWxq5du2jdujUArVq14tlnn63We/voo4+w2Wy8+eab9t3O5syZg7+/P0lJSSQkJADQuHFjXnnlFZydnWnbti2DBw9m6dKlPPjgg+zcuZPvvvuOdevW0bVrVwDefPNNWrVqZb/O6WkV/v7+Z4yhzWbjnXfewcfHB4Dhw4ezdOnSSofXTp06MWXKFAAmTZrEzJkzadq0qT1Y/+Uvf2H27Nls3ryZ7t2721/3t7/9jZ49ewJw//33M2nSJNLT04mOjgbgtttuY/ny5QqvIiJSZUWlVub+sI/Xlqfx88kSAGJDfJlwfRt6t25Wb3cYbRDh1dPVme1/rfqfumviupdaXFzcGZ9v3LjxnG1cXFzo2rUrO3bsAGDTpk0sX74cb2/vM/pOT0+3h9cuXbpUu8ZNmzaRlpZmD46nFRUVkZ6ebv+8ffv2Fe6WhoSEsGXLFgBSU1NxcXGhc+fO9vMtW7akcePGlaohMjKywvVDQkIq3LW9kCuuuML+387OzjRp0oSOHTvajwUFBQGc0eevXxcUFISXl5c9uJ4+tnbt2krXISIiUma1Mf+ng7z43S6y8ooAiGraiPEDWjO4YwhOTvUztJ7WIMKrxWKp9J/vG5oTJ05w44038ve///2McyEhIfb/vphVDk6cOEGXLl2YO3fuGed+/RCaq6trhXMWiwWbzVbt6/7axfZ9ttf/+tjpf93+ts/ftrmU71FEROo3YwyLt+bw3LeppB8+CUCwrweP9W/FbV1CcXVuEI8yNYzwWp99//33jBgxosLnV1111Rlt4uPjASgrK2P9+vWMHTsWgM6dO/Pf//6XyMjIM1Y3qCmdO3fmo48+IjAwEF9f32r10aZNG8rKytiwYYP9LnBaWhrHjh2r0M7V1RWr1XrRNYuIiNQWxhhWpR3hH9+ksvlAHgD+Xq483DuGEXGReFyGv/TWJg0jotdjn3zyCW+//Ta7du1i6tSprF271h5MT3v11VdZsGABO3fuJDExkWPHjnHfffcBkJiYyNGjR7nzzjtZt24d6enpfPPNN9x77701FgLvvvtumjZtys0338zKlSvJzMwkKSmJRx99lAMHDlSqj7Zt29K/f38eeugh1q5dy4YNG3jooYfw9PSsMKcnMjKSpUuXkpOTc0awFRERqWs27DvGXf/+geFvrWXzgTy83Jx5tG9LVkzsw0PxMQ0uuILCa503bdo0PvzwQ6644gree+89PvjgA2JjYyu0mTlzJjNnzqRTp06sWrWKL774gqZNmwLQvHlzVq9ejdVqJSEhgY4dOzJu3Dj8/f1xcqqZbw8vLy9WrFhBeHg4v/vd72jXrh33338/RUVFVboT+9577xEUFER8fDy33HILDz74ID4+Pnh4eNjb/POf/2TJkiWEhYWdcQdaRESkrkjNKeDB937kltfWkJLxM27OTtzbM5IVE/swPqENvh6uF+6knrKYqixGWkvk5+fj5+dHXl7eGeGnqKiIzMxMoqKiKoSa+uhCu0nt2bOHqKgoNmzYUC+3Rz1w4ABhYWF899139OvXz9HlXBIN6ftZRERg38+FvPDdLj7beBBjwMkCt3YO5bH+rQht7HXhDmqR8+W1i6E5r1JnLFu2jBMnTtCxY0eys7OZOHEikZGR9vm8IiIidVVufhEvL9vNh2v3U2Yrv684qGMw4we0pmWgzwVe3bAovEqVzZ07lz/84Q9nPRcREcG2bdsuyXVLS0uZPHkyGRkZ+Pj40KNHD+bOnXvGE/xVsW/fvjOmWfza9u3bCQ8Pr3b/IiIi53O8sITXkzN4Z00mRaXlq8/Et27GhIQ2dAz1c3B1tZPCax12oRkfkZGRVdqitrJuuukmrrnmmrOeu5ggeSEDBw6s1ta059O8efMz1sX97XkREZGadrK4jDmrM/nXigwKisp3xeoc7s/E69vSPbqJg6ur3RRepcp8fHzO2HCgrnJxcaFly5aOLkNERBqI4jIrH/ywj1eWp3HkRPmuWG2Dffi/hDb0axdYb3fFqkn1NrzWwefQRM6g72MRkfrBajPM/+kAL363m4PHTwEQ0cSL8QNac+MVzev9rlg1qd6F19N/ti4sLMTT09PB1YhcnMLCQuDSTscQEZFL5/SuWP9csou03BMABPm682i/VgzrGtZgdsWqSfUuvDo7O+Pv72/fY97Ly0u34KXOMcZQWFhIbm4u/v7+ODs3vEWoRUTqMu2KdenUu/AKEBwcDGAPsCJ1lb+/v/37WURE6ob1e4/y3De7SMn4GQAvN2ceuDaKB+KjG/TmAjWlXoZXi8VCSEgIgYGBlJaWOrockWpxdXXVHVcRkTrkh4yfeXnZblanlYdWN2cn7ukewcN9Ymjq7e7g6uqPehleT3N2dtYvfxEREblkjDGkZPzMy0t3833GUQBcnCz8vmsoY/u2ooW/nr+pafU6vIqIiIhcCsYYVqeVh9a1e8pDq6uzhWFdwxjTO6bObeValyi8ioiIiFSSMYYVu4/w8tLdrN97DCifHnDH1WGM7hVDc91pveQUXkVEREQuwBhDUuphXlq6m437jwPg7uLEnVeHM7pXDMF+Ho4tsAFReBURERE5B2MMS3fk8vKy3fYlrzxcnbj7mgj+EB9NoK9C6+Wm8CoiIiLyGzab4dvth5i1bDfbsvIB8HR1ZkRcBA9cF00zH60e4ChV2tZhxowZdOvWDR8fHwIDAxk6dCipqakV2vTu3RuLxVLhY/To0RXa7Nu3j8GDB+Pl5UVgYCATJkygrKzs4t+NiIiIyEWw2Qxfb8lm0MsrGf2f9WzLyqeRmzNjesew6ok+TBrUTsHVwap05zU5OZnExES6detGWVkZkydPJiEhge3bt9OoUSN7uwcffJC//vWv9s+9vP73xJ3VamXw4MEEBwezZs0asrOzGTFiBK6urjzzzDM18JZEREREqsZmM3y1JZtZy3az61D5Nq7e7i6M6hHJ/ddG0biRm4MrlNMsxhhT3RcfPnyYwMBAkpOTiY+PB8rvvF555ZW8+OKLZ33NokWLGDJkCFlZWQQFBQHw+uuv88QTT3D48GHc3C78zZGfn4+fnx95eXn4+vpWt3wRERERftp3jGlfbGPTL3NafTxcuLdnFPf3jMLPSztiVdelymtVmjbwW3l55V/kgICACsfnzp1L06ZN6dChA5MmTaKwsNB+LiUlhY4dO9qDK8DAgQPJz89n27ZtZ71OcXEx+fn5FT5ERERELsah/CIe/2gjv3ttDZsO5OHt7sLj/Vuz6om+jB/QWsG1lqr2A1s2m41x48bRs2dPOnToYD9+1113ERERQfPmzdm8eTNPPPEEqampzJ8/H4CcnJwKwRWwf56Tk3PWa82YMYNp06ZVt1QRERERu6JSK2+tyuTV5WkUllixWOD3XUL5v4FtCPTR6gG1XbXDa2JiIlu3bmXVqlUVjj/00EP2/+7YsSMhISH069eP9PR0YmJiqnWtSZMmMX78ePvn+fn5hIWFVa9wERERaZCMKV9BYPpXO9h3tPyvwp3D/XnqpvZcEerv2OKk0qoVXseOHcvChQtZsWIFoaGh5217zTXXAJCWlkZMTAzBwcGsXbu2QptDhw4BEBwcfNY+3N3dcXfXk30iIiJSPbsPFfDXhdtZufsIAEG+7jx5Q1uGXtkCi8Xi4OqkKqoUXo0xPPLIIyxYsICkpCSioqIu+JqNGzcCEBISAkBcXBzTp08nNzeXwMBAAJYsWYKvry+xsbFVLF9ERETk3PIKS3nhu128//1erDaDm7MTD8ZH8XDvljRy13L3dVGVvmqJiYnMmzePzz//HB8fH/scVT8/Pzw9PUlPT2fevHkMGjSIJk2asHnzZh5//HHi4+O54oorAEhISCA2Npbhw4fz7LPPkpOTw5QpU0hMTNTdVREREakRVpvhg7X7+Oe3qRwrLAUgITaIKYNjCW/idYFXS21WpaWyznVbfc6cOYwaNYr9+/dzzz33sHXrVk6ePElYWBi33HILU6ZMqbBEwt69exkzZgxJSUk0atSIkSNHMnPmTFxcKpeltVSWiIiInMv3GT8z7cvt7MguX52odZA3fxnSnmtbNXVwZQ3LpcprF7XOq6MovIqIiMhvHThWyIyvd/LVlmwAfD1cGD+gNfd0j8DF+aJWB5VquFR5TZM9REREpE47VWLl9eR0Xk9Op7jMhpMF7rw6nD8mtCFAO2PVOwqvIiIiUied3tJ1xtc7yMorAuCaqACm3tie2Ob6y2x9pfAqIiIidYrNZvhmWw4vLd3NzpwCAFr4ezJ5UDsGdQzW0lf1nMKriIiI1Ak2m2Hxthxe/lVo9XZ34cHronkoPhpPN2cHVyiXg8KriIiI1Go2m+HrrdnMWppG6qHy0Orj7sK9PSO579oo/L00r7UhUXgVERGRWslqM3y9JZuXl+5md+4J4JfQem0U9/eMws/L1cEViiMovIqIiEitYrUZFm7OYtayNNJOh1YPF+7rGcV9Cq0NnsKriIiI1AqnQ+vLS3eTfvgkUL5W6/3XRjOqZyR+ngqtovAqIiIiDma1Gb7clMXLy3aT8avQ+sB15aHV10OhVf5H4VVEREQcosxq44tNWbyyLI2MI+Wh1c/TlQeujWKkQqucg8KriIiIXFZlVhufb8zileVpZP4SWv29XHnwumhGxEXgo9Aq56HwKiIiIpfF6R2xnl+yyx5aG3u58sB10YzsEYm3u2KJXJi+S0REROSSMsawcvcRnv1mJ1sP5gPlofWh+BiGx0UotEqV6LtFRERELpkN+47x7OJUUjJ+BqCRmzMPxcdw/3VRCq1SLfquERERkRqXllvAP75J5ZtthwBwc3binu4RJPaJoYm3u4Ork7pM4VVERERqTNbxU7z43S4+XX8AmwEnC/yucyjj+rcitLGXo8uTekDhVURERC7asZMlvLo8jfe+30tJmQ2AAbFBTBjYhtZBPg6uTuoThVcRERGptpPFZby9KpM3VmRQUFwGwNVRATxxfVu6RDR2cHVSHym8ioiISJWVlNn4YO0+Zi3bzZETJQC0C/Fl4vVt6N26GRaLxcEVSn2l8CoiIiKVZrMZvtiUxT+XpLL/6CkAwgO8+GNCa268ojlOTgqtcmkpvIqIiMgFGWNYnprLs4tT2ZlTAEBTb3ce69eS27uF4+bi5OAKpaFQeBUREZHz+j7jZ/75bSrr9hwDwMfdhdG9Y7i3ZyRebooScnnpO05ERETO6qd9x3j+212sSjsCgJuLE6N6RDKmVwyNG7k5uDppqBReRUREpIKtB/N4Yckulu7MBcDV2cLt3cIY26cVwX4eDq5OGjqFVxEREQFg96ECXvhuF19vyQHKNxi4tXMoj/ZrRViANhiQ2kHhVUREpIHbc+QkLy3dzWcbD2IMWCxwU6fmPNavFdHNvB1dnkgFCq8iIiIN1MHjp5i1dDefrD+A1WYAGNg+iPED2tAmWLtiSe2k8CoiItLA5OYX8eryND5Yu58Sa/lWrn3aNGP8gDZ0DPVzcHUi56fwKiIi0kD8fKKYf63I4N01eyguKw+tPWKa8MeE1nSJCHBwdSKVo/AqIiJSz+WdKuXNlRm8vSqTkyVWALpENOaPA1rTo2VTB1cnUjUKryIiIvVUflEp763ZwxsrMsgvKgOgQwtf/pjQht6tm2GxaCtXqXsUXkVEROqZLQfy+M/3e/liUxanSsvvtLYO8mb8gDYMbB+k0Cp1msKriIhIPXCqxMqXm7OY+/1eNh3Isx9vHeRNYp+WDLmiOc5OCq1S9ym8ioiI1GFpuSeY+8Ne/rv+gH1qgJuzEzd0DObuayLoFtlYd1qlXlF4FRERqWNKymx8uz2H/3y/l+8zjtqPhwV4ctfVEfy+ayhNvd0dWKHIpaPwKiIiUkccPH6KD37Yx4fr9nPkRDFQvoVr37ZB3NM9nPhWzXDS1ACp5xReRUREajGrzbBi12H+8/1elqfm8stGWDTzceeObmHccXU4Lfw9HVukyGWk8CoiIlILHTlRzMc/7mfeD/s4cOyU/XiPmCbc0z2CAbFBuDo7ObBCEcdQeBUREaklrDbDmvQjfPzjARZvzabUWn6b1c/Tldu6hHLXNeHENPN2cJUijlWlf7LNmDGDbt264ePjQ2BgIEOHDiU1NfWsbY0x3HDDDVgsFj777LMK5/bt28fgwYPx8vIiMDCQCRMmUFZWVu03ISIiUpftPlTAzEU76TlzGcPfWsuXm7IotRo6hfnzj9uu4IfJ/fjzkFgFVxGqeOc1OTmZxMREunXrRllZGZMnTyYhIYHt27fTqFGjCm1ffPHFsy7NYbVaGTx4MMHBwaxZs4bs7GxGjBiBq6srzzzzzMW9GxERkTri6MkSvth4kPkbDrL5V+uy+nm6cmOnEO7oFk6HFn4OrFCkdrIYY0x1X3z48GECAwNJTk4mPj7efnzjxo0MGTKEH3/8kZCQEBYsWMDQoUMBWLRoEUOGDCErK4ugoCAAXn/9dZ544gkOHz6Mm5vbBa+bn5+Pn58feXl5+Pr6Vrd8ERGRy6qkzMaynbn896cDLN+ZS9kvT1+5OFno3SaQ27q0oE/bQNxdnB1cqcjFu1R57aLmvObllf9LMSAgwH6ssLCQu+66i1dffZXg4OAzXpOSkkLHjh3twRVg4MCBjBkzhm3btnHVVVddTEkiIiK1ijGGzQfy+O9PB/hiUxbHC0vt5zq08OXWzqHc2Km51mUVqaRqh1ebzca4cePo2bMnHTp0sB9//PHH6dGjBzfffPNZX5eTk1MhuAL2z3Nycs76muLiYoqLi+2f5+fnV7dsERGRyyI77xQLNhxk/k8HScs9YT8e6OPOLVe14HedQ2kT7OPACkXqpmqH18TERLZu3cqqVavsx7744guWLVvGhg0baqS402bMmMG0adNqtE8REZGaVlhSxjfbcvjv+oOsTj/C6Yl57i5ODGwfzK1dQukZ0wQXLXElUm3VCq9jx45l4cKFrFixgtDQUPvxZcuWkZ6ejr+/f4X2t956K9dddx1JSUkEBwezdu3aCucPHToEcNZpBgCTJk1i/Pjx9s/z8/MJCwurTukiIiI1LjvvFC99t5svN2VxssRqP351VAC3dm7BoI4h+Hi4OrBCkfqjSuHVGMMjjzzCggULSEpKIioqqsL5J598kgceeKDCsY4dO/LCCy9w4403AhAXF8f06dPJzc0lMDAQgCVLluDr60tsbOxZr+vu7o67u+YCiYhI7VJUauWNFRnMTkrnVGl5aA0P8OLWzqHcclULwpt4ObhCkfqnSuE1MTGRefPm8fnnn+Pj42Ofo+rn54enpyfBwcFnvXsaHh5uD7oJCQnExsYyfPhwnn32WXJycpgyZQqJiYkKqCIiUicYY/h6Sw7PfL2Dg8fLd7/qFtmY/0tow9VRAWddKlJEakaVwuvs2bMB6N27d4Xjc+bMYdSoUZXqw9nZmYULFzJmzBji4uJo1KgRI0eO5K9//WtVShEREXGIbVl5/PXL7fyQeRSA5n4eTBrUjiFXhCi0ilwGVZ42UFVne01ERARff/11lfsSERFxlJ9PFPPct7v4aN0+bAY8XJ0Y3SuGP8TH4OmmdVlFLpeLWudVRESkviu12ngvZS8vfreLgqLyrcyHXBHCpEHtaOHv6eDqRBoehVcREZFzSErN5emF20k/fBKA9s19mXpje66OCrjAK0XkUlF4FRER+Y2Mwyf421c7WLYzF4Cm3m78X0Ibft81DGcnzWsVcSSFVxERkV/kF5Uya+lu5qzeQ5nN4Ops4d6eUYzt2xJfrdMqUisovIqISINntRk++XE///gmlZ9PlgDQt20gUwa3I7qZt4OrE5FfU3gVEZEGbW3mUaZ9uY1tWfkARDdrxJ+HxNKnTaCDKxORs1F4FRGRBscYw/cZR5mzOpNvt5dvUe7j4cK4/q0ZEReBq7OTgysUkXNReBURkQbjRHEZC346wPvf72XXoRMAWCxw59Xh/HFAa5p4a6dHkdpO4VVEROq9tNwC3kvZy/yfDnKiuHytVi83Z265qgUje0TSOsjHwRWKSGUpvIqISL1UZrWxZPsh3kvZS0rGz/bj0c0aMaJ7BL/rEqoVBETqIIVXERGpV3ILivhw7X7m/bCPnPwiAJwsMCA2iBFxkfSIaYLForVaReoqhVcREanzjDH8uPcY76XsZfHWbEqtBoAmjdy44+ow7romQlu5itQTCq8iIlJnFZaU8fnGLN5L2cuO7Hz78c7h/oyIi+SGjsG4uzg7sEIRqWkKryIiUudkHjnJ+yl7+WT9fgqKyh/A8nB14uZOLRgeF0GHFn4OrlBELhWFVxERqTP2Hy1kxqIdfL0lx34sookX91wTwe+7huLv5ebA6kTkclB4FRGRWq+o1MrspHReT06nuMyGxQJ92gQyPC6CXq2a4eSkB7BEGgqFVxERqbWMMSzemsPfvtrBweOnAOgeHcBTN7WnbbCvg6sTEUdQeBURkVpp16ECpn25jdVp5Wu0Nvfz4E+DYxnUMVhLXYk0YAqvIiJSq+SdKuWl73bzbsoerDaDm4sTo+OjGd07Bi83/doSaej0U0BERGoFm83w6foDPPvNTo6cKAEgITaIKYNjCW/i5eDqRKS2UHgVERGH27j/OFM/38qmA3lA+RauT93YnvjWzRxcmYjUNgqvIiLiMIcLinl28U4+WX8AAG93Fx7r14qRPSJxc3FycHUiUhspvIqIyGVXarXx7po9vPTdbgqKyzcZuLVzKE9c34ZAXw8HVycitZnCq4iIXFardh/hqS+3kZZ7AoCOLfx46qb2dIlo7ODKRKQuUHgVEZHLYv/RQqZ/tYPF28p3xwpo5MbEgW0Y1jVMmwyISKUpvIqIyCWVW1DEmyszeXfNHorLbDg7WRjePYLH+7fGz8vV0eWJSB2j8CoiIpdETl4R/1qRzrwf9lFcZgMgLroJU2+K1e5YIlJtCq8iIlKjDh4/xetJ6Xy0bj8l1vLQelW4P4/2bUXvNs20O5aIXBSFVxERqRH7jxbyWlIan64/QKnVANAtsjGP9WtNz5ZNFFpFpEYovIqIyEXJPHKSV5ensWDDQay28tAaF92ER/u1ont0gEKriNQohVcREamWtNwCXlmWxhebsvgls3Jdq6Y82q8V3SIDHFuciNRbCq8iIlIlqTkFzFq2m6+2ZGN+Ca192wbySN+WXBWutVpF5NJSeBURkUrZejCPV5al2ddpBUiIDeKRvq3oGOrnwMpEpCFReBURkfPatP84s5bt5rsduQBYLDCoQwhj+7akXYiWvBKRy0vhVUREzmpt5lFeXZ5G8q7DADhZ4MZOzRnbpyWtgnwcXJ2INFQKryIiYmezGZan5jI7KZ0f9x4DwNnJws1XNiexT0timnk7uEIRaegUXkVEhDKrjS83Z/F6UgaphwoAcHN24tYuoYzuFU1Ek0YOrlBEpJzCq4hIA1ZUauXjH/fzxooMDhw7BYC3uwt3dw/n/p5RBPp6OLhCEZGKFF5FRBqgvFOl/Of7vby9KpOfT5YA0KSRG/ddG8U93SPw83R1cIUiImen8Coi0oDk5hfx1qpM5v6wjxPFZQC08PfkD72iGdY1DA9XZwdXKCJyfk5VaTxjxgy6deuGj48PgYGBDB06lNTU1Apt/vCHPxATE4OnpyfNmjXj5ptvZufOnRXa7Nu3j8GDB+Pl5UVgYCATJkygrKzs4t+NiIic1Z4jJ5k0fwvX/n05/1qRwYniMtoE+fDi7VeSNKE3I+IiFVxFpE6o0p3X5ORkEhMT6datG2VlZUyePJmEhAS2b99Oo0blk/m7dOnC3XffTXh4OEePHuWpp54iISGBzMxMnJ2dsVqtDB48mODgYNasWUN2djYjRozA1dWVZ5555pK8SRGRhmrrwTxeT07n6y3Z9i1cu0Q05uHeMfRpE4iTk8WxBYqIVJHFmNOb+1Xd4cOHCQwMJDk5mfj4+LO22bx5M506dSItLY2YmBgWLVrEkCFDyMrKIigoCIDXX3+dJ554gsOHD+Pm5nbB6+bn5+Pn50deXh6+vlogW0Tk14wx/JB5lNeS0lnxyxqtAH3aNGNM75ZcHRXgwOpEpKG4VHntoua85uXlARAQcPYfhCdPnmTOnDlERUURFhYGQEpKCh07drQHV4CBAwcyZswYtm3bxlVXXXVGP8XFxRQXF9s/z8/Pv5iyRUTqJZvNsHRnLq8lpbFh33GgfGOBIVc0Z3SvGGKb6x/7IlL3VTu82mw2xo0bR8+ePenQoUOFc6+99hoTJ07k5MmTtGnThiVLltjvqObk5FQIroD985ycHM5mxowZTJs2rbqliojUa6VWG19szOL15HR2554AwM3FiWFdQ3nouhjCm3g5uEIRkZpT7fCamJjI1q1bWbVq1Rnn7r77bgYMGEB2djbPPfccw4YNY/Xq1Xh4VG+9wEmTJjF+/Hj75/n5+fY7uSIiDdWpEisfrtvHmyszOXi8fI1WH3cX7omL4N6ekQT6aI1WEal/qhVex44dy8KFC1mxYgWhoaFnnPfz88PPz49WrVrRvXt3GjduzIIFC7jzzjsJDg5m7dq1FdofOnQIgODg4LNez93dHXd39+qUKiJS7xwvLOG9lL28s2YPR39Zo7Wptzv3XxvF3d3D8fXQGq0iUn9VKbwaY3jkkUdYsGABSUlJREVFVeo1xhj7nNW4uDimT59Obm4ugYGBACxZsgRfX19iY2Or8RZERBqGnLwi3lyZwby1+ygssQIQHuDFQ/HR3NYlVEtdiUiDUKXwmpiYyLx58/j888/x8fGxz1H18/PD09OTjIwMPvroIxISEmjWrBkHDhxg5syZeHp6MmjQIAASEhKIjY1l+PDhPPvss+Tk5DBlyhQSExN1d1VE5CzSD5/gjeQM5m84QKm1fIGYdiG+jOkdw6AOwbg4V2nJbhGROq1KS2VZLGdfD3DOnDmMGjWKrKwsHnjgAdavX8+xY8cICgoiPj6ev/zlL7Rp08befu/evYwZM4akpCQaNWrEyJEjmTlzJi4ulcvSWipLRBqCzQeOMzspncXbcjj9k/rqqADG9I6hd+tm5/yZLCJSG1yqvHZR67w6isKriNRXxhjWpP/M7KR0VqUdsR/v3y6IMb2j6RKhNVpFpG6oleu8iohIzbDZDN9uz2F2UjqbDpSvoe3sZOHmTs0Z3TuG1kE+Dq5QRKR2UHgVEXEgm83w9dZsXvxuN2m/rNHq4erEHd3CeeC6KEIba41WEZFfU3gVEXGA03daX1iym9RDBQD4ergwskcko3pE0sRbD7CKiJyNwquIyGVkjGHpjlyeX7KL7dnlW137eLjwwLXR3HttpNZoFRG5AIVXEZHLwBhD0q7DvLBkF5t/mdPq7e7CfT0juf/aaPy8FFpFRCpD4VVE5BIyxrAq7QjPL9nFhn3HAfByc2Zkj0geui6axo3cHFugiEgdo/AqInKJpKT/zAtLdrF2z1Gg/EGsEXGRPBQfTVPNaRURqRaFVxGRGrZuz1Ge/3YXKRk/A+Dm4sQ910Qwunc0gT4eDq5ORKRuU3gVEakhP+07xgtLdrFyd/nmAm7OTtxxdRgP925JsJ9Cq4hITVB4FRG5SJsPHOeFJbtYnnoYABcnC8O6hZHYpyUt/D0dXJ2ISP2i8CoiUk1bD+bx4ne7+W7HIaB8R6zbOocytm9LwgK0uYCIyKWg8CoiUkU/7jnKK8vTSPrlTquTBYZe1YJH+7YismkjB1cnIlK/KbyKiFTC6SWvXlmWxg+Z5asHOFngpk7NeaRfK2KaeTu4QhGRhkHhVUTkPGw2w9KdubyyPI1N+48D4Ops4bYuoYzuFUNEE91pFRG5nBReRUTOwmozfLUlm9eWp7EzpwAoX6f1jm7hPBQfTXM9iCUi4hAKryIiv1JSZuOzDQeZnZxO5pGTQPk2rsPjIrj/2ihtLiAi4mAKryIiQFGplY/W7eeNFRkcPH4KAH8vV+7tEcWoHpH4ebk6uEIREQGFVxFp4E4UlzH3+738e2UmR04UA9DMx50Hr4vi7msiaOSuH5MiIrWJfiqLSIN0vLCEd9bsYc7qPeSdKgWghb8no3tF8/uuYXi4Oju4QhERORuFVxFpUHLzi3h79R7eT9nDyRIrAFFNGzGmdwxDr2yBm4uTgysUEZHzUXgVkQZhW1Yeb63K5MtNWZRaDQBtg31I7NOSQR1DcHayOLhCERGpDIVXEam3bDZD0q5c3lyZyZr0n+3Hu0Q0ZnSvGPq1DcRJoVVEpE5ReBWReudUiZX5Gw7w1qpMMg6XL3fl7GThhg7B3H9tFFeFN3ZwhSIiUl0KryJSb+TmF/Feyl7m/rCXY4XlD2H5uLtwx9VhjOwRSWhjLwdXKCIiF0vhVUTqvO1Z+by1KpMvNh20z2cNbezJfT2jGNYtDG8tdyUiUm/oJ7qI1Ek2myF512HeXJXB6rSK81kfuDaKhPbBeghLRKQeUngVkTpF81lFRBo2hVcRqRNyC4p4P2Uv//le81lFRBoyhVcRqdVy84t4LSmdeT/so8RqAzSfVUSkIdNPfRGplQ4XFPN6cjr/+X4vxWXlobVzuD8PXhet+awiIg2YwquI1CpHT5bwrxXpvLdmL6dKy7dv7RLRmD8OaE2Plk0dXJ2IiDiawquI1Ap5haX8e2UGc1ZncrKkPLR2CvNn/IDWxLdqisWiO60iIqLwKiIOll9UyturMnlrZSYFxWUAdGjhy/gBrenTJlChVUREKlB4FRGHOFFcxrtr9vDGigzyTpWvHtA22IfHB7QmITZIoVVERM5K4VVELqvCkjLeT9nLv1ZkcPRkCQAtA715vH9rbugQjJMexBIRkfNQeBWRy6Ko1MrcH/YxOymNIyfKQ2tU00aM69+KIVc01+oBIiJSKQqvInJJFZdZ+XDtfl5dnkZuQTEA4QFePNqvFUOvbI6Ls5ODKxQRkbpE4VVELgmrzfDxj/t5eelusvOKAGjh78kjfVtya5dQXBVaRUSkGhReRaTGrd97jL98vpVtWfkABPt6kNi3Jbd3DcPNRaFVRESqr0q/RWbMmEG3bt3w8fEhMDCQoUOHkpqaaj9/9OhRHnnkEdq0aYOnpyfh4eE8+uij5OXlVehn3759DB48GC8vLwIDA5kwYQJlZWU1845ExGEOFxTzx483cevsNWzLysfHw4Upg9uRNKE3w7tHKLiKiMhFq9Kd1+TkZBITE+nWrRtlZWVMnjyZhIQEtm/fTqNGjcjKyiIrK4vnnnuO2NhY9u7dy+jRo8nKyuLTTz8FwGq1MnjwYIKDg1mzZg3Z2dmMGDECV1dXnnnmmUvyJkXk0iq12ngvZS8vLtllX6v19q5hTLi+DU293R1cnYiI1CcWY4yp7osPHz5MYGAgycnJxMfHn7XNJ598wj333MPJkydxcXFh0aJFDBkyhKysLIKCggB4/fXXeeKJJzh8+DBubm4XvG5+fj5+fn7k5eXh6+tb3fJFpAasST/CU19sY9ehEwBcEerHtJvac1V4YwdXJiIijnSp8tpFzXk9PR0gICDgvG18fX1xcSm/VEpKCh07drQHV4CBAwcyZswYtm3bxlVXXXVGH8XFxRQXF9s/z8/Pv5iyRaQGZOedYvpXO1i4ORuAxl6uTLy+Lbd3DdNarSIicslUO7zabDbGjRtHz5496dChw1nbHDlyhKeffpqHHnrIfiwnJ6dCcAXsn+fk5Jy1nxkzZjBt2rTqlioiNai4zMpbqzKZtTSNU6VWnCxw9zUR/DGhNf5eF/7LiYiIyMWodnhNTExk69atrFq16qzn8/PzGTx4MLGxsTz11FPVvQwAkyZNYvz48RX6DgsLu6g+RaTqklJzmfbldjKPnASga0Rjpt3cnvbN/RxcmYiINBTVCq9jx45l4cKFrFixgtDQ0DPOFxQUcP311+Pj48OCBQtwdXW1nwsODmbt2rUV2h86dMh+7mzc3d1xd9dDHyKOsv9oIX9duJ0l28v/X23m487kQW0ZemULLBZNERARkcunSuHVGMMjjzzCggULSEpKIioq6ow2+fn5DBw4EHd3d7744gs8PDwqnI+Li2P69Onk5uYSGBgIwJIlS/D19SU2NvYi3oqI1LSiUiuzk9KZnZxOSZkNFycL9/aM5NF+rfDxcL1wByIiIjWsSuE1MTGRefPm8fnnn+Pj42Ofo+rn54enpyf5+fkkJCRQWFjIf/7zH/Lz8+0PVzVr1gxnZ2cSEhKIjY1l+PDhPPvss+Tk5DBlyhQSExN1d1WkljDG8M22Q/ztq+0cOHYKgB4xTZh2U3taBfk4uDoREWnIqrRU1rn+PDhnzhxGjRpFUlISffr0OWubzMxMIiMjAdi7dy9jxowhKSmJRo0aMXLkSGbOnGlfkeBCtFSWyKWz61ABTy/czsrdRwBo7ufBlCGx3NAhWFMERESk0i5VXruodV4dReFVpGYZY1iVdoS3VmWSlHoYADdnJx6MjyKxT0u83LSTtIiIVE2tXOdVROq2olIrX2zM4q1VmaQeKgDAYoGBscE8cUNbopo2cnCFIiIiFSm8ijRAR04U85/v9/Kf7/dy5EQJAF5uzgzrGsa9PSOJaKLQKiIitZPCq0gDkppTwNurMlmw8SAlZTagfE7rqJ6R3N4tHD9PrSAgIiK1m8KrSD1njCF512HeWpVpfwgLoFOYPw9cG8X1HYJxdXZyYIUiIiKVp/AqUk8VlVpZsOEgb63KJC33BABOFri+QzD3XxtF5/DGWj1ARETqHIVXkXomt6CI91P2MveHfRw9WT6f1dvdhdu7hTGqRyRhAV4OrlBERKT6FF5F6ontWfm8tSqTLzdlUWItn8/awt+Te3tGcnu3MO2IJSIi9YLCq0gdtzbzKK8sT2PFrsP2Y10iGnP/tVEkxAbhovmsIiJSjyi8itRBxhhW7j7CK8vTWJt5FABnJws3/DKf9arwxg6uUERE5NJQeBWpQ2w2w9KdubyybDebDuQB5Tth3dY1lNHxMYQ30XxWERGp3xReReoAq83w9ZZsXl2exs6c8p2wPFyduPPqcB6KjybEz9PBFYqIiFweCq8itVip1cZnGw4yOymdjCMngfKVA4bHRXD/tVE09XZ3cIUiIiKXl8KrSC1UVGrlk/UHeD0pnYPHTwHg5+nKfT2jGNUjEj8vrRwgIiINk8KrSC1SWFLGvB/28caKDHILigFo6u3GA9dFc0/3CLzd9b+siIg0bPpNKFIL5BeV8n7KXt5alWnfWCDEz4M/xEdzx9XheLg6O7hCERGR2kHhVcSBjp4sYc7qTN5Zs4eCojIAwgO8eLh3DL/rHIqbi9ZoFRER+TWFVxEH2H+0kLdWZfLxj/spLLEC0DLQm7F9WjLkihBtLCAiInIOCq8il9FP+47x5soMFm/NwWbKj7Vv7svYPi0Z2D4YJyeLYwsUERGp5RReRS4xq82wZPsh/r0yg/V7j9mPX9eqKQ9eF811rZpisSi0ioiIVIbCq8glUlhSxqfrD/DWqkz2/lwIgKuzhZs6teCB66JoF+Lr4ApFRETqHoVXkRqWm1/Euyl7mPvDPo4XlgLla7TefU04I3tEEuTr4eAKRURE6i6FV5EakppTwJsrM/h8YxYlVhtQvnLA/ddGcVuXUBppjVYREZGLpt+mIhfBGMOqtCP8e2UmK3Ydth/vHO7Pg9dFk9A+GGc9hCUiIlJjFF5FqqGkzMYXm7J4c2UGO3MKAHCywMD2wTxwXTRdIho7uEIREZH6SeFVpAqKSq28s2YPb6/KtG/f6uXmzLCuYdzbM5KIJo0cXKGIiEj9pvAqUknfbT/EtIXb2H/0FACBPu6M6hnJ3VdH4Ofl6uDqREREGgaFV5EL2PvzSaZ9uZ1lO3MBCPb1YHxCa4Ze2ULbt4qIiFxmCq8i53CqxMrspDReX5FBSZkNV2cL918bzSN9W2rlABEREQfRb2CR3zDG8O32Q/z1y+0cPF4+ReC6Vk2ZemN7WgZ6O7g6ERGRhk3hVeRXMg6fYNqX20n+Zdmr5n4e/HlILNd3CNYWriIiIrWAwqsI5Vu5vrIsjTdXZlJiteHm7MSD8VEk9mmJl5v+NxEREakt9FtZGjRjDIu25vC3hdvJyisCoFfrZjx1U3uimmrZKxERkdpG4VUarLTcEzz1xTZWpR0BoIW/J1NvjGVAbJCmCIiIiNRSCq/S4JwsLuPlZbt5e1UmpVaDm4sTo3vFMKZXDJ5uzo4uT0RERM5D4VUaDGMMCzdnM/2rHeTkl08R6Nc2kL/cGKudsUREROoIhVdpEHYdKmDq59tIyfgZgPAAL6beGEu/dkEOrkxERESqQuFV6rX0wyd4eeluvtiUhTHg7uLEw71b8ode0Xi4aoqAiIhIXaPwKvVS5pGTvLx0N59vPIjNlB+7vn0wfxrcjrAAL8cWJyIiItWm8Cr1yp4jJ3l52W4+2/C/0Nq/XSDj+remQws/xxYnIiIiF82pKo1nzJhBt27d8PHxITAwkKFDh5KamlqhzRtvvEHv3r3x9fXFYrFw/PjxM/o5evQod999N76+vvj7+3P//fdz4sSJi3oj0rDt+7mQCZ9sot/zycz/qTy49m0byBdje/LmyG4KriIiIvVElcJrcnIyiYmJfP/99yxZsoTS0lISEhI4efKkvU1hYSHXX389kydPPmc/d999N9u2bWPJkiUsXLiQFStW8NBDD1X/XUiDtf9oIU98upm+/0zik/UHsNoMvds047PEnrw9qhtXhPo7ukQRERGpQRZjjKnuiw8fPkxgYCDJycnEx8dXOJeUlESfPn04duwY/v7+9uM7duwgNjaWdevW0bVrVwAWL17MoEGDOHDgAM2bN7/gdfPz8/Hz8yMvLw9fX9/qli912IFjhby6PI1PfjxA2S/zA+JbN2Nc/1Z0Dm/s4OpERETkUuW1i5rzmpeXB0BAQEClX5OSkoK/v789uAL0798fJycnfvjhB2655ZaLKUnquYPHT/0SWvdTai0Prde1asq4/q3pEqHQKiIiUt9VO7zabDbGjRtHz5496dChQ6Vfl5OTQ2BgYMUiXFwICAggJyfnrK8pLi6muLjY/nl+fn71ipY6KzuvPLR+tO5/obVnyyY83r81XSMr/48nERERqduqHV4TExPZunUrq1atqsl6zmrGjBlMmzbtkl9Hap+cvCJeS0rjw7X7KbHaAIiLbsLjA1pzdZRCq4iISENTrfA6duxY+4NWoaGhVXptcHAwubm5FY6VlZVx9OhRgoODz/qaSZMmMX78ePvn+fn5hIWFVb1wqTPyi0qZtXQ376bspaSsPLReExXA4wNa0z26iYOrExEREUepUng1xvDII4+wYMECkpKSiIqKqvIF4+LiOH78OOvXr6dLly4ALFu2DJvNxjXXXHPW17i7u+Pu7l7la0ndY7MZ/vvTAf6+OJUjJ8qnilwdGcC4Aa3oEdPUwdWJiIiIo1UpvCYmJjJv3jw+//xzfHx87HNU/fz88PT0BMrntObk5JCWlgbAli1b8PHxITw8nICAANq1a8f111/Pgw8+yOuvv05paSljx47ljjvuqNRKA1J/bdx/nKlfbGPT/uMARDdtxJ9vjKV362ZYLBbHFiciIiK1QpWWyjpXgJgzZw6jRo0C4Kmnnjrr/NRftzl69Chjx47lyy+/xMnJiVtvvZWXX34Zb2/vStWhpbLql8MFxfzjm518/OMBABq5OfNY/1aM6hGFm0uVliIWERGRWuJS5bWLWufVURRe64dSq4131+zhpe92U1BcBsDvOrfgyevbEujr4eDqRERE5GLUynVeRapr1e4jPPXlNtJyy7cF7tjCj6duaq+1WkVEROS8FF7lstp/tJDpX+1g8bby+dIBjdyYOLANv+8ahrOT5rWKiIjI+Sm8ymVxqsTK7OR0/pWcTnGZDWcnC8O7R/B4/9b4ebk6ujwRERGpIxRe5ZIyxrBoaw7Tv9rBweOngPJNBqbeFEvbYM1XFhERkapReJVLJjWngGlfbmNN+s8AtPD35E+D23FDh2AtfSUiIiLVovAqNS6vsJQXvtvF+9/vxWozuLk4MbpXDGN6xeDp5uzo8kRERKQOU3iVGnO8sIQP1+3njRUZHD1ZAsDA9kFMGRxLWICXg6sTERGR+kDhVS5aak4B76zZw4INBygqtQHQMtCbqTfGcl2rZg6uTkREROoThVepFqvNsGxnLu+syWR12s/24+1CfLm3RyS3dG6Bq7N2xxIREZGapfAqVZJfVMrH6/bzXspe9h0tBMDJAgmxwdzbM5KrowL0MJaIiIhcMgqvUinph0/w7po9fLr+AIUlVgB8PVy48+pwhsdFENpYc1pFRETk0lN4lXOy2Qwrdh9mzuo9JO86bD/eKtCbUT0jueWqFni56VtIRERELh8lDznDyeIy/vvTAd5Zs4eMwycBsFigX9tARvWIomfLJpoaICIiIg6h8Cp2+34u5N2UPXy8bj8FxWUA+Li78PuuYYyIiyCyaSMHVygiIiINncJrA2eMYW3mUd5clcl3Ow5hTPnxqKaNGNUjklu7hOLtrm8TERERqR2UShqoMquNRVtzeHNlBpsO5NmPx7duxr09I+nVqhlOTpoaICIiIrWLwmsDc6K4jI/X7eetVZkcPH4KAHcXJ27tEsp9PaNoGejt4ApFREREzk3htYHIySvinTV7mPvDXgqKyuezBjRyY0RcBMO7R9DE293BFYqIiIhcmMJrPbczJ59/r8jki00HKbWWT2iNbtqIB66L5nedW+Dh6uzgCkVEREQqT+G1HjLGsCrtCG+syGDl7iP241dHBvBgfDT92gZqPquIiIjUSQqv9UhJmY2Fm7N4Y0UGO3MKgPKtW2/oGMKD10VzZZi/YwsUERERuUgKr/VA3qlSPli7jzmrMzmUXwyAl5szw7qGcf+1UYQFaOtWERERqR8UXuuwA8cKmbN6Dx+u3cfJEisAgT7ujOoZyd1XR+Dn5ergCkVERERqlsJrHZRx+ASvLk/ns40HsdrKH8JqE+TDg/HR3NgpBHcXPYQlIiIi9ZPCax2yMyefV5al8dWWbPtOWD1bNuGh+BjiWzXFYtFDWCIiIlK/KbzWAZv2H+eV5Wks2X7Ifqx/uyDG9m2ph7BERESkQVF4rcXW7TnKrGVprNh1GACLBQZ1DCGxd0tim/s6uDoRERGRy0/htZYxxrA67WdmLdvND5lHAXB2snDzlc15uHdLbd8qIiIiDZrCay1hjGHZzlxmLUtj4/7jALg6W7itSxhjesUQ3kTLXYmIiIgovDqYzWZYvC2HWcvS2JGdD4C7ixN3Xh3OH3pFE+Ln6eAKRURERGoPhVcHKbPa+GJTFq8lpZOWewKARm7O3BMXwQPXRtPMx93BFYqIiIjUPgqvl1lxmZX5Px1kdlI6+44WAuDr4cKonlHc1zMSfy83B1coIiIiUnspvF4mJ4rL+HDtPt5cmUlOfhEAAY3cuP/aKIbHReDrod2wRERERC5E4fUSO3KimHdW7+G9lD3kF5UB5Vu4/qFXDHdeHYaXm74EIiIiIpWl5HSJ7Pu5kDdWpvPJjwcoLrMBEN20EQ/FR3NL5xbawlVERESkGhRea9jWg3m8npzO11uysf2yhWunMH/G9IpmQGwwzk7awlVERESkuhRea4AxhpT0n5mdnM7K3Ufsx3u1bsboXjF0jw7AYlFoFREREblYCq8XwWozfLMth9eT09l8IA8AJwsMuaI5f+gVTfvmfg6uUERERKR+UXithtPLXb2xIoPMIyeB8o0Fbu8WxoPXRRMWoN2wRERERC4FhdcqyC8qZe73+3h7dSaHC4oB8PN0ZWRcBCN7RNLEWxsLiIiIiFxKTlVpPGPGDLp164aPjw+BgYEMHTqU1NTUCm2KiopITEykSZMmeHt7c+utt3Lo0KEKbfbt28fgwYPx8vIiMDCQCRMmUFZWdvHv5hLJzS9i5qKd9JyxjL8v3snhgmJC/DyYMrgda57sy/iENgquIiIiIpdBle68Jicnk5iYSLdu3SgrK2Py5MkkJCSwfft2GjVqBMDjjz/OV199xSeffIKfnx9jx47ld7/7HatXrwbAarUyePBggoODWbNmDdnZ2YwYMQJXV1eeeeaZmn+HF6nMamPIrFXk/nKntVWgN3/oFcNNnZrj5lKl7C8iIiIiF8lijDHVffHhw4cJDAwkOTmZ+Ph48vLyaNasGfPmzeO2224DYOfOnbRr146UlBS6d+/OokWLGDJkCFlZWQQFBQHw+uuv88QTT3D48GHc3C68PWp+fj5+fn7k5eXh6+tb3fIr7Z/fppKS/jOje8XQt20gTlruSkREROS8LlVeu6hbh3l55U/YBwQEALB+/XpKS0vp37+/vU3btm0JDw8nJSUFgJSUFDp27GgPrgADBw4kPz+fbdu2nfU6xcXF5OfnV/i4nB7r14pPx/Sgf2yQgquIiIiIA1U7vNpsNsaNG0fPnj3p0KEDADk5Obi5ueHv71+hbVBQEDk5OfY2vw6up8+fPnc2M2bMwM/Pz/4RFhZW3bKrxcVZ0wNEREREaoNqp7LExES2bt3Khx9+WJP1nNWkSZPIy8uzf+zfv/+SX1NEREREap9qLZU1duxYFi5cyIoVKwgNDbUfDw4OpqSkhOPHj1e4+3ro0CGCg4PtbdauXVuhv9OrEZxu81vu7u64u+tpfhEREZGGrkp3Xo0xjB07lgULFrBs2TKioqIqnO/SpQuurq4sXbrUfiw1NZV9+/YRFxcHQFxcHFu2bCE3N9feZsmSJfj6+hIbG3sx70VERERE6rkq3XlNTExk3rx5fP755/j4+NjnqPr5+eHp6Ymfnx/3338/48ePJyAgAF9fXx555BHi4uLo3r07AAkJCcTGxjJ8+HCeffZZcnJymDJlComJibq7KiIiIiLnVaWlsiyWsz9pP2fOHEaNGgWUb1Lwxz/+kQ8++IDi4mIGDhzIa6+9VmFKwN69exkzZgxJSUk0atSIkSNHMnPmTFxcKpelL/dSWSIiIiJSNZcqr13UOq+OovAqIiIiUrvVynVeRUREREQuJ4VXEREREakzFF5FREREpM5QeBURERGROqNamxQ42ulnzPLz8x1ciYiIiIiczemcVtNrA9TJ8FpQUABAWFiYgysRERERkfMpKCjAz8+vxvqrk0tl2Ww2srKy8PHxOefaszUpPz+fsLAw9u/fr6W5LiONu2No3B1D4+4YGnfH0Lg7xuUed2MMBQUFNG/eHCenmpupWifvvDo5OREaGnrZr+vr66v/yRxA4+4YGnfH0Lg7hsbdMTTujnE5x70m77iepge2RERERKTOUHgVERERkTpD4bUS3N3dmTp1Ku7u7o4upUHRuDuGxt0xNO6OoXF3DI27Y9SXca+TD2yJiIiISMOkO68iIiIiUmcovIqIiIhInaHwKiIiIiJ1hsKriIiIiNQZtSq8zpgxg27duuHj40NgYCBDhw4lNTW1QpuioiISExNp0qQJ3t7e3HrrrRw6dMh+ftOmTdx5552EhYXh6elJu3bteOmll855zdWrV+Pi4sKVV155wfqMMfzlL38hJCQET09P+vfvz+7duyu0uemmmwgPD8fDw4OQkBCGDx9OVlbWefvNzs7mrrvuonXr1jg5OTFu3Lgz2pSWlvLXv/6VmJgYPDw86NSpE4sXL75gzZXRUMd9/vz5DBgwgGbNmuHr60tcXBzffPPNOdvPnDkTi8Vy1q9PdWjczz3uBQUFjBs3joiICDw9PenRowfr1q27YM2VUR/G/bTi4mKuvPJKLBYLGzduvGDfSUlJdO7cGXd3d1q2bMk777xT4fzs2bO54oor7AuYx8XFsWjRogv2Wxka93OPu9Vq5c9//jNRUVF4enoSExPD008/XSP7wTfUca/M79X58+fTtWtX/P39adSoEVdeeSXvv//+BWuuDI37uccd4MUXX6RNmzZ4enoSFhbG448/TlFR0QXr/vUbqDUGDhxo5syZY7Zu3Wo2btxoBg0aZMLDw82JEyfsbUaPHm3CwsLM0qVLzY8//mi6d+9uevToYT//1ltvmUcffdQkJSWZ9PR08/777xtPT08za9asM6537NgxEx0dbRISEkynTp0uWN/MmTONn5+f+eyzz8ymTZvMTTfdZKKiosypU6fsbZ5//nmTkpJi9uzZY1avXm3i4uJMXFzcefvNzMw0jz76qHn33XfNlVdeaR577LEz2kycONE0b97cfPXVVyY9Pd289tprxsPDw/z0008XrPtCGuq4P/bYY+bvf/+7Wbt2rdm1a5eZNGmScXV1PeuYrl271kRGRporrrjirF+f6tC4n3vchw0bZmJjY01ycrLZvXu3mTp1qvH19TUHDhy4YN0XUh/G/bRHH33U3HDDDQYwGzZsOG+/GRkZxsvLy4wfP95s377dzJo1yzg7O5vFixfb23zxxRfmq6++Mrt27TKpqalm8uTJxtXV1WzduvWCdV+Ixv3c4z59+nTTpEkTs3DhQpOZmWk++eQT4+3tbV566aUL1n0hDXXcK/N7dfny5Wb+/Plm+/btJi0tzbz44otnfG2qS+N+7nGfO3eucXd3N3PnzjWZmZnmm2++MSEhIebxxx+/YN2n1arw+lu5ubkGMMnJycYYY44fP25cXV3NJ598Ym+zY8cOA5iUlJRz9vPwww+bPn36nHH89ttvN1OmTDFTp0694BfbZrOZ4OBg849//MN+7Pjx48bd3d188MEH53zd559/biwWiykpKTlv/6f16tXrrF/skJAQ88orr1Q49rvf/c7cfffdleq3KhriuJ8WGxtrpk2bVuFYQUGBadWqlVmyZMk5vz41QeNePu6FhYXG2dnZLFy4sEKbzp07mz/96U9V6rcy6uq4f/3116Zt27Zm27ZtlfqlMnHiRNO+ffszahs4cOB5X9e4cWPz5ptvnrdNdWjc/zfugwcPNvfdd1+FNvr5fnHj/mtV+bl91VVXmSlTplS678rSuP9PYmKi6du3b4Vj48ePNz179qx037Vq2sBv5eXlARAQEADA+vXrKS0tpX///vY2bdu2JTw8nJSUlPP2c7qP0+bMmUNGRgZTp06tVC2ZmZnk5ORUuLafnx/XXHPNOa999OhR5s6dS48ePXB1da3Udc6luLgYDw+PCsc8PT1ZtWrVRfV7Ng113G02GwUFBWfUnJiYyODBgyvUcClo3MtrLisrw2q16vv9F2cb90OHDvHggw/y/vvv4+XlVam+U1JSzvgeHjhw4Dnfk9Vq5cMPP+TkyZPExcVV6hpVoXH/X789evRg6dKl7Nq1Cyj/c/GqVau44YYbKnWNqmgo415VxhiWLl1Kamoq8fHxNd6/xv1/evTowfr161m7di0AGRkZfP311wwaNKjSfbjUWDU1zGazMW7cOHr27EmHDh0AyMnJwc3NDX9//wptg4KCyMnJOWs/a9as4aOPPuKrr76yH9u9ezdPPvkkK1euxMWlckNwuv+goKALXvuJJ57glVdeobCwkO7du7Nw4cJKXeN8Bg4cyPPPP098fDwxMTEsXbqU+fPnY7VaL7rvX2vI4/7cc89x4sQJhg0bZj/24Ycf8tNPP9XYfMtz0bj/b9x9fHyIi4vj6aefpl27dgQFBfHBBx+QkpJCy5Ytq9T3hdTFcTfGMGrUKEaPHk3Xrl3Zs2dPpfs+W7/5+fmcOnUKT09PALZs2UJcXBxFRUV4e3uzYMECYmNjK3WNytK4Vxz3J598kvz8fNq2bYuzszNWq5Xp06dz9913V+oaldWQxr2y8vLyaNGiBcXFxTg7O/Paa68xYMCAGr2Gxr2iu+66iyNHjnDttddijKGsrIzRo0czefLkSvdRa++8JiYmsnXrVj788MNq97F161Zuvvlmpk6dSkJCAlB+N+Guu+5i2rRptG7d+qyvmzt3Lt7e3vaPlStXVum6EyZMYMOGDXz77bc4OzszYsQI+8T7X/c7evToSvf50ksv0apVK9q2bYubmxtjx47l3nvvxcmpZr+EDXXc582bx7Rp0/j4448JDAwEYP/+/Tz22GPMnTv3jLuANU3j/r9xB3j//fcxxtCiRQvc3d15+eWXufPOO/X9DsyaNYuCggImTZp0zjbV/TkD0KZNGzZu3MgPP/zAmDFjGDlyJNu3b69SHxeica/o448/Zu7cucybN4+ffvqJd999l+eee45333230n1Uhsb9TD4+PmzcuJF169Yxffp0xo8fT1JSUpX6uBCNe0VJSUk888wzvPbaa/z000/Mnz+fr776iqeffrrSfdTKOa+JiYkmNDTUZGRkVDi+dOlSA5hjx45VOB4eHm6ef/75Cse2bdtmAgMDzeTJkyscP3bsmAGMs7Oz/cNisdiPLV261OTn55vdu3fbPwoLC016evpZ53vEx8ebRx999JzvZf/+/QYwa9asMcaYCv0eOnTojPYXmptz6tQpc+DAAWOz2czEiRNNbGzsOdtWVUMd9w8++MB4enqeMcdywYIFZ9QMGIvFYpydnU1ZWdk5r18VGveFZ+vKGGPMiRMnTFZWljGm/CGuQYMGnbNtVdXVcb/55puNk5PTGd+Xzs7OZsSIEcaYs4/7ddddd8bPlrffftv4+vqed5z69etnHnroofO2qQqN+5njHhoaesYzDU8//bRp06bNhQe0khrauP9aVea83n///SYhIaFSbStD4/7YGcevvfZa83//938Vjp1+GM1qtZ5zLH+tVoVXm81mEhMTTfPmzc2uXbvOOH96gvOnn35qP7Zz584zJjhv3brVBAYGmgkTJpzRh9VqNVu2bKnwMWbMGNOmTRuzZcuWCk8C/ra24OBg89xzz9mP5eXlXfABlr179xrALF++vDJDUOn/yUpKSkxMTIyZNGlSpfo9n4Y87vPmzTMeHh7ms88+O+Ncfn7+GTV37drV3HPPPWbLli3n7bcyNO5nH/ezOXr0qPHz8zP/+te/KtX+fOr6uO/du7dCv998840BzKeffmr2799/zvc9ceJE06FDhwrH7rzzzgs+sNWnTx8zcuTI87apDI37//x23AMCAsxrr71Woc0zzzxjWrVqdc5+K6uhjvuvVSW83nvvvaZXr16Vans+Gvdzj3vnzp3NxIkTKxybN2+e8fT0rPRNoVoVXseMGWP8/PxMUlKSyc7Otn8UFhba24wePdqEh4ebZcuWmR9//PGMpXm2bNlimjVrZu65554KfeTm5p7zupV5Os+Y8qUl/P39zeeff242b95sbr755gpLS3z//fdm1qxZZsOGDWbPnj1m6dKlpkePHiYmJsYUFRWdt+8NGzaYDRs2mC5dupi77rrLbNiwwWzbts1+/vvvvzf//e9/TXp6ulmxYoXp27eviYqKOuNfbdXRUMd97ty5xsXFxbz66qsVaj5+/Pg5X1OTqw1o3M897osXLzaLFi0yGRkZ5ttvvzWdOnUy11xzTZVXMTibuj7uv5WZmVmlJZsmTJhgduzYYV599dUzlgV68sknTXJyssnMzDSbN282Tz75pLFYLObbb7+9YN0XonE/97iPHDnStGjRwr5U1vz5803Tpk3P+AVfHQ113I258O/VZ555xnz77bcmPT3dbN++3Tz33HPGxcXF/Pvf/75g3xeicT/3uE+dOtX4+PiYDz74wP4zPiYmxgwbNuyCfZ9Wq8IrcNaPOXPm2NucOnXKPPzww6Zx48bGy8vL3HLLLSY7O9t+furUqWftIyIi4pzXrewX22azmT//+c8mKCjIuLu7m379+pnU1FT7+c2bN5s+ffqYgIAA4+7ubiIjI83o0aMrtTblhWpOSkoy7dq1M+7u7qZJkyZm+PDh5uDBgxfstzIa6rj36tXrrDWf7y5TTYZXjfu5x/2jjz4y0dHRxs3NzQQHB5vExMTz/qOiKur6uP9WVX6pLF++3Fx55ZXGzc3NREdHV3jPxhhz3333mYiICOPm5maaNWtm+vXrVyPB1RiN+/nGPT8/3zz22GMmPDzceHh4mOjoaPOnP/3JFBcXX7DvC2nI436hmv/0pz+Zli1bGg8PD9O4cWMTFxdnPvzwwwv2Wxka93PXXFpaap566ikTExNjPDw8TFhYmHn44YerdDPO8suFRERERERqvVq72oCIiIiIyG8pvIqIiIhInaHwKiIiIiJ1hsKriIiIiNQZCq8iIiIiUmcovIqIiIhInaHwKiIiIiJ1hsKriIiIiNQZCq8iIjVg1KhRWCwWLBYLrq6uBAUFMWDAAN5++21sNtsZ7QcOHIizszPr1q0DYM+ePfbXn+vjnXfeISkp6Zznc3JyLvfbFhG57BReRURqyPXXX092djZ79uxh0aJF9OnTh8cee4whQ4ZQVlZmb7dv3z7WrFnD2LFjefvttwEICwsjOzvb/vHHP/6R9u3bVzh2++232/tITU2tcC47O5vAwMDL/p5FRC43F0cXICJSX7i7uxMcHAxAixYt6Ny5M927d6dfv3688847PPDAAwDMmTOHIUOGMGbMGLp3787zzz+Pp6en/bUA3t7euLi4VDj2a4GBgfj7+1/y9yQiUtvozquIyCXUt29fOnXqxPz58wEwxjBnzhzuuece2rZtS8uWLfn0008dXKWISN2h8Coicom1bduWPXv2APDdd99RWFjIwIEDAbjnnnt46623qtxnaGgo3t7e9o/27dvXZMkiIrWWpg2IiFxixhgsFgsAb7/9NrfffjsuLuU/fu+8804mTJhAeno6MTExle5z5cqV+Pj42D93dXWt2aJFRGophVcRkUtsx44dREVFcfToURYsWEBpaSmzZ8+2n7darbz99ttMnz690n1GRUVpzquINEiaNiAicgktW7aMLVu2cOuttzJ37lxCQ0PZtGkTGzdutH/885//5J133sFqtTq6XBGRWk93XkVEakhxcTE5OTlYrVYOHTrE4sWLmTFjBkOGDGHEiBF06dKF2267jQ4dOlR4XVhYGJMmTWLx4sUMHjy4UtfKzc2lqKiowrEmTZpo+oCI1HsKryIiNWTx4sWEhITg4uJC48aN6dSpEy+//DIjR45kw4YNbNq0iX//+99nvM7Pz49+/frx1ltvVTq8tmnT5oxjKSkpdO/e/aLfh4hIbWYxxhhHFyEiIiIiUhma8yoiIiIidYbCq4iIiIjUGQqvIiIiIlJnKLyKiIiISJ2h8CoiIiIidYbCq4iIiIjUGQqvIiIiIlJnKLyKiIiISJ2h8CoiIiIidYbCq4iIiIjUGQqvIiIiIlJnKLyKiIiISJ3x/3GVMT2THc0FAAAAAElFTkSuQmCC",
            "text/plain": [
              "\u003cFigure size 800x400 with 1 Axes\u003e"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "serve['flipper_length_mm'].plot(legend = True, figsize = (8, 4))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "tags": [],
        "id": "o4JOtUWsxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "\u003cAxes: xlabel='DATE'\u003e"
            ]
          },
          "execution_count": 28,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAArgAAAFzCAYAAADVB3rFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABZsUlEQVR4nO3deVzVVf7H8ddlB2VRlE0BQVREcy/FXMoNDZvWadHUUjMLy+U3aU5NaptNu2Vm06bTaLbqNJombqVJ7hbugigoIG5wQWS79/v7g7xFbqDoZXk/H4/7mPh+zz338z0ivOd4vudrMgzDQERERESkhnCwdwEiIiIiIpVJAVdEREREahQFXBERERGpURRwRURERKRGUcAVERERkRpFAVdEREREahQFXBERERGpURRwRURERKRGcbJ3AVeL1WolPT0dT09PTCaTvcsRERERkT8xDIPc3FyCgoJwcKi8edcaG3DT09MJDg62dxkiIiIicglpaWk0bty40vqrsQHX09MTKB0wLy8vO1cjIiIiIn9mNpsJDg625bbKUmMD7tllCV5eXgq4IiIiIlVYZS8n1U1mIiIiIlKjKOCKiIiISI2igCsiIiIiNUqNXYNbHoZhUFJSgsVisXcpUk04Ojri5OSkredERESqsFobcIuKisjIyCA/P9/epUg14+HhQWBgIC4uLvYuRURERM6jVgZcq9VKSkoKjo6OBAUF4eLiohk5uSTDMCgqKuLYsWOkpKTQrFmzSt2UWkRERCpHrQy4RUVFWK1WgoOD8fDwsHc5Uo24u7vj7OzMoUOHKCoqws3Nzd4liYiIyJ/U6uknzb7J5dD3jYiISNWm39QiIiIiUqMo4IqIiIgIJRYrC7cd5pe0bHuXcsUUcKuRm266iXHjxlVqn3PmzMHHx6dS+xQREZHqo6jEyuebUun9xg+M//wX3ojfZ++SrlitvMlMREREpLYrKLbwxeY0Zq9JJj2nAID6dVy4Iaw+VquBg0P13WFKAVdERESkFskvKmH+hlTe//EAx3ILAWjo6cojPcIZ1DkED5fqHw+1RIHS/U3zi0rs8jIMo0K1lpSUMGbMGLy9vWnQoAH/+Mc/bH2cOnWKoUOHUq9ePTw8PBgwYAD79+8v8/45c+YQEhKCh4cHd9xxBydOnLCdO3jwIA4ODmzevLnMe9566y1CQ0OxWq0XrW3NmjWYTCa+//572rdvj7u7O7169SIrK4ulS5fSsmVLvLy8GDRoUJkHbCxbtoxu3brh4+ODr68vAwcOJDk52Xa+qKiIMWPGEBgYiJubG6GhoUyfPt32Zzd16lRCQkJwdXUlKCiIJ554olxjmZGRQWxsLO7u7oSFhTF//nyaNGnCW2+9Va73i4iIVCe5BcW8uzqJbv9czQtLdnMst5Agbzeeu60VayfezMju4TUi3IJmcAE4U2wh6tnv7fLZu56LqdA309y5cxkxYgQbN25k8+bNjBo1ipCQEB5++GEefPBB9u/fz7fffouXlxeTJk3illtuYdeuXTg7O7NhwwZGjBjB9OnTuf3221m2bBlTpkyx9d2kSRP69OnDJ598QqdOnWzHP/nkEx588MFyb481depUZs6ciYeHB/fccw/33HMPrq6uzJ8/n7y8PO644w7eeecdJk2aBMDp06eZMGECbdq0IS8vj2effZY77riD7du34+DgwNtvv823337LF198QUhICGlpaaSlpQHw9ddf8+abb7JgwQJatWpFZmYmv/zyS7nqHDp0KMePH2fNmjU4OzszYcIEsrKyyvtHISIiUi1k5xfxyU8H+eSnFMwFJQCE1PfgsZuacmeHxrg41bz5TgXcaiY4OJg333wTk8lEixYtSExM5M033+Smm27i22+/5aeffqJr164AzJs3j+DgYBYtWsRf//pXZsyYQf/+/Zk4cSIAzZs3Z/369SxbtszW/8iRIxk9ejRvvPEGrq6ubN26lcTERP773/+Wu8YXXniBG2+8EYARI0YwefJkkpOTCQ8PB+Duu+9m9erVtoB71113lXn/xx9/TMOGDdm1axetW7cmNTWVZs2a0a1bN0wmE6Ghoba2qampBAQE0KdPH5ydnQkJCeGGG264ZI179uxhxYoVbNq0yRbmP/zwQ5o1a1bu6xQREanKjucV8tG6FD5NOEReYWmwbdqwDmN6RXBrmyCcHGtesD1LARdwd3Zk13MxdvvsiujSpUuZxwpHR0fz+uuvs2vXLpycnOjcubPtnK+vLy1atGD37t0A7N69mzvuuKNMf9HR0WUC7u23305cXBwLFy7kvvvuY86cOdx88800adKk3DW2adPG9t/+/v54eHjYwu3ZYxs3brR9vX//fp599lk2bNjA8ePHbUshUlNTad26NQ8++CB9+/alRYsW9O/fn4EDB9KvXz8A/vrXv/LWW28RHh5O//79ueWWW7j11ltxcrr4t/bevXtxcnKiQ4cOtmMRERHUq1ev3NcpIiJSFR01F/CvHw8wb8MhCopLf6dGBnjyeK9m9G8dgGM1vnmsvBRwAZPJVGPWnFwpFxcXhg4dyieffMKdd97J/PnzmTFjRoX6cHZ2tv23yWQq8/XZY39cz3vrrbcSGhrKBx98QFBQEFarldatW1NUVARAhw4dSElJYenSpaxYsYJ77rmHPn368NVXXxEcHMzevXtZsWIF8fHxPPbYY7z66qv88MMP53yuiIhITXb4VD7v/3CAzzenUVRS+nu2TWNvHu/VjN6RftV6V4SKUqqrZjZs2FDm659//plmzZoRFRVFSUkJGzZssC1ROHHiBHv37iUqKgqAli1bnvf9fzZy5Ehat27NrFmzKCkp4c4777xKV/N7jR988AHdu3cHYN26dee08/Ly4t577+Xee+/l7rvvpn///pw8eZL69evj7u7Orbfeyq233kpcXByRkZEkJiaWmZ39sxYtWlBSUsK2bdvo2LEjAElJSZw6derqXKiIiMhVcvD4aWatSeKbrUcosZbeeN4ptB6P925Gj2YNyvzLb22hgFvNpKamMmHCBB555BG2bt3KO++8w+uvv06zZs247bbbePjhh3n//ffx9PTkqaeeolGjRtx2220APPHEE9x444289tpr3HbbbXz//fdlliec1bJlS7p06cKkSZMYPnw47u7uV+166tWrh6+vL//6178IDAwkNTWVp556qkybN954g8DAQNq3b4+DgwNffvklAQEB+Pj4MGfOHCwWC507d8bDw4P//Oc/uLu7l1mnez6RkZH06dOHUaNG8d577+Hs7Mz//d//4e7uXit/EIiISPVz8Php3lmVxKLtR7D8Fmy7NvXl8V7N6BJev1b/Pqu5q4trqKFDh3LmzBluuOEG4uLiGDt2LKNGjQJKdzvo2LEjAwcOJDo6GsMw+O6772z/VN+lSxc++OADZsyYQdu2bVm+fDnPPPPMeT9nxIgRFBUVMXz48Kt6PQ4ODixYsIAtW7bQunVrxo8fz6uvvlqmjaenJ6+88gqdOnXi+uuv5+DBg3z33Xc4ODjg4+PDBx98wI033kibNm1YsWIF//vf//D19b3kZ//73//G39+fHj16cMcdd/Dwww/j6emJm5vb1bpcERGRK3boxGn+9uUv9H7jB77eehiL1eCmFg35+tFo5j/cheimvrU63AKYjIpuxFpNmM1mvL29ycnJwcvLq8y5goICUlJSCAsLU5i5gOeff54vv/ySX3/91d6lXDOHDx8mODiYFStW0Lt37wu20/ePiIjYQ+qJfN5ZtZ9vtv0+Y3tTi4aM69OcdsE+9i3uMl0sr10JLVGQMvLy8jh48CAzZ87khRdesHc5V9WqVavIy8vjuuuuIyMjg4kTJ9KkSRN69Ohh79JERERs0k6WBtuvt/4ebHs2b8i4Ps1oH6Ldf85HAVfKGDNmDJ999hm33377OcsTRo8ezX/+85/zvu+BBx5g9uzZ16LEclm7di0DBgy44Pm8vDyKi4v5+9//zoEDB/D09KRr167MmzdPuy+IiEiVkHYyn5mrkvh662HbzWM9fgu2HRRsL0pLFPRPzOWWlZWF2Ww+7zkvLy/8/PyucUUXdubMGY4cOXLB8xEREZfdt75/RETkako7mc+7q5P4asvvwbZ7swaM69OcjqE1K9hqiYLYnZ+fX5UKsRfj7u5+RSFWRETkWjt8qjTYfrn5z8G2GR1D69u5uuqlVgfcGjp5LVeZvm9ERKQylQbbZL7akkax5fdgO7Z3Mzo1UbC9HLUy4J5dY5mfn39V93iVmik/Px9Aa3VFROSKHMk+89uM7e/BtltEA8b2acb1CrZXpFYGXEdHR3x8fMjKygLAw8Oj1u8XJ5dmGAb5+flkZWXh4+ODo6OjvUsSEZFqKCOnNNh+vun3YHtjhC9jezfnhjAF28pQKwMuQEBAAIAt5IqUl4+Pj+37R0REpLyyzAXMWpPM/A2pFFmsQOmTx8b2bkbn8Es/oEjKr9YGXJPJRGBgIH5+fhQXF9u7HKkmnJ2dNXMrIiIVciKvkNk/JPPpz4coKC4NtjeE1WdC3+Z0UbC9KmptwD3L0dFRgUVEREQqXXZ+Ef/68QBz1h8kv8gCQPsQH/6vbwtujNDjdK+mWh9wRURERCpTzpliPlqXwsfrUsgrLAHgukbeTOjXnJuaN1SwvQYUcEVEREQqQV5hCXN+SuFfPx7AXFAabCMDPJnQtzl9o/wVbK8hBVwRERGRK5BfVMK/Ew7x/g/JnMovva+nmV9dxvdtTv9WATg4KNheaw4VaTx16lRMJlOZV2RkJAAnT57k8ccfp0WLFri7uxMSEsITTzxBTk5OmT5SU1OJjY3Fw8MDPz8/nnzySUpKSsq0WbNmDR06dMDV1ZWIiAjmzJlzZVcpIiIiUskKii18tC6FHq+s5uWleziVX0xYgzrMuK8dy8b14JbrAhVu7aTCM7itWrVixYoVv3fgVNpFeno66enpvPbaa0RFRXHo0CFGjx5Neno6X331FQAWi4XY2FgCAgJYv349GRkZDB06FGdnZ1566SUAUlJSiI2NZfTo0cybN4+VK1cycuRIAgMDiYmJqYxrFhEREblshSUWPt+UxrurkzhqLgQguL47Y3s35/Z2QTg5Vmj+UK4Ck1GB545OnTqVRYsWsX379nK1//LLL3nggQc4ffo0Tk5OLF26lIEDB5Keno6/vz8As2fPZtKkSRw7dgwXFxcmTZrEkiVL2LFjh62f++67j+zsbJYtW1buCzObzXh7e5OTk4OXl1e53yciIiJyPsUWK19tOczMVUkcyT4DQJC3G4/3bsbdHRvjrGBbYVcrr1X4T2L//v0EBQURHh7O4MGDSU1NvWDbs8WeneVNSEjguuuus4VbgJiYGMxmMzt37rS16dOnT5l+YmJiSEhIuGhdhYWFmM3mMi8RERGRyvBT0nH6v/Ujk79J5Ej2Gfw8XXnutlasfvIm7r8hROG2iqnQEoXOnTszZ84cWrRoQUZGBtOmTaN79+7s2LEDT0/PMm2PHz/O888/z6hRo2zHMjMzy4RbwPZ1ZmbmRduYzWbOnDmDu7v7eWubPn0606ZNq8jliIiIiFxURs4ZXli8myWJGQDUr+NC3M0RDO4cgpuz9tGvqioUcAcMGGD77zZt2tC5c2dCQ0P54osvGDFihO2c2WwmNjaWqKgopk6dWmnFXszkyZOZMGFCmRqCg4OvyWeLiIhIzVJUYuWjdSm8s2o/+UUWHEwwpEsoE/q2wNvD2d7lySVc0TZhPj4+NG/enKSkJNux3Nxc+vfvj6enJwsXLsTZ+fdvgoCAADZu3Fimj6NHj9rOnf3fs8f+2MbLy+uCs7cArq6uuLq6XsnliIiIiLB2/zGmfLuTA8dOA9AptB7TbmtFqyBvO1cm5XVFC0by8vJITk4mMDAQKJ017devHy4uLnz77be4ubmVaR8dHU1iYiJZWVm2Y/Hx8Xh5eREVFWVrs3LlyjLvi4+PJzo6+kpKFREREbmo9OwzPDZvC0M+2siBY6dpUNeF1//ali9HRyvcVjMV2kXhb3/7G7feeiuhoaGkp6czZcoUtm/fzq5du3B1daVfv37k5+ezcOFC6tSpY3tfw4YNcXR0xGKx0K5dO4KCgnjllVfIzMxkyJAhjBw5ssw2Ya1btyYuLo7hw4ezatUqnnjiCZYsWVKhbcK0i4KIiIiUR2GJhQ/XpjBzVRJnikuXIwyNbsL4vs3xdtdyhKvpauW1Ci1ROHz4MPfffz8nTpygYcOGdOvWjZ9//pmGDRuyZs0aNmzYAEBERESZ96WkpNCkSRMcHR1ZvHgxjz76KNHR0dSpU4dhw4bx3HPP2dqGhYWxZMkSxo8fz4wZM2jcuDEffvih9sAVERGRSvfDvmNM/XYnKcdLlyNc36Qez93WmpaBmhyrzio0g1udaAZXRERELuRI9hme/98ulu0s3cWpQV1Xno6N5PZ2jTCZ9PSxa6VKzOCKiIiIVGeFJRY++PEAM1cnUVBsxdHBxLDoJozr2wwvNy1HqCkUcEVERKRWWL03i2nf7uTgiXwAbgirz3O3tSIyQP/SW9Mo4IqIiEiNlnYyn+cW7yJ+V+k2pH6erjwd25K/tA3ScoQaSgFXREREaqTcgmLe/+EAH6w9QGFJ6XKEh7o2YWyfZnhqOUKNpoArIiIiNUpRiZX5Gw7x9qokTp4uAqBLeH2eu601zf097VydXAsKuCIiIlIjGIbBksQMXv1+L4d+W2cb3qAOE/tHEtPKX8sRahEFXBEREan2fj5wgulL9/BLWjZQuu3XuD7NuPf6YJwdr+jBrVINKeCKiIhItbXvaC7/XLqHlXuyAPBwcWRUj3Ae7h5OHVfFnNpKf/IiIiJS7WTmFPBm/D6+3JKG1QBHBxP33xDM2N7Naejpau/yxM4UcEVERKTaMBcU8/4PyXy0LoWCYisA/VsF8GT/FjRtWNfO1UlVoYArIiIiVV5RiZV5Gw7xzh92RugUWo/Jt0TSMbS+nauTqkYBV0RERKqsszsjvLJsL6knf9sZoWEdnuofSd8o7Ywg56eAKyIiIlVSQvIJXl66m18O5wDQ0NOV8X2ac0+nxjhpZwS5CAVcERERqVJ+PZzNWyv2s+q3nRHquDgyqkdTRnYP084IUi76LhEREZEqYcuhk7y9Mokf9h0DwMnBxKDOITzeq5l2RpAKUcAVERERuzEMg4QDJ3hnZRIJB04ApVt+3dYuiDE3RxCunRHkMijgioiIyDVnGAY/7j/OOyv3s/nQKQCcHU3c1aExj97UlFDfOnauUKozBVwRERG5ZgzDYMXuLGau2m+7eczFyYH7rg/mkZ5NaeTjbucKpSZQwBUREZGrzmo1WLojk5mrk9idYQbAzdmBwZ1DGdUjHH8vNztXKDWJAq6IiIhcNSUWK4t/zWDm6iSSsvKA0l0RhnZtwohuYTSoq5vHpPIp4IqIiEilK7ZYWbj1CLPWJHHwROkDGjzdnHjoxjCG39gEHw8XO1coNZkCroiIiFSawhILX24+zHtrkjmSfQaAeh7OjOwezpDoULzcnO1codQGCrgiIiJyxfIKS/h8Uxr/+jGZo+ZCABrUdWVUjzAGdw7VAxrkmtJ3m4iIiFy2w6fymfPTQT7flEZuYQkAAV5ujO4Zzn03hODm7GjnCqU2UsAVERGRCtty6BQfr0th6Y4MrEbpsfCGdRjZLZy7OjbC1UnBVuxHAVdERETKpcRiZdnOTD5al8K21Gzb8RsjfBnRLYybmvvh4GCyX4Eiv1HAFRERkYvKOVPM55tSmbv+kO3GMRdHB25rF8TwbmG0DPSyc4UiZSngioiIyHkdOnGaT346yJeb0zhdZAHAt44Lg7uEMqRLKA09tYetVE0KuCIiImJjGAabDp7iw7UHiN99FOO39bXN/esyolsYt7VrpBvHpMpTwBURERGKSqx8l5jBR+tSSDySYzves3lDRnQLo3uzBphMWl8r1YMCroiISC2WnV/E/I2pzF1/0LZ/rauTA3d2aMzwG5vQzN/TzhWKVJwCroiISC10PK+QD9em8GnCQdv62oaergztEsqgziH41tX6Wqm+FHBFRERqkcycAv714wHmbzxEQbEVgMgAT0Z2D+fWtoHav1ZqBAVcERGRWuDwqXzeW5PMl5sPU2QpDbZtG3vzeK9m9G7pp/W1UqMo4IqIiNRgKcdPM2t1Egu3HaHkt0eOXd+kHo/3aqYbx6TGUsAVERGpgfYfzWXm6iT+90u67VG63SIaMKZXBF3Cfe1bnMhVpoArIiJSg+xMz2HmqiSW7cy07WHbK9KPuJsj6Bhaz77FiVwjCrgiIiI1wLbUU8xclcTKPVm2Y/1bBTCmVwStG3nbsTKRa08BV0REpBrbcOAEM1cnsXb/cQAcTHBr2yDibo6gufawlVpKAVdERKSaMQyDdUnHeWdVEhtTTgLg5GDijvaNePSmpoQ3rGvnCkXsy6EijadOnYrJZCrzioyMtJ0vKCggLi4OX19f6taty1133cXRo0fL9JGamkpsbCweHh74+fnx5JNPUlJSUqbNmjVr6NChA66urkRERDBnzpzLv0IREZEaZH3Sce6encCQjzayMeUkLo4ODO4cwuq/3cSrf22rcCvCZczgtmrVihUrVvzegdPvXYwfP54lS5bw5Zdf4u3tzZgxY7jzzjv56aefALBYLMTGxhIQEMD69evJyMhg6NChODs789JLLwGQkpJCbGwso0ePZt68eaxcuZKRI0cSGBhITEzMlV6viIhItbT54EleX76PhAMngNLH6Q7qHMKoHuEEervbuTqRqsVkGGfvsby0qVOnsmjRIrZv337OuZycHBo2bMj8+fO5++67AdizZw8tW7YkISGBLl26sHTpUgYOHEh6ejr+/v4AzJ49m0mTJnHs2DFcXFyYNGkSS5YsYceOHba+77vvPrKzs1m2bFm5L8xsNuPt7U1OTg5eXl7lfp+IiEhV8ktaNm/E7+OHfccAcHEsDbaP3dQUPy83O1cncmWuVl6r0BIFgP379xMUFER4eDiDBw8mNTUVgC1btlBcXEyfPn1sbSMjIwkJCSEhIQGAhIQErrvuOlu4BYiJicFsNrNz505bmz/2cbbN2T4upLCwELPZXOYlIiJSXe1KNzNy7mZue/cnfth3DCcHE/ffEMLqJ29i6l9aKdyKXESFlih07tyZOXPm0KJFCzIyMpg2bRrdu3dnx44dZGZm4uLigo+PT5n3+Pv7k5mZCUBmZmaZcHv2/NlzF2tjNps5c+YM7u7n/2eY6dOnM23atIpcjoiISJWTlJXLmyv2s+TXDKB0V4Tb2zdibO9mhPrWsXN1ItVDhQLugAEDbP/dpk0bOnfuTGhoKF988cUFg+e1MnnyZCZMmGD72mw2ExwcbMeKREREyu/QidPMWLGfRduP2J48NrBNIOP6NCfCTzeOiVTEFW0T5uPjQ/PmzUlKSqJv374UFRWRnZ1dZhb36NGjBAQEABAQEMDGjRvL9HF2l4U/tvnzzgtHjx7Fy8vroiHa1dUVV1fXK7kcERGRa+7wqXxmrkriyy2HsfyWbPtF+TO+b3NaBuoeEpHLUeE1uH+Ul5dHcnIygYGBdOzYEWdnZ1auXGk7v3fvXlJTU4mOjgYgOjqaxMREsrJ+f8pKfHw8Xl5eREVF2dr8sY+zbc72ISIiUhMcNRfw7H93cPNra1iwKQ2L1eCmFg35dsyN/GtoJ4VbkStQoRncv/3tb9x6662EhoaSnp7OlClTcHR05P7778fb25sRI0YwYcIE6tevj5eXF48//jjR0dF06dIFgH79+hEVFcWQIUN45ZVXyMzM5JlnniEuLs42+zp69GhmzpzJxIkTGT58OKtWreKLL75gyZIllX/1IiIi19jxvEJmr0nm058PUVhiBaBrU1/+r19zOobWt3N1IjVDhQLu4cOHuf/++zlx4gQNGzakW7du/PzzzzRs2BCAN998EwcHB+666y4KCwuJiYlh1qxZtvc7OjqyePFiHn30UaKjo6lTpw7Dhg3jueees7UJCwtjyZIljB8/nhkzZtC4cWM+/PBD7YErIiLVWm5BMbN/SOaTnw6SX2QBoGNoPf6vb3O6RjSwc3UiNUuF9sGtTrQProiIVBVr9mbx928SSc8pAOC6Rt78X7/m9GzeEJPJZOfqROznauW1K7rJTERERC4sO7+I5xbv4putRwAIru/OM7FR9IvyV7AVuYoUcEVERK6CpYkZ/OO/OzmeV4jJBA91DeNvMc3xcNGvXpGrTX/LREREKlFWbgFT/ruTpTtKH2AU4VeXf97Vho6h9excmUjtoYArIiJSCQzD4OutR3h+8S5yzhTj6GDi0Z5NGdMrAjdnR3uXJ1KrKOCKiIhcoSPZZ/j7N4n8sO8YAK2CvHjl7ja0CvK2c2UitZMCroiIyGWyWg3mbTjEy0v3cLrIgouTA2N7N2NUj3CcHa/oWUoicgUUcEVERC5DyvHTTPr6VzamnARK97T9511tiPCra+fKREQBV0REpAJKLFY+WpfCG/H7KCyx4uHiyMSYFgyJboKjg7b+EqkKFHBFRETKaU+mmYlf/cqvh3MA6BbRgOl3XkdwfQ87VyYif6SAKyIicglFJVbeXZ3ErDVJFFsMPN2c+EdsFH/t1FgPbBCpghRwRURELuKXtGwmfvUre4/mAtA3yp8Xbm+Nv5ebnSsTkQtRwBURETmP9OwzvL58H99sO4xhgG8dF6b+pRUD2wRq1lakilPAFRER+YOc/GJm/ZDEJz8dpKjECsDt7YJ49tZW1K/jYufqRKQ8FHBFRESAgmILnyYcYubqJHLOFAPQOaw+k29pSbtgH/sWJyIVooArIiK1mtVq8N9fjvDa9/s4kn0GgOb+dXlqQCQ3t/DTcgSRakgBV0REaq21+48x/bs97MowAxDg5caEvs25q2Nj7WkrUo0p4IqISK2z40gO/1y2h7X7jwPg6erE6JuaMvzGMNxdHO1cnYhcKQVcERGpNQ6fyuf15ftYuO0IAM6OJoZ0acKYXhG6gUykBlHAFRGRGi87v4h3Vycxd/0hiiylOyP8pW0Qf+vXghBfPYVMpKZRwBURkRqroNjCnPUHmbU6CXNBCQBdm/oyeUBLrmvsbefqRORqUcAVEZEax2I1WLjtCG8s30t6TgEAkQGePDUgkp7NG2pnBJEaTgFXRERqjIJiC4u2HeGjdSnsz8oDIMjbjQn9WnBH+0baGUGkllDAFRGRai8rt4D/JBziPxtSOXm6CAAvNyfibo5gWNcmuDlrZwSR2kQBV0REqq2d6Tl8tC6F//2STrHFAKCRjzsPdm3CPdcH4+3ubOcKRcQeFHBFRKRasVgNVu4+ykfrUtiQctJ2vGNoPUZ0C6NflD9Ojg52rFBE7E0BV0REqoXThSV8uTmNT9Yf5NCJfACcHEzccl0gw7uF0S7Yx74FikiVoYArIiJV2uFT+cxdf5AFm9LI/W2rL293Z+6/IYRhXUMJ9Ha3c4UiUtUo4IqISJVjGAZbU0/x8bqDLNuZicVaur42vEEdHuoWxl0dGuHhol9hInJ++ukgIiJVRrHFytIdmXy0LoVf0rJtx7tFNGB4tybc1NwPB231JSKXoIArIiJ2l3Yyn4XbjvDZxlQyfnswg4uTA7e3C2J4tzAiA7zsXKGIVCcKuCIiYhfZ+UUsScxg4dYjbD50yna8QV0XhnRpwuAuITSo62rHCkWkulLAFRGRa6awxMLqPVks3HaE1XuOUWSxAuBgghsjGnBH+0bEtgnE1UkPZhCRy6eAKyIiV5XVarAl9RTfbD3Ckl/TMf+2EwJAy0Av7mzfiL+0C8Lfy82OVYpITaKAKyIiV0XysTwWbTvCwm1HOHzqjO14gJcbt7UP4o72jbS2VkSuCgVcERGpNMfzCvnfL+ks2naEXw7n2I7XdXWif+sA7mzfiM7hvjhqJwQRuYoUcEVE5IqcKbIQv/soC7ce5sf9x2171jo6mOjZvCF3tG9En5b+uLtoXa2IXBsKuCIiclmO5xXy/g/JfLYxjbzC39fVtm3szR3tGzGwbZB2QRARu1DAFRGRCjl5uoh//XiAuesPcqbYAkDjeu7c0b4Rt7dvRNOGde1coYjUdgq4IiJSLtn5RXy4NoVPfkrhdFFpsG3b2JtxfZtzU/OGmExaVysiVYPDlbz55ZdfxmQyMW7cONuxzMxMhgwZQkBAAHXq1KFDhw58/fXXZd538uRJBg8ejJeXFz4+PowYMYK8vLwybX799Ve6d++Om5sbwcHBvPLKK1dSqoiIXKacM8W8Eb+P7v9czczVSZwustAqyIuPhnViUdyN3NzCT+FWRKqUy57B3bRpE++//z5t2rQpc3zo0KFkZ2fz7bff0qBBA+bPn88999zD5s2bad++PQCDBw8mIyOD+Ph4iouLeeihhxg1ahTz588HwGw2069fP/r06cPs2bNJTExk+PDh+Pj4MGrUqCu4XBERKa/cgmI++ekgH6w9QO5ve9dGBngyvm9z+kX5K9SKSJVlMgzDqOib8vLy6NChA7NmzeKFF16gXbt2vPXWWwDUrVuX9957jyFDhtja+/r68s9//pORI0eye/duoqKi2LRpE506dQJg2bJl3HLLLRw+fJigoCDee+89nn76aTIzM3FxcQHgqaeeYtGiRezZs6dcNZrNZry9vcnJycHLS/ssioiUV15hCXPXlwbb7PxiAJr712V8n+bEtArAQVt8iUgluVp57bKWKMTFxREbG0ufPn3OOde1a1c+//xzTp48idVqZcGCBRQUFHDTTTcBkJCQgI+Pjy3cAvTp0wcHBwc2bNhga9OjRw9buAWIiYlh7969nDp1ivMpLCzEbDaXeYmISPnlF5Uw+4dkeryymle/30t2fjFNG9bhnfvbs2xsDwZcF6hwKyLVQoWXKCxYsICtW7eyadOm857/4osvuPfee/H19cXJyQkPDw8WLlxIREQEULpG18/Pr2wRTk7Ur1+fzMxMW5uwsLAybfz9/W3n6tWrd87nTp8+nWnTplX0ckREar0zRRbmbTjE7B+SOZ5XBEBYgzqM7d2MW9sG6aEMIlLtVCjgpqWlMXbsWOLj43FzO/8zw//xj3+QnZ3NihUraNCgAYsWLeKee+5h7dq1XHfddZVS9PlMnjyZCRMm2L42m80EBwdftc8TEanuCootfLYxlVlrkjmWWwhASH0PnujdjNvbBeHkeEX3IYuI2E2FAu6WLVvIysqiQ4cOtmMWi4Uff/yRmTNnsnfvXmbOnMmOHTto1aoVAG3btmXt2rW8++67zJ49m4CAALKyssr0W1JSwsmTJwkICAAgICCAo0ePlmlz9uuzbf7M1dUVV1dtKC4icikFxRa+2JzGu6uTOGouDbaN67nzRK9m3NGhEc4KtiJSzVUo4Pbu3ZvExMQyxx566CEiIyOZNGkS+fn5ADg4lP3h6OjoiNVqBSA6Oprs7Gy2bNlCx44dAVi1ahVWq5XOnTvb2jz99NMUFxfj7OwMQHx8PC1atDjv8gQREbm0Y7mFfPrzIf7z8yFOni5dihDk7caYXs24u2NjXJwUbEWkZqhQwPX09KR169ZljtWpUwdfX19at25NcXExERERPPLII7z22mv4+vqyaNEi4uPjWbx4MQAtW7akf//+PPzww8yePZvi4mLGjBnDfffdR1BQEACDBg1i2rRpjBgxgkmTJrFjxw5mzJjBm2++WUmXLSJSe+w7mstHa1NYuP0IRSWlkw2NfNx5pGc4914fjKuTo50rFBGpXJX6JDNnZ2e+++47nnrqKW699Vby8vKIiIhg7ty53HLLLbZ28+bNY8yYMfTu3RsHBwfuuusu3n77bdt5b29vli9fTlxcHB07dqRBgwY8++yz2gNXRKScDMNgXdJxPlybwg/7jtmOtwv24eHu4cS08tcaWxGpsS5rH9zqQPvgikhtVFhi4dvt6Xy0LoU9mbkAmEwQExXAwz3C6BBSTw9oEJEq42rltUqdwRUREfs4dbqIeRsOMTfhkG1HBA8XR+7pFMxDNzYh1LeOnSsUEbl2FHBFRKqxA8fy+PinFL7acpiC4tL1tQFebjx4YxPuvz4Ebw9nO1coInLtKeCKiFQzhmGwIeUkH65NYeWeo5xdaNYqyIuHu4dzy3WB2hFBRGo1BVwRkWqi2GLlu8QMPlybQuKRHNvxPi39GNEtnC7h9bW+VkQEBVwRkSovPfsMn29K4/NNaWSaCwBwdXLg7o6NGd4tjKYN69q5QhGRqkUBV0SkCrJYDdbszWL+hlRW783C+tsyhAZ1XRkWHcrgLqHUr+Ni3yJFRKooBVwRkSokI+f32dqMnALb8S7h9RnUOZSYVv56MIOIyCUo4IqI2JnFavDDvtLZ2lV7fp+trefhzN0dG3PfDSFahiAiUgEKuCIidpKZU/DbbG0q6X+Yre0cVp9BnUOIaRWAm7Nma0VEKkoBV0TkGrJYDX7cd4z5G0tnay2/Tdf6eDhzV4fG3H9DCBF+mq0VEbkSCrgiItfAUXMBX2xKY8GmNI5kn7EdvyGsPoNuCKF/a83WiohUFgVcEZGraH3yceb8dJCVf5it9XYvna0d1DmYCD9PO1coIlLzKOCKiFwFu9LNTF+6m7X7j9uOXd+kHoM6hzCgdaBma0VEriIFXBGRSnQk+wyvL9/Lwm1HMAxwdjRx3/UhDIkOpbm/ZmtFRK4FBVwRkUqQc6aYWWuS+OSngxSVWAEY2CaQJ2NaEOpbx87ViYjULgq4IiJXoLDEwqcJh5i5Oons/GKg9Maxv9/SknbBPvYtTkSkllLAFRG5DFarwf9+TefV7/dy+FTprggRfnWZPCCSXpF+mEwmO1coIlJ7KeCKiFTQ+uTjTP9uD4lHcgDw83RlQt/m3N2xMU6ODnauTkREFHBFRMppb2Yu/1y2h1V7sgCo4+LI6J5NGdE9DA8X/TgVEakq9BNZROQSMnMKeCN+L19tOYzVACcHE4M6h/BE72Y0qOtq7/JERORPFHBFRC7AXFDM+z8k89G6FAqKS3dGGNA6gCdjWhDeUI/TFRGpqhRwRUT+pKjEyvwNh3h7VRInTxcB0Cm0HpNvaUnH0Hp2rk5ERC5FAVdE5A9W783i+f/t4sDx0wCEN6jDpAGR9Ivy184IIiLVhAKuiAiQcvw0zy/eZbuBrEFdF8b1ac691wfjrJ0RRESqFQVcEanV8gpLmLkqiY/WHaDYYuDkYOLBrk14ok8zvNyc7V2eiIhcBgVcEamVrFaDRduP8PLSPWTlFgLQo3lDnh0YRYSfbiATEanOFHBFpNb59XA2U7/dydbUbABCfT34R2wUvVvqCWQiIjWBAq6I1BrH8wp5ddlevtiShmGAh4sjY3pFMKJbGK5OjvYuT0REKokCrojUeMUWK3PXH2TGiv3kFpYAcEf7RkzqH0mAt5udqxMRkcqmgCsiNdqP+44x7X87ST5Wuu1X60ZeTL21FZ2a1LdzZSIicrUo4IpIjXToxGmeX7ybFbuPAuBbx4UnY1rw107BODpona2ISE2mgCsiNcrpwhJmrUnigx9TKLJYcXQwMSy6CWP7NMPbXdt+iYjUBgq4IlIjWK0G//s1nZe+281Rc+m2X90iGjDl1iia+XvauToREbmWFHBFpFo7llvIV1sOs2BTKodO5AMQXN+dZ2Kj9HhdEZFaSgFXRKodq9VgffIJPtuYyvJdmRRbDAA8XZ0Y1SOch3uE4+asbb9ERGorBVwRqTbON1sL0C7Yh0E3hDCwbSAeLvqxJiJS2+k3gYhUaWdna+dvPMTynUcpsf4+W3t7+0bcf0MIUUFedq5SRESqEgVcEamSjuUW8uWWNBZsTCP15J9mazuHMLCNZmtFROT89NtBRKoMq9Xgp+TjpWtr/zRbe0eHRtx3vWZrRUTk0hRwRcTusnILStfW/mm2tn2ID/ffoNlaERGpGIcrefPLL7+MyWRi3LhxZY4nJCTQq1cv6tSpg5eXFz169ODMmTO28ydPnmTw4MF4eXnh4+PDiBEjyMvLK9PHr7/+Svfu3XFzcyM4OJhXXnnlSkoVkSpo39Fc4uZtpev0VbyybC+pJ/PxdHViaHQo3z3RnYWP3cg9nYIVbkVEpEIu+7fGpk2beP/992nTpk2Z4wkJCfTv35/Jkyfzzjvv4OTkxC+//IKDw+9ZevDgwWRkZBAfH09xcTEPPfQQo0aNYv78+QCYzWb69etHnz59mD17NomJiQwfPhwfHx9GjRp1uSWLSBVxLLeQN1fsY8HGVH5bhaDZWhERqTQmwzCMir4pLy+PDh06MGvWLF544QXatWvHW2+9BUCXLl3o27cvzz///Hnfu3v3bqKioti0aROdOnUCYNmyZdxyyy0cPnyYoKAg3nvvPZ5++mkyMzNxcXEB4KmnnmLRokXs2bOnXDWazWa8vb3JycnBy0tr9kSqgoJiCx+tS2HW6iROF1kAiGnlz9jezbW2VkSkFrpaee2ylijExcURGxtLnz59yhzPyspiw4YN+Pn50bVrV/z9/enZsyfr1q2ztUlISMDHx8cWbgH69OmDg4MDGzZssLXp0aOHLdwCxMTEsHfvXk6dOnXemgoLCzGbzWVeIlI1WK0GC7cdptdra3j1+72cLrLQprE3n4/qwvtDOinciohIparwvwMuWLCArVu3smnTpnPOHThwAICpU6fy2muv0a5dO/7973/Tu3dvduzYQbNmzcjMzMTPz69sEU5O1K9fn8zMTAAyMzMJCwsr08bf3992rl69eud89vTp05k2bVpFL0dErrKfD5zgxSW7STySA0CQtxsT+0fyl7ZBODjoMboiIlL5KhRw09LSGDt2LPHx8bi5uZ1z3mq1AvDII4/w0EMPAdC+fXtWrlzJxx9/zPTp0yuh5PObPHkyEyZMsH1tNpsJDg6+ap8nIhd34FgeLy/dw/JdRwGo6+rEYzc3ZfiNYXqMroiIXFUVCrhbtmwhKyuLDh062I5ZLBZ+/PFHZs6cyd69ewGIiooq876WLVuSmpoKQEBAAFlZWWXOl5SUcPLkSQICAmxtjh49WqbN2a/PtvkzV1dXXF1dK3I5InIVnDpdxIyV+/nPz4cosRo4Opi4/4ZgxvVpToO6+jsqIiJXX4UCbu/evUlMTCxz7KGHHiIyMpJJkyYRHh5OUFCQLeietW/fPgYMGABAdHQ02dnZbNmyhY4dOwKwatUqrFYrnTt3trV5+umnKS4uxtnZGYD4+HhatGhx3uUJImJ/hSUW5q4/yDurksgtKAGgV6QfkwdE0szf087ViYhIbVKhgOvp6Unr1q3LHKtTpw6+vr62408++SRTpkyhbdu2tGvXjrlz57Jnzx6++uoroHQ2t3///jz88MPMnj2b4uJixowZw3333UdQUBAAgwYNYtq0aYwYMYJJkyaxY8cOZsyYwZtvvlkZ1ywilcgwDJYkZvDPZXtIO1m633XLQC+eiW3JjREN7FydiIjURpW+2eS4ceMoKChg/PjxnDx5krZt2xIfH0/Tpk1tbebNm8eYMWPo3bs3Dg4O3HXXXbz99tu2897e3ixfvpy4uDg6duxIgwYNePbZZ7UHrkgVs+XQKV5csoutqdkA+Hm68reYFtzVoTGOuoFMRETs5LL2wa0OtA+uyNWTdjKfl5ftYcmvGQC4OzvySM9wRvUI10MaRESk3K5WXtNvIhEptwPH8nhvTTILtx2hxGpgMsFfOzbm//q1wN/r3J1VRERE7EEBV0QuaXeGmXdXJ/FdYobt0brdmzVg8oCWekiDiIhUOQq4InJB21JP8e7qJFbs/n1rv96RfsT1iqBDiHY0ERGRqkkBV0TKMAyDhAMneHd1Ej8lnQDAZIJbrgsk7qYIzdiKiEiVp4ArIkBpsF29N4uZq5JsuyI4OZi4o30jRt/UlKYN69q3QBERkXJSwBWp5SxWg6U7Mnh3dTK7M8wAuDg5cN/1wYzqEU7jeh52rlBERKRiFHBFaqlii5VF247w3g/JHDh2GoA6Lo480CWUEd3D8PPUrggiIlI9KeCK1DIFxRa+3JzG7B8OcCS79Mlj3u7OPNi1CQ/d2AQfDxc7VygiInJlFHBFaonThSXM23CID9amcCy3EIAGdV0Y2T2cB7qEUtdVPw5ERKRm0G80kRru0InT/DvhEF9sTiO3oASAIG83HunZlHuvD8bN2dHOFYqIiFQuBVyRGshqNVibdJy56w+yem8WZx/IHdagDo/2bMrt7Rvh4uRg3yJFRESuEgVckRokt6CYr7cc5t8Jhzhw/LTteM/mDXnwxib0bNYQBweTHSsUERG5+hRwRWqA5GN5/Hv9Qb7eeoS8wtJlCHVdnbi7Y2OGRocSrj1sRUSkFlHAFammrNbSBzPMWX+QtfuP2443bViHYV2bcGeHxrpxTEREaiX99hOpZnLOFPPl5jQ+/fkQh07kA6WP0u0d6cewrk3oFtEAk0nLEEREpPZSwBWpJvYdzWXu+oN8s/UIZ4otAHi5OXHv9cEM6dKEEF89cUxERAQUcEWqNIvVYMXuo8xdf5D1ySdsx5v712VY1ybc0b4RHi76aywiIvJH+s0oUgVZrQbf7cjgzfh9JP/2GF0HE/SN8mdY1yZEh/tqGYKIiMgFKOCKVCGGYbBydxavx+9jd4YZKH2M7v03hPBAlxAa19MyBBERkUtRwBWpAgzDYF3ScV5bvo9f0rKB0m2+RnYPY3i3MLzcnO1boIiISDWigCtiZ5sOnuTV7/eyMeUkAG7ODjzYNYxHeoRTr46LnasTERGpfhRwRezkl7RsXo/fx4/7jgHg4ujA4C4hPHpTU/w83excnYiISPWlgCtyje3JNPP68n3E7zoKgJODiXuuD2bMzREE+bjbuToREZHqTwFX5BpJPpbHWyv2s/jXdAyjdFeE29s3Ylzv5trDVkREpBIp4IpcZWkn83l75X6+3noYq1F6LLZNIOP7NCPCz9O+xYmIiNRACrgiV0lmTgEzV+/n801pFFtKk22fln5M6NuCqCAvO1cnIiJScyngilSyE3mFzFqTzKc/H6KoxApA92YNmNC3Oe1D6tm5OhERkZpPAVekkpwpsvDxTym8tyaZvMISAK5vUo//69eCLuG+dq5ORESk9lDAFblCJRYrX205zJsr9nHUXAhAqyAvJvaPpEezBnqkroiIyDWmgCtymc4+Vvefy/awPysPgMb13HkypgW3tgnCwUHBVkRExB4UcEUuw7bUU0z/bg8bD5Y+fczHw5kxN0cwJDoUVydHO1cnIiJSuyngilRAyvHTvPr9Hr5LzATA1cmB4d3CGN2zKd7uznauTkREREABV6RcjucV8vbK/czfkEqJ1cBkgrs7NGZCv+YEeuvpYyIiIlWJAq7IRZwuLOHDtSn868dkThdZALi5RUMmDYgkMkB72YqIiFRFCrgi51FisfL55jTeWrGfY7mlOyO0bezNUwNaEt1UW36JiIhUZQq4In9gGAbLdx3llWV7SD52GoCQ+h5M7N+C2OsCteWXiIhINaCAK/KbLYdOMv27PWw+dAqA+nVceKJXBIM6h+Li5GDn6kRERKS8FHCl1tt88CRvr0rix33HAHBzdmBkt3Ae6RmOp5t2RhAREaluFHCl1vr5wAneXrmf9cknAHB0MPHXjo0Z37c5/l5udq5ORERELtcV/bvryy+/jMlkYty4ceecMwyDAQMGYDKZWLRoUZlzqampxMbG4uHhgZ+fH08++SQlJSVl2qxZs4YOHTrg6upKREQEc+bMuZJSRYDS78t1+49zz+wE7vvXz6xPPoGzo4n7bwhhzd9u4uW72ijcioiIVHOXPYO7adMm3n//fdq0aXPe82+99dZ5b8ixWCzExsYSEBDA+vXrycjIYOjQoTg7O/PSSy8BkJKSQmxsLKNHj2bevHmsXLmSkSNHEhgYSExMzOWWLLWYYRis2XeMt1fuZ1tqNgAujg7cd0Mwj/RsSiMf7WUrIiJSU1xWwM3Ly2Pw4MF88MEHvPDCC+ec3759O6+//jqbN28mMDCwzLnly5eza9cuVqxYgb+/P+3ateP5559n0qRJTJ06FRcXF2bPnk1YWBivv/46AC1btmTdunW8+eabCrhSIYZhsGJ3Fu+s2s+vh3OA0qePDeocwiM9mhLgrdlaERGRmuaylijExcURGxtLnz59zjmXn5/PoEGDePfddwkICDjnfEJCAtdddx3+/v62YzExMZjNZnbu3Glr8+e+Y2JiSEhIuGBNhYWFmM3mMi+pvaxWg6WJGcS+vY6H/72ZXw/n4O7syKge4ayddDNTbm2lcCsiIlJDVXgGd8GCBWzdupVNmzad9/z48ePp2rUrt91223nPZ2Zmlgm3gO3rzMzMi7Yxm82cOXMGd/dz/zl5+vTpTJs2raKXIzWMxWqwJDGDmav2s+9oHgB1XBwZ2rUJI7uF4VvX1c4VioiIyNVWoYCblpbG2LFjiY+Px83t3Nmvb7/9llWrVrFt27ZKK7C8Jk+ezIQJE2xfm81mgoODr3kdYh8lFiv/+zWdd1YlceC3BzR4ujnxUNcmDO8Who+Hi50rFBERkWulQgF3y5YtZGVl0aFDB9sxi8XCjz/+yMyZM3n00UdJTk7Gx8enzPvuuusuunfvzpo1awgICGDjxo1lzh89ehTAtqQhICDAduyPbby8vM47ewvg6uqKq6tm52qbYouVhduO8O7qJA6dyAfA292ZEd3CGNa1Cd7u2sdWRESktqlQwO3duzeJiYlljj300ENERkYyadIkGjRowCOPPFLm/HXXXcebb77JrbfeCkB0dDQvvvgiWVlZ+Pn5ARAfH4+XlxdRUVG2Nt99912ZfuLj44mOjq7Y1UmNVWKx8s22I7yzaj9pJ88ApU8eG9k9jCFdQvWABhERkVqsQgHX09OT1q1blzlWp04dfH19bcfPd2NZSEgIYWFhAPTr14+oqCiGDBnCK6+8QmZmJs888wxxcXG2GdjRo0czc+ZMJk6cyPDhw1m1ahVffPEFS5YsuayLlJrDYjX47/YjvL1yPwd/m7FtUNeFUT3CGdw5lDquenaJiIhIbXfN04CjoyOLFy/m0UcfJTo6mjp16jBs2DCee+45W5uwsDCWLFnC+PHjmTFjBo0bN+bDDz/UFmG1mMVqsPjXdGas3G9bY+tbx4XRPZvyQJdQ3F0c7VyhiIiIVBUmwzAMexdxNZjNZry9vcnJycHLy8ve5chlsloNlu3M5K0V+2y7Ivh4OPNIj6YMjdaMrYiISHV2tfKa0oFUSYZhsHzXUd6M38eezFwAvNyceLh7OA/e2ERrbEVEROSCFHClSjEMg1V7snhzxT52HCl9WIenqxPDu4UxvFuYdkUQERGRS1LAlSrBMAx+3H+cN+L38UtaNlD6gIaHbgxjZHftYysiIiLlp4ArdmUYBuuTT/BG/D62HDoFgLuzI0O7hvJIj6bUr6NgKyIiIhWjgCt28/OB0mC7MeUkAK5ODgzpEsojPZvS0FMP7RAREZHLo4Ar19y21FO8tnwvPyWdAMDF0YFBnUN47Kam+Hmd+whoERERkYpQwJVr5lhuIf9ctoevthwGwNnRxL3XBxN3cwSB3ud/BLOIiIhIRSngylVXbLHy74RDvBW/j9zCEgDu6tCY8X2b0bieh52rExERkZpGAVeuqvVJx5n6v522hzRc18ibabe1okNIPTtXJiIiIjWVAq5cFUeyz/DSkt0sScwAoJ6HMxP7R3JPp2AcHUx2rk5ERERqMgVcqVQFxRY+XHuAd1cnc6bYgoMJHugSyoS+zbWXrYiIiFwTCrhSaVbuPspzi3dx6EQ+ANc3qce0v7QmKqjyni0tIiIicikKuHLFDh4/zXOLd7FqTxYAfp6uPB3bkr+0DcJk0nIEERERubYUcOWy5ReV8O7qJD74MYUiixVnRxPDu4XxeK9m1HXVt5aIiIjYh1KIVJhhGCxJzODFJbvJyCkAoHuzBkz9SyuaNqxr5+pERESktlPAlQrZm5nL1G93knCg9Clkjeu584+BUfSL8tdyBBEREakSFHClXHLOFDNjxX7mJhzEYjVwdXLgsZsieKRnOG7OjvYuT0RERMRGAVcu6kyRhbkJB3lvTTI5Z4oBiGnlzzOxUQTX11PIREREpOpRwJXzKrZY+XLzYWas3MdRcyEAzf3r8kxsFD2aN7RzdSIiIiIXpoArZVitpTeQvb58Lwd/28+2kY87E/o25/b2jfQUMhEREanyFHAFKN0Z4cf9x3ll2R52ppsB8K3jwuO9Iri/cwiuTlpnKyIiItWDAq6wNfUUryzbw88HTgJQ19WJUT3CGd4tTPvZioiISLWj9FKL7Tuay2vf72X5rqMAuDg5MLRLKI/dHEH9Oi52rk5ERETk8ijg1kKHT+XzZvx+Fm47jNUABxPc3bExY/s0p5GPu73LExEREbkiCri1yPG8Qt5dncS8n1MpslgB6N8qgL/FNCfCz9PO1YmIiIhUDgXcWiC3oJgP16bw4doDnC6yANC1qS8T+0fSLtjHvsWJiIiIVDIF3Bosv6iE+RtSmbUmmZOniwC4rpE3k/pH0q1ZAztXJyIiInJ1KODWQIdOnObfCYf4YnMauQUlAIQ3qMPfYlowoHUAJpP2shUREZGaSwG3hrBaDdYmHWfu+oOs3puFYZQeD/X1YHTPpvy1Y2OcHB3sW6SIiIjINaCAW83lFhTz9ZbD/DvhEAeOn7Yd79m8IQ92bULP5g1x0NPHREREpBZRwK2mkrLy+HfCQb7ecth241hdVyfu7tiYodGhhDesa+cKRUREROxDAbcasVgNVu/JYm7CQdbuP247HuFXl2HRodzRobGePCYiIiK1ntJQNZCTX8wXm9P4988HSTt5BgCTCXpH+vNg1ybcGOGrG8dEREREfqOAW4XtyTQzd/1BFm47QkFx6YMZvN2duff6YIZ0CSW4voedKxQRERGpehRwq5iiEisrdh9l7vqDbEg5aTseGeDJg12bcFu7Rri7ONqxQhEREZGqTQG3CjAMg53pZr7acpj/bj/CqfxiABwdTMS08mdYdBNuCKuvZQgiIiIi5aCAa0fH8wpZtO0IX205zJ7MXNtxP09X/tqpMYM7hxLk427HCkVERESqHwXca6yoxMqqPVl8teUwa/ZmUWItfSKDi5MD/aL8ubtjY7pFNNBDGUREREQukwLuNXChJQgA7YJ9uLtjY25tE4S3h7MdqxQRERGpGRRwr6JjuYX8d/u5SxD8vVy5o31j7u7YiAg/TztWKCIiIlLzXNG/g7/88suYTCbGjRsHwMmTJ3n88cdp0aIF7u7uhISE8MQTT5CTk1PmfampqcTGxuLh4YGfnx9PPvkkJSUlZdqsWbOGDh064OrqSkREBHPmzLmSUq+ZohIry3ZkMHLuJrpMX8kLS3azJzMXFycHBrYJZM5D17P+qd48NSBS4VZERETkKrjsGdxNmzbx/vvv06ZNG9ux9PR00tPTee2114iKiuLQoUOMHj2a9PR0vvrqKwAsFguxsbEEBASwfv16MjIyGDp0KM7Ozrz00ksApKSkEBsby+jRo5k3bx4rV65k5MiRBAYGEhMTc4WXXPm0BEFERESk6jAZhmFU9E15eXl06NCBWbNm8cILL9CuXTveeuut87b98ssveeCBBzh9+jROTk4sXbqUgQMHkp6ejr+/PwCzZ89m0qRJHDt2DBcXFyZNmsSSJUvYsWOHrZ/77ruP7Oxsli1bVq4azWYz3t7e5OTk4OXlVdFLrJADx/Lo9foPtq+1BEFERETk0q5WXrusJQpxcXHExsbSp0+fS7Y9W7CTU+lkcUJCAtddd50t3ALExMRgNpvZuXOnrc2f+46JiSEhIeGCn1NYWIjZbC7zulbCG9bl+ib1uLVtEHOH36AlCCIiIiJ2VOElCgsWLGDr1q1s2rTpkm2PHz/O888/z6hRo2zHMjMzy4RbwPZ1ZmbmRduYzWbOnDmDu/u5e8NOnz6dadOmVfRyKs3no6JxcNCDGERERETsrUIzuGlpaYwdO5Z58+bh5uZ20bZms5nY2FiioqKYOnXqldRYLpMnTyYnJ8f2SktLu+qf+UcKtyIiIiJVQ4VmcLds2UJWVhYdOnSwHbNYLPz444/MnDmTwsJCHB0dyc3NpX///nh6erJw4UKcnX+/uSogIICNGzeW6ffo0aO2c2f/9+yxP7bx8vI67+wtgKurK66urhW5HBERERGpgSo0g9u7d28SExPZvn277dWpUycGDx7M9u3bcXR0xGw2069fP1xcXPj222/PmemNjo4mMTGRrKws27H4+Hi8vLyIioqytVm5cmWZ98XHxxMdHX251ykiIiIitUSFZnA9PT1p3bp1mWN16tTB19eX1q1b28Jtfn4+//nPf8rc7NWwYUMcHR3p168fUVFRDBkyhFdeeYXMzEyeeeYZ4uLibDOwo0ePZubMmUycOJHhw4ezatUqvvjiC5YsWVJJly0iIiIiNVWlPsls69atbNiwAYCIiIgy51JSUmjSpAmOjo4sXryYRx99lOjoaOrUqcOwYcN47rnnbG3DwsJYsmQJ48ePZ8aMGTRu3JgPP/ywSu6BKyIiIiJVy2Xtg1sdXMt9cEVERESk4qrUPrgiIiIiIlWVAq6IiIiI1CgKuCIiIiJSoyjgioiIiEiNooArIiIiIjVKpW4TVpWc3Rzi7D68IiIiIlK1nM1plb2pV40NuLm5uQAEBwfbuRIRERERuZjc3Fy8vb0rrb8auw+u1WolPT0dT09PTCbTVf88s9lMcHAwaWlp2nf3GtK424fG3T407vahcbcPjbt9XOtxNwyD3NxcgoKCcHCovJWzNXYG18HBgcaNG1/zz/Xy8tJfRDvQuNuHxt0+NO72oXG3D427fVzLca/MmduzdJOZiIiIiNQoCrgiIiIiUqMo4FYSV1dXpkyZgqurq71LqVU07vahcbcPjbt9aNztQ+NuHzVl3GvsTWYiIiIiUjtpBldEREREahQFXBERERGpURRwRURERKRGUcAVERERkRql2gXc6dOnc/311+Pp6Ymfnx+33347e/fuLdOmoKCAuLg4fH19qVu3LnfddRdHjx61nf/ll1+4//77CQ4Oxt3dnZYtWzJjxowLfuZPP/2Ek5MT7dq1u2R9hmHw7LPPEhgYiLu7O3369GH//v1l2vzlL38hJCQENzc3AgMDGTJkCOnp6RftNyMjg0GDBtG8eXMcHBwYN27cOW2Ki4t57rnnaNq0KW5ubrRt25Zly5ZdsubyqK3j/s0339C3b18aNmyIl5cX0dHRfP/99xds//LLL2Mymc7753M5NO4XHvfc3FzGjRtHaGgo7u7udO3alU2bNl2y5vKoCeN+VmFhIe3atcNkMrF9+/ZL9r1mzRo6dOiAq6srERERzJkzp8z59957jzZt2tg2gY+Ojmbp0qWX7Lc8NO4XHneLxcI//vEPwsLCcHd3p2nTpjz//PNUxn3itXXcy/N79ZtvvqFTp074+PhQp04d2rVrx6effnrJmstD437hcQd46623aNGiBe7u7gQHBzN+/HgKCgouWfcfL6BaiYmJMT755BNjx44dxvbt241bbrnFCAkJMfLy8mxtRo8ebQQHBxsrV640Nm/ebHTp0sXo2rWr7fxHH31kPPHEE8aaNWuM5ORk49NPPzXc3d2Nd95555zPO3XqlBEeHm7069fPaNu27SXre/nllw1vb29j0aJFxi+//GL85S9/McLCwowzZ87Y2rzxxhtGQkKCcfDgQeOnn34yoqOjjejo6Iv2m5KSYjzxxBPG3LlzjXbt2hljx449p83EiRONoKAgY8mSJUZycrIxa9Ysw83Nzdi6desl676U2jruY8eONf75z38aGzduNPbt22dMnjzZcHZ2Pu+Ybty40WjSpInRpk2b8/75XA6N+4XH/Z577jGioqKMH374wdi/f78xZcoUw8vLyzh8+PAl676UmjDuZz3xxBPGgAEDDMDYtm3bRfs9cOCA4eHhYUyYMMHYtWuX8c477xiOjo7GsmXLbG2+/fZbY8mSJca+ffuMvXv3Gn//+98NZ2dnY8eOHZes+1I07hce9xdffNHw9fU1Fi9ebKSkpBhffvmlUbduXWPGjBmXrPtSauu4l+f36urVq41vvvnG2LVrl5GUlGS89dZb5/zZXC6N+4XHfd68eYarq6sxb948IyUlxfj++++NwMBAY/z48Zes+6xqF3D/LCsrywCMH374wTAMw8jOzjacnZ2NL7/80tZm9+7dBmAkJCRcsJ/HHnvMuPnmm885fu+99xrPPPOMMWXKlEt+Q1itViMgIMB49dVXbceys7MNV1dX47PPPrvg+/773/8aJpPJKCoqumj/Z/Xs2fO83xCBgYHGzJkzyxy78847jcGDB5er34qojeN+VlRUlDFt2rQyx3Jzc41mzZoZ8fHxF/zzqQwa99Jxz8/PNxwdHY3FixeXadOhQwfj6aefrlC/5VFdx/27774zIiMjjZ07d5brF8/EiRONVq1anVNbTEzMRd9Xr14948MPP7xom8uhcf993GNjY43hw4eXaaOf71c27n9UkZ/b7du3N5555ply911eGvffxcXFGb169SpzbMKECcaNN95Y7r6r3RKFP8vJyQGgfv36AGzZsoXi4mL69OljaxMZGUlISAgJCQkX7edsH2d98sknHDhwgClTppSrlpSUFDIzM8t8tre3N507d77gZ588eZJ58+bRtWtXnJ2dy/U5F1JYWIibm1uZY+7u7qxbt+6K+j2f2jruVquV3Nzcc2qOi4sjNja2TA1Xg8a9tOaSkhIsFou+339zvnE/evQoDz/8MJ9++ikeHh7l6jshIeGc7+GYmJgLXpPFYmHBggWcPn2a6Ojocn1GRWjcf++3a9eurFy5kn379gGl/zS9bt06BgwYUK7PqIjaMu4VZRgGK1euZO/evfTo0aPS+9e4/65r165s2bKFjRs3AnDgwAG+++47brnllnL34VRp1diB1Wpl3Lhx3HjjjbRu3RqAzMxMXFxc8PHxKdPW39+fzMzM8/azfv16Pv/8c5YsWWI7tn//fp566inWrl2Lk1P5huls//7+/pf87EmTJjFz5kzy8/Pp0qULixcvLtdnXExMTAxvvPEGPXr0oGnTpqxcuZJvvvkGi8VyxX3/UW0e99dee428vDzuuece27EFCxawdevWSlv/eSEa99/H3dPTk+joaJ5//nlatmyJv78/n332GQkJCURERFSo70upjuNuGAYPPvggo0ePplOnThw8eLDcfZ+vX7PZzJkzZ3B3dwcgMTGR6OhoCgoKqFu3LgsXLiQqKqpcn1FeGvey4/7UU09hNpuJjIzE0dERi8XCiy++yODBg8v1GeVVm8a9vHJycmjUqBGFhYU4Ojoya9Ys+vbtW6mfoXEva9CgQRw/fpxu3bphGAYlJSWMHj2av//97+Xuo1rP4MbFxbFjxw4WLFhw2X3s2LGD2267jSlTptCvXz+gdFZi0KBBTJs2jebNm5/3ffPmzaNu3bq219q1ayv0uU8++STbtm1j+fLlODo6MnToUNvNAn/sd/To0eXuc8aMGTRr1ozIyEhcXFwYM2YMDz30EA4OlfvHXFvHff78+UybNo0vvvgCPz8/ANLS0hg7dizz5s07Zzaxsmncfx93gE8//RTDMGjUqBGurq68/fbb3H///fp+B9555x1yc3OZPHnyBdtc7s8ZgBYtWrB9+3Y2bNjAo48+yrBhw9i1a1eF+rgUjXtZX3zxBfPmzWP+/Pls3bqVuXPn8tprrzF37txy91EeGvdzeXp6sn37djZt2sSLL77IhAkTWLNmTYX6uBSNe1lr1qzhpZdeYtasWWzdupVvvvmGJUuW8Pzzz5e7j2q7BjcuLs5o3LixceDAgTLHV65caQDGqVOnyhwPCQkx3njjjTLHdu7cafj5+Rl///vfyxw/deqUARiOjo62l8lksh1buXKlYTabjf3799te+fn5RnJy8nnXn/To0cN44oknLngtaWlpBmCsX7/eMAyjTL9Hjx49p/2l1gqdOXPGOHz4sGG1Wo2JEycaUVFRF2xbUbV13D/77DPD3d39nDWfCxcuPKdmwDCZTIajo6NRUlJywc+vCI374vN1ZRiGYeTl5Rnp6emGYZTeeHbLLbdcsG1FVddxv+222wwHB4dzvi8dHR2NoUOHGoZx/nHv3r37OT9bPv74Y8PLy+ui49S7d29j1KhRF21TERr3c8e9cePG59xj8fzzzxstWrS49ICWU20b9z+qyBrcESNGGP369StX2/LQuI8953i3bt2Mv/3tb2WOnb2BzmKxXHAs/6jaBVyr1WrExcUZQUFBxr59+845f3ZR9ldffWU7tmfPnnMWZe/YscPw8/MznnzyyXP6sFgsRmJiYpnXo48+arRo0cJITEwsc4fjn2sLCAgwXnvtNduxnJycS950c+jQIQMwVq9eXZ4hKPdfxKKiIqNp06bG5MmTy9XvxdTmcZ8/f77h5uZmLFq06JxzZrP5nJo7depkPPDAA0ZiYuJF+y0Pjfv5x/18Tp48aXh7exvvv/9+udpfTHUf90OHDpXp9/vvvzcA46uvvjLS0tIueN0TJ040WrduXebY/ffff8mbzG6++WZj2LBhF21THhr33/153OvXr2/MmjWrTJuXXnrJaNas2QX7La/aOu5/VJGA+9BDDxk9e/YsV9uL0bhfeNw7dOhgTJw4scyx+fPnG+7u7uWeOKp2AffRRx81vL29jTVr1hgZGRm2V35+vq3N6NGjjZCQEGPVqlXG5s2bz9mWKDEx0WjYsKHxwAMPlOkjKyvrgp9bnrsODaN0Ww0fHx/jv//9r/Hrr78at912W5ltNX7++WfjnXfeMbZt22YcPHjQWLlypdG1a1ejadOmRkFBwUX73rZtm7Ft2zajY8eOxqBBg4xt27YZO3futJ3/+eefja+//tpITk42fvzxR6NXr15GWFjYOf/v73LU1nGfN2+e4eTkZLz77rtlas7Ozr7geypzFwWN+4XHfdmyZcbSpUuNAwcOGMuXLzfatm1rdO7cucK7M5xPdR/3P0tJSanQdlVPPvmksXv3buPdd989Z0ukp556yvjhhx+MlJQU49dffzWeeuopw2QyGcuXL79k3Zeicb/wuA8bNsxo1KiRbZuwb775xmjQoME5IeBy1NZxN4xL/1596aWXjOXLlxvJycnGrl27jNdee81wcnIyPvjgg0v2fSka9wuP+5QpUwxPT0/js88+s/2Mb9q0qXHPPfdcsu+zql3ABc77+uSTT2xtzpw5Yzz22GNGvXr1DA8PD+OOO+4wMjIybOenTJly3j5CQ0Mv+Lnl/YawWq3GP/7xD8Pf399wdXU1evfubezdu9d2/tdffzVuvvlmo379+oarq6vRpEkTY/To0eXau/NSNa9Zs8Zo2bKl4erqavj6+hpDhgwxjhw5csl+y6O2jnvPnj3PW/PFZqsqM+Bq3C887p9//rkRHh5uuLi4GAEBAUZcXNxF/49HRVT3cf+zivziWb16tdGuXTvDxcXFCA8PL3PNhmEYw4cPN0JDQw0XFxejYcOGRu/evSsl3BqGxv1i4242m42xY8caISEhhpubmxEeHm48/fTTRmFh4SX7vpTaPO6Xqvnpp582IiIiDDc3N6NevXpGdHS0sWDBgkv2Wx4a9wvXXFxcbEydOtVo2rSp4ebmZgQHBxuPPfZYhSbsTL99kIiIiIhIjVCtd1EQEREREfkzBVwRERERqVEUcEVERESkRlHAFREREZEaRQFXRERERGoUBVwRERERqVEUcEVERESkRlHAFREREZEaRQFXROQaefDBBzGZTJhMJpydnfH396dv3758/PHHWK3Wc9rHxMTg6OjIpk2bADh48KDt/Rd6zZkzhzVr1lzwfGZm5rW+bBGRa04BV0TkGurfvz8ZGRkcPHiQpUuXcvPNNzN27FgGDhxISUmJrV1qairr169nzJgxfPzxxwAEBweTkZFhe/3f//0frVq1KnPs3nvvtfWxd+/eMucyMjLw8/O75tcsInKtOdm7ABGR2sTV1ZWAgAAAGjVqRIcOHejSpQu9e/dmzpw5jBw5EoBPPvmEgQMH8uijj9KlSxfeeOMN3N3dbe8FqFu3Lk5OTmWO/ZGfnx8+Pj5X/ZpERKoazeCKiNhZr169aNu2Ld988w0AhmHwySef8MADDxAZGUlERARfffWVnasUEak+FHBFRKqAyMhIDh48CMCKFSvIz88nJiYGgAceeICPPvqown02btyYunXr2l6tWrWqzJJFRKosLVEQEakCDMPAZDIB8PHHH3Pvvffi5FT6I/r+++/nySefJDk5maZNm5a7z7Vr1+Lp6Wn72tnZuXKLFhGpohRwRUSqgN27dxMWFsbJkydZuHAhxcXFvPfee7bzFouFjz/+mBdffLHcfYaFhWkNrojUSlqiICJiZ6tWrSIxMZG77rqLefPm0bhxY3755Re2b99ue73++uvMmTMHi8Vi73JFRKo8zeCKiFxDhYWFZGZmYrFYOHr0KMuWLWP69OkMHDiQoUOH0rFjR+6++25at25d5n3BwcFMnjyZZcuWERsbW67PysrKoqCgoMwxX19fLVUQkRpPAVdE5BpatmwZgYGBODk5Ua9ePdq2bcvbb7/NsGHD2LZtG7/88gsffPDBOe/z9vamd+/efPTRR+UOuC1atDjnWEJCAl26dLni6xARqcpMhmEY9i5CRERERKSyaA2uiIiIiNQoCrgiIiIiUqMo4IqIiIhIjaKAKyIiIiI1igKuiIiIiNQoCrgiIiIiUqMo4IqIiIhIjaKAKyIiIiI1igKuiIiIiNQoCrgiIiIiUqMo4IqIiIhIjaKAKyIiIiI1yv8DDnrXSraIeAsAAAAASUVORK5CYII=",
            "text/plain": [
              "\u003cFigure size 800x400 with 1 Axes\u003e"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "serve['body_mass_g'].plot(legend = True, figsize = (8, 4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "kQR3nLMQxLVM"
      },
      "source": [
        "---\n",
        "## Monitoring With Evaluations\n",
        "\n",
        "It might seem like ongoing model evaluations are a great solution to model monitoring.  If the actual known values are available immediately then this could be possible to use for monitoring.  However, detecting a change in evaluation metrics means the model has realized a degradation.  It is a lagging signal.  To be more proactive we should monitor the inputs to the model, the features.  More on that in the next section.\n",
        "\n",
        "First, look at the evaluation metrics for the most recent days data which has the cumulative change of the past 30 days introduced above.  In this case, the actual known values is known. This may not be possible in many real-world examples where the actual known values are:\n",
        "- delayed by minutes, hours, days, or even longer\n",
        "- not always known for all instances\n",
        "- potentially in a different status that can change over a period of time before being confirmed as final\n",
        "\n",
        "In reviewing the evaluation for the most recent instances here, the `precision` has dropped and the `log_loss` has increased drastically.  The confusion matrix shows Chinstrap penguins's are now being misclassified much more often than when the model was trained.  \n",
        "\n",
        "The question becomes \"How soon could this have been noticed?\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "tags": [],
        "id": "N6Dmv3SDxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "87d1e674038e406a86a7531c9f879a5a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "032eb1e3d765418983a42f6e237e8c59",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eprecision\u003c/th\u003e\n",
              "      \u003cth\u003erecall\u003c/th\u003e\n",
              "      \u003cth\u003eaccuracy\u003c/th\u003e\n",
              "      \u003cth\u003ef1_score\u003c/th\u003e\n",
              "      \u003cth\u003elog_loss\u003c/th\u003e\n",
              "      \u003cth\u003eroc_auc\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003e0.899338\u003c/td\u003e\n",
              "      \u003ctd\u003e0.8262\u003c/td\u003e\n",
              "      \u003ctd\u003e0.869565\u003c/td\u003e\n",
              "      \u003ctd\u003e0.845232\u003c/td\u003e\n",
              "      \u003ctd\u003e0.624126\u003c/td\u003e\n",
              "      \u003ctd\u003e0.983902\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "   precision  recall  accuracy  f1_score  log_loss   roc_auc\n",
              "0   0.899338  0.8262  0.869565  0.845232  0.624126  0.983902"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.EVALUATE (\n",
        "    MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "        WHERE DATE(instance_timestamp) \u003e= CURRENT_DATE()\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "tags": [],
        "id": "napnxCvkxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1954b8a0397640faaf9daf644f333327",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ed6b8477a83a4f21a2d099bbb64b138c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eexpected_label\u003c/th\u003e\n",
              "      \u003cth\u003eAdelie_Penguin__Pygoscelis_adeliae_\u003c/th\u003e\n",
              "      \u003cth\u003eChinstrap_penguin__Pygoscelis_antarctica_\u003c/th\u003e\n",
              "      \u003cth\u003eGentoo_penguin__Pygoscelis_papua_\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003e31\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e4\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eChinstrap penguin (Pygoscelis antarctica)\u003c/td\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e7\u003c/td\u003e\n",
              "      \u003ctd\u003e3\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e1\u003c/td\u003e\n",
              "      \u003ctd\u003e0\u003c/td\u003e\n",
              "      \u003ctd\u003e22\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                              expected_label  \\\n",
              "0        Adelie Penguin (Pygoscelis adeliae)   \n",
              "1  Chinstrap penguin (Pygoscelis antarctica)   \n",
              "2          Gentoo penguin (Pygoscelis papua)   \n",
              "\n",
              "   Adelie_Penguin__Pygoscelis_adeliae_  \\\n",
              "0                                   31   \n",
              "1                                    1   \n",
              "2                                    1   \n",
              "\n",
              "   Chinstrap_penguin__Pygoscelis_antarctica_  \\\n",
              "0                                          0   \n",
              "1                                          7   \n",
              "2                                          0   \n",
              "\n",
              "   Gentoo_penguin__Pygoscelis_papua_  \n",
              "0                                  4  \n",
              "1                                  3  \n",
              "2                                 22  "
            ]
          },
          "execution_count": 30,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.CONFUSION_MATRIX (\n",
        "    MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "        WHERE DATE(instance_timestamp) \u003e= CURRENT_DATE()\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OJfz6QzgxLVM"
      },
      "source": [
        "### Evaluations: Over Time\n",
        "\n",
        "In this case, we know the actual identification of the `species` and are measuring the same penguins daily.  Reviewing the evaluations daily could give a good sense of how the model is performing on a daily basis.\n",
        "\n",
        "The result of the daily evaluations below shows that the log_loss starts to increase immediately and it takes more than a week for a noticable impact on precision/recall/accuracy/f1_score. \n",
        "\n",
        "What if the actual values where only known after a lag of days or weeks?  How quickly would evaluations be able to indicate the model's accuracy is possibly questionable?  The answer is a different approach - Model Monitoring.  This is covered in the section that follows."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "tags": [],
        "id": "ZNPTZfBrxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c1d97e56f49642909e03af6c5dc0c44e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a965ab76cad14700830c234d1da356c1",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "%%bigquery evaluations\n",
        "DECLARE counter INT64 DEFAULT 30;\n",
        "DECLARE result STRUCT\u003cprecision FLOAT64, recall FLOAT64, accuracy FLOAT64, f1_score FLOAT64, log_loss FLOAT64, roc_auc FLOAT64, eval_date DATE\u003e;\n",
        "DECLARE results ARRAY\u003cSTRUCT\u003cprecision FLOAT64, recall FLOAT64, accuracy FLOAT64, f1_score FLOAT64, log_loss FLOAT64, roc_auc FLOAT64, eval_date DATE\u003e\u003e DEFAULT [];\n",
        "LOOP\n",
        "    IF counter \u003c 0 THEN LEAVE;\n",
        "    END IF;\n",
        "    SET result = (\n",
        "            SELECT AS STRUCT precision, recall, accuracy, f1_score, log_loss, roc_auc, DATE_SUB(CURRENT_DATE(), INTERVAL counter DAY) AS eval_date\n",
        "            FROM ML.EVALUATE (\n",
        "                MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "                (\n",
        "                    SELECT *\n",
        "                    FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "                    WHERE DATE(instance_timestamp) = DATE_SUB(CURRENT_DATE(), INTERVAL counter DAY)\n",
        "                )\n",
        "            )\n",
        "    );\n",
        "    SET results = ARRAY_CONCAT(results,[result]);\n",
        "    SET counter = counter - 1;\n",
        "END LOOP;\n",
        "SELECT * FROM UNNEST(results);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "tags": [],
        "id": "RM8XMdp_xLVM"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eprecision\u003c/th\u003e\n",
              "      \u003cth\u003erecall\u003c/th\u003e\n",
              "      \u003cth\u003eaccuracy\u003c/th\u003e\n",
              "      \u003cth\u003ef1_score\u003c/th\u003e\n",
              "      \u003cth\u003elog_loss\u003c/th\u003e\n",
              "      \u003cth\u003eroc_auc\u003c/th\u003e\n",
              "      \u003cth\u003eeval_date\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003e0.981982\u003c/td\u003e\n",
              "      \u003ctd\u003e0.955204\u003c/td\u003e\n",
              "      \u003ctd\u003e0.971014\u003c/td\u003e\n",
              "      \u003ctd\u003e0.967460\u003c/td\u003e\n",
              "      \u003ctd\u003e0.342359\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998511\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-19\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003e0.981982\u003c/td\u003e\n",
              "      \u003ctd\u003e0.955204\u003c/td\u003e\n",
              "      \u003ctd\u003e0.971014\u003c/td\u003e\n",
              "      \u003ctd\u003e0.967460\u003c/td\u003e\n",
              "      \u003ctd\u003e0.342535\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998511\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-20\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003e0.981982\u003c/td\u003e\n",
              "      \u003ctd\u003e0.955204\u003c/td\u003e\n",
              "      \u003ctd\u003e0.971014\u003c/td\u003e\n",
              "      \u003ctd\u003e0.967460\u003c/td\u003e\n",
              "      \u003ctd\u003e0.347173\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998715\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-21\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003e0.981982\u003c/td\u003e\n",
              "      \u003ctd\u003e0.955204\u003c/td\u003e\n",
              "      \u003ctd\u003e0.971014\u003c/td\u003e\n",
              "      \u003ctd\u003e0.967460\u003c/td\u003e\n",
              "      \u003ctd\u003e0.352256\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998715\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-22\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003e0.981982\u003c/td\u003e\n",
              "      \u003ctd\u003e0.955204\u003c/td\u003e\n",
              "      \u003ctd\u003e0.971014\u003c/td\u003e\n",
              "      \u003ctd\u003e0.967460\u003c/td\u003e\n",
              "      \u003ctd\u003e0.365219\u003c/td\u003e\n",
              "      \u003ctd\u003e0.998113\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-23\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e5\u003c/th\u003e\n",
              "      \u003ctd\u003e0.981982\u003c/td\u003e\n",
              "      \u003ctd\u003e0.955204\u003c/td\u003e\n",
              "      \u003ctd\u003e0.971014\u003c/td\u003e\n",
              "      \u003ctd\u003e0.967460\u003c/td\u003e\n",
              "      \u003ctd\u003e0.379829\u003c/td\u003e\n",
              "      \u003ctd\u003e0.996667\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-24\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e6\u003c/th\u003e\n",
              "      \u003ctd\u003e0.981982\u003c/td\u003e\n",
              "      \u003ctd\u003e0.955204\u003c/td\u003e\n",
              "      \u003ctd\u003e0.971014\u003c/td\u003e\n",
              "      \u003ctd\u003e0.967460\u003c/td\u003e\n",
              "      \u003ctd\u003e0.392832\u003c/td\u003e\n",
              "      \u003ctd\u003e0.995250\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-25\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e7\u003c/th\u003e\n",
              "      \u003ctd\u003e0.966989\u003c/td\u003e\n",
              "      \u003ctd\u003e0.945680\u003c/td\u003e\n",
              "      \u003ctd\u003e0.956522\u003c/td\u003e\n",
              "      \u003ctd\u003e0.955550\u003c/td\u003e\n",
              "      \u003ctd\u003e0.410668\u003c/td\u003e\n",
              "      \u003ctd\u003e0.994368\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-26\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e8\u003c/th\u003e\n",
              "      \u003ctd\u003e0.966989\u003c/td\u003e\n",
              "      \u003ctd\u003e0.945680\u003c/td\u003e\n",
              "      \u003ctd\u003e0.956522\u003c/td\u003e\n",
              "      \u003ctd\u003e0.955550\u003c/td\u003e\n",
              "      \u003ctd\u003e0.434265\u003c/td\u003e\n",
              "      \u003ctd\u003e0.991149\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-27\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e9\u003c/th\u003e\n",
              "      \u003ctd\u003e0.966989\u003c/td\u003e\n",
              "      \u003ctd\u003e0.945680\u003c/td\u003e\n",
              "      \u003ctd\u003e0.956522\u003c/td\u003e\n",
              "      \u003ctd\u003e0.955550\u003c/td\u003e\n",
              "      \u003ctd\u003e0.446073\u003c/td\u003e\n",
              "      \u003ctd\u003e0.989968\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-28\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e10\u003c/th\u003e\n",
              "      \u003ctd\u003e0.940952\u003c/td\u003e\n",
              "      \u003ctd\u003e0.905854\u003c/td\u003e\n",
              "      \u003ctd\u003e0.927536\u003c/td\u003e\n",
              "      \u003ctd\u003e0.919841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.470836\u003c/td\u003e\n",
              "      \u003ctd\u003e0.986984\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-29\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e11\u003c/th\u003e\n",
              "      \u003ctd\u003e0.940952\u003c/td\u003e\n",
              "      \u003ctd\u003e0.905854\u003c/td\u003e\n",
              "      \u003ctd\u003e0.927536\u003c/td\u003e\n",
              "      \u003ctd\u003e0.919841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.476310\u003c/td\u003e\n",
              "      \u003ctd\u003e0.986170\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-30\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e12\u003c/th\u003e\n",
              "      \u003ctd\u003e0.940952\u003c/td\u003e\n",
              "      \u003ctd\u003e0.905854\u003c/td\u003e\n",
              "      \u003ctd\u003e0.927536\u003c/td\u003e\n",
              "      \u003ctd\u003e0.919841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.484181\u003c/td\u003e\n",
              "      \u003ctd\u003e0.992104\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-03-31\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e13\u003c/th\u003e\n",
              "      \u003ctd\u003e0.940952\u003c/td\u003e\n",
              "      \u003ctd\u003e0.905854\u003c/td\u003e\n",
              "      \u003ctd\u003e0.927536\u003c/td\u003e\n",
              "      \u003ctd\u003e0.919841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.498877\u003c/td\u003e\n",
              "      \u003ctd\u003e0.991782\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-01\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e14\u003c/th\u003e\n",
              "      \u003ctd\u003e0.940952\u003c/td\u003e\n",
              "      \u003ctd\u003e0.905854\u003c/td\u003e\n",
              "      \u003ctd\u003e0.927536\u003c/td\u003e\n",
              "      \u003ctd\u003e0.919841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.504442\u003c/td\u003e\n",
              "      \u003ctd\u003e0.991474\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-02\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e15\u003c/th\u003e\n",
              "      \u003ctd\u003e0.940952\u003c/td\u003e\n",
              "      \u003ctd\u003e0.905854\u003c/td\u003e\n",
              "      \u003ctd\u003e0.927536\u003c/td\u003e\n",
              "      \u003ctd\u003e0.919841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.517814\u003c/td\u003e\n",
              "      \u003ctd\u003e0.990829\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-03\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e16\u003c/th\u003e\n",
              "      \u003ctd\u003e0.940952\u003c/td\u003e\n",
              "      \u003ctd\u003e0.905854\u003c/td\u003e\n",
              "      \u003ctd\u003e0.927536\u003c/td\u003e\n",
              "      \u003ctd\u003e0.919841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.524964\u003c/td\u003e\n",
              "      \u003ctd\u003e0.990199\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-04\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e17\u003c/th\u003e\n",
              "      \u003ctd\u003e0.940952\u003c/td\u003e\n",
              "      \u003ctd\u003e0.905854\u003c/td\u003e\n",
              "      \u003ctd\u003e0.927536\u003c/td\u003e\n",
              "      \u003ctd\u003e0.919841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.536792\u003c/td\u003e\n",
              "      \u003ctd\u003e0.990199\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-05\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e18\u003c/th\u003e\n",
              "      \u003ctd\u003e0.940952\u003c/td\u003e\n",
              "      \u003ctd\u003e0.905854\u003c/td\u003e\n",
              "      \u003ctd\u003e0.927536\u003c/td\u003e\n",
              "      \u003ctd\u003e0.919841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.539436\u003c/td\u003e\n",
              "      \u003ctd\u003e0.989232\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-06\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e19\u003c/th\u003e\n",
              "      \u003ctd\u003e0.940952\u003c/td\u003e\n",
              "      \u003ctd\u003e0.905854\u003c/td\u003e\n",
              "      \u003ctd\u003e0.927536\u003c/td\u003e\n",
              "      \u003ctd\u003e0.919841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.557831\u003c/td\u003e\n",
              "      \u003ctd\u003e0.988924\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-07\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e20\u003c/th\u003e\n",
              "      \u003ctd\u003e0.940952\u003c/td\u003e\n",
              "      \u003ctd\u003e0.905854\u003c/td\u003e\n",
              "      \u003ctd\u003e0.927536\u003c/td\u003e\n",
              "      \u003ctd\u003e0.919841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.567827\u003c/td\u003e\n",
              "      \u003ctd\u003e0.988924\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-08\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e21\u003c/th\u003e\n",
              "      \u003ctd\u003e0.940952\u003c/td\u003e\n",
              "      \u003ctd\u003e0.905854\u003c/td\u003e\n",
              "      \u003ctd\u003e0.927536\u003c/td\u003e\n",
              "      \u003ctd\u003e0.919841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.572287\u003c/td\u003e\n",
              "      \u003ctd\u003e0.988602\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-09\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e22\u003c/th\u003e\n",
              "      \u003ctd\u003e0.918664\u003c/td\u003e\n",
              "      \u003ctd\u003e0.866027\u003c/td\u003e\n",
              "      \u003ctd\u003e0.898551\u003c/td\u003e\n",
              "      \u003ctd\u003e0.883214\u003c/td\u003e\n",
              "      \u003ctd\u003e0.578770\u003c/td\u003e\n",
              "      \u003ctd\u003e0.987730\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-10\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e23\u003c/th\u003e\n",
              "      \u003ctd\u003e0.918664\u003c/td\u003e\n",
              "      \u003ctd\u003e0.866027\u003c/td\u003e\n",
              "      \u003ctd\u003e0.898551\u003c/td\u003e\n",
              "      \u003ctd\u003e0.883214\u003c/td\u003e\n",
              "      \u003ctd\u003e0.581189\u003c/td\u003e\n",
              "      \u003ctd\u003e0.987730\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-11\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e24\u003c/th\u003e\n",
              "      \u003ctd\u003e0.908369\u003c/td\u003e\n",
              "      \u003ctd\u003e0.856503\u003c/td\u003e\n",
              "      \u003ctd\u003e0.884058\u003c/td\u003e\n",
              "      \u003ctd\u003e0.872205\u003c/td\u003e\n",
              "      \u003ctd\u003e0.583999\u003c/td\u003e\n",
              "      \u003ctd\u003e0.987730\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-12\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e25\u003c/th\u003e\n",
              "      \u003ctd\u003e0.908369\u003c/td\u003e\n",
              "      \u003ctd\u003e0.856503\u003c/td\u003e\n",
              "      \u003ctd\u003e0.884058\u003c/td\u003e\n",
              "      \u003ctd\u003e0.872205\u003c/td\u003e\n",
              "      \u003ctd\u003e0.596816\u003c/td\u003e\n",
              "      \u003ctd\u003e0.987407\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-13\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e26\u003c/th\u003e\n",
              "      \u003ctd\u003e0.908369\u003c/td\u003e\n",
              "      \u003ctd\u003e0.856503\u003c/td\u003e\n",
              "      \u003ctd\u003e0.884058\u003c/td\u003e\n",
              "      \u003ctd\u003e0.872205\u003c/td\u003e\n",
              "      \u003ctd\u003e0.601596\u003c/td\u003e\n",
              "      \u003ctd\u003e0.986455\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-14\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e27\u003c/th\u003e\n",
              "      \u003ctd\u003e0.908369\u003c/td\u003e\n",
              "      \u003ctd\u003e0.856503\u003c/td\u003e\n",
              "      \u003ctd\u003e0.884058\u003c/td\u003e\n",
              "      \u003ctd\u003e0.872205\u003c/td\u003e\n",
              "      \u003ctd\u003e0.606968\u003c/td\u003e\n",
              "      \u003ctd\u003e0.986165\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-15\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e28\u003c/th\u003e\n",
              "      \u003ctd\u003e0.899338\u003c/td\u003e\n",
              "      \u003ctd\u003e0.826200\u003c/td\u003e\n",
              "      \u003ctd\u003e0.869565\u003c/td\u003e\n",
              "      \u003ctd\u003e0.845232\u003c/td\u003e\n",
              "      \u003ctd\u003e0.619822\u003c/td\u003e\n",
              "      \u003ctd\u003e0.984532\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-16\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e29\u003c/th\u003e\n",
              "      \u003ctd\u003e0.899338\u003c/td\u003e\n",
              "      \u003ctd\u003e0.826200\u003c/td\u003e\n",
              "      \u003ctd\u003e0.869565\u003c/td\u003e\n",
              "      \u003ctd\u003e0.845232\u003c/td\u003e\n",
              "      \u003ctd\u003e0.620190\u003c/td\u003e\n",
              "      \u003ctd\u003e0.984532\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-17\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e30\u003c/th\u003e\n",
              "      \u003ctd\u003e0.899338\u003c/td\u003e\n",
              "      \u003ctd\u003e0.826200\u003c/td\u003e\n",
              "      \u003ctd\u003e0.869565\u003c/td\u003e\n",
              "      \u003ctd\u003e0.845232\u003c/td\u003e\n",
              "      \u003ctd\u003e0.624126\u003c/td\u003e\n",
              "      \u003ctd\u003e0.983902\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-18\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "    precision    recall  accuracy  f1_score  log_loss   roc_auc   eval_date\n",
              "0    0.981982  0.955204  0.971014  0.967460  0.342359  0.998511  2024-03-19\n",
              "1    0.981982  0.955204  0.971014  0.967460  0.342535  0.998511  2024-03-20\n",
              "2    0.981982  0.955204  0.971014  0.967460  0.347173  0.998715  2024-03-21\n",
              "3    0.981982  0.955204  0.971014  0.967460  0.352256  0.998715  2024-03-22\n",
              "4    0.981982  0.955204  0.971014  0.967460  0.365219  0.998113  2024-03-23\n",
              "5    0.981982  0.955204  0.971014  0.967460  0.379829  0.996667  2024-03-24\n",
              "6    0.981982  0.955204  0.971014  0.967460  0.392832  0.995250  2024-03-25\n",
              "7    0.966989  0.945680  0.956522  0.955550  0.410668  0.994368  2024-03-26\n",
              "8    0.966989  0.945680  0.956522  0.955550  0.434265  0.991149  2024-03-27\n",
              "9    0.966989  0.945680  0.956522  0.955550  0.446073  0.989968  2024-03-28\n",
              "10   0.940952  0.905854  0.927536  0.919841  0.470836  0.986984  2024-03-29\n",
              "11   0.940952  0.905854  0.927536  0.919841  0.476310  0.986170  2024-03-30\n",
              "12   0.940952  0.905854  0.927536  0.919841  0.484181  0.992104  2024-03-31\n",
              "13   0.940952  0.905854  0.927536  0.919841  0.498877  0.991782  2024-04-01\n",
              "14   0.940952  0.905854  0.927536  0.919841  0.504442  0.991474  2024-04-02\n",
              "15   0.940952  0.905854  0.927536  0.919841  0.517814  0.990829  2024-04-03\n",
              "16   0.940952  0.905854  0.927536  0.919841  0.524964  0.990199  2024-04-04\n",
              "17   0.940952  0.905854  0.927536  0.919841  0.536792  0.990199  2024-04-05\n",
              "18   0.940952  0.905854  0.927536  0.919841  0.539436  0.989232  2024-04-06\n",
              "19   0.940952  0.905854  0.927536  0.919841  0.557831  0.988924  2024-04-07\n",
              "20   0.940952  0.905854  0.927536  0.919841  0.567827  0.988924  2024-04-08\n",
              "21   0.940952  0.905854  0.927536  0.919841  0.572287  0.988602  2024-04-09\n",
              "22   0.918664  0.866027  0.898551  0.883214  0.578770  0.987730  2024-04-10\n",
              "23   0.918664  0.866027  0.898551  0.883214  0.581189  0.987730  2024-04-11\n",
              "24   0.908369  0.856503  0.884058  0.872205  0.583999  0.987730  2024-04-12\n",
              "25   0.908369  0.856503  0.884058  0.872205  0.596816  0.987407  2024-04-13\n",
              "26   0.908369  0.856503  0.884058  0.872205  0.601596  0.986455  2024-04-14\n",
              "27   0.908369  0.856503  0.884058  0.872205  0.606968  0.986165  2024-04-15\n",
              "28   0.899338  0.826200  0.869565  0.845232  0.619822  0.984532  2024-04-16\n",
              "29   0.899338  0.826200  0.869565  0.845232  0.620190  0.984532  2024-04-17\n",
              "30   0.899338  0.826200  0.869565  0.845232  0.624126  0.983902  2024-04-18"
            ]
          },
          "execution_count": 32,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "evaluations"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "tags": [],
        "id": "fPxkpZ5wxLVM"
      },
      "outputs": [],
      "source": [
        "evaluations.set_index('eval_date', inplace = True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "tags": [],
        "id": "235y6QQBxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "\u003cAxes: title={'center': 'Evaluation Metrics Over Last 30 days'}, xlabel='eval_date'\u003e"
            ]
          },
          "execution_count": 34,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqsAAAE8CAYAAADnkPiXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABzwElEQVR4nO3deVhU1f8H8PfMMBs7iiyyKm64geKG5pIb7rZrmpKlZWpq/solS9RKK/cstfrmUmlpapZLKpKa+477voALm7JvwzBzfn+MjI6AAgIzyPv1PPeBOffce858GPTDueeeKxFCCBARERERWSCpuTtARERERFQYJqtEREREZLGYrBIRERGRxWKySkREREQWi8kqEREREVksJqtEREREZLGYrBIRERGRxWKySkREREQWi8kqEREREVksJqtEFkYikWDq1KlmaXvXrl2QSCTYtWuXWdq3RB06dECHDh3M3Q2qwN588034+vqauxtEFRaTVaICLF++HBKJpNDt4MGD5u7iU1m0aBGWL19u7m6Y6NChAyQSCWrXrl3g/vDwcGP8165dW+zz37lzB1OnTkVkZORT9rTsZWRk4LPPPkPjxo1hbW0NBwcHtG3bFj///DMs8QnZHTp0QMOGDcutvVWrVmH+/PlFrj9jxgy0atUK1apVg0qlQu3atTF27FgkJCTkq6vX6/H111+jRo0aUKlUaNy4MX777bdS7D0RFZeVuTtAZMmmT5+OGjVq5CuvVauWGXpTehYtWgRnZ2e8+eabJuXt2rVDVlYWFAqFWfqlUqlw5coVHD58GC1atDDZt3LlSqhUKmRnZ5fo3Hfu3MG0adPg6+uLwMDAIh+3ffv2ErVXUnFxcejUqRPOnz+P/v37Y9SoUcjOzsa6desQGhqKLVu2YOXKlZDJZOXaL0uyatUqnDlzBmPHji1S/WPHjiEwMBD9+/eHnZ0dzp8/jx9//BGbN29GZGQkbGxsjHUnT56ML7/8EsOGDUPz5s3x119/YcCAAZBIJOjfv38ZvSMiehwmq0SP0b17dzRr1szc3Sg3UqkUKpXKbO37+fkhNzcXv/32m0mymp2djT///BM9e/bEunXryqUvmZmZsLa2LvfEPTQ0FOfPn8eff/6JPn36GMtHjx6Njz76CLNnz0aTJk0wYcKEcuuTXq9HTk6OWT8bT6Ogz0xwcDBeeeUVbNy40ZiE3r59G3PmzMHIkSPx7bffAgCGDh2K9u3b46OPPsKrr75aqf9IIDIXTgMgKiGtVosqVapgyJAh+falpqZCpVLhww8/BADk5ORgypQpCAoKgoODA2xsbNC2bVvs3Lnzie0UNt9t6tSpkEgkJmXLli1Dx44d4eLiAqVSifr162Px4sUmdXx9fXH27Fns3r3beFk9b05mYXNW//jjDwQFBUGtVsPZ2RlvvPEGbt++na+ftra2uH37Nl544QXY2tqiWrVq+PDDD6HT6Z74PvO8/vrrWL16NfR6vbFs48aNyMzMxGuvvVbgMbdv38Zbb70FV1dXKJVKNGjQAEuXLjXu37VrF5o3bw4AGDJkiPF9502FyLuMfezYMbRr1w7W1tb4+OOPjfsenbOanZ2NqVOnok6dOlCpVHB3d8dLL72Eq1evGuv8/vvvCAoKgp2dHezt7dGoUSMsWLDgse/94MGD2LZtG958802TRDXPzJkzUbt2bXz11VfIysoq1mcQADQaDcLCwlCrVi0olUp4eXlh/Pjx0Gg0JsdKJBKMGjUKK1euRIMGDaBUKrF169bH9v1JTp06hTfffBM1a9aESqWCm5sb3nrrLdy7d8+kXlpaGsaOHQtfX18olUq4uLigS5cuOH78OADDz2Pz5s2Iiooy/hxLMh8075jk5GRj2V9//QWtVosRI0YYyyQSCd577z3cunULBw4ceOJ5N2zYgIYNG0KlUqFhw4b4888/C6w3e/ZstG7dGlWrVoVarUZQUFC+6S3t27dHQEBAgcfXrVsXISEhxtcl+bwRVRQcWSV6jJSUFNy9e9ekTCKRoGrVqpDL5XjxxRexfv16fP/99yYjcBs2bIBGozGO2KSmpuJ///sfXn/9dQwbNgxpaWn46aefEBISgsOHDxfrsvTjLF68GA0aNECfPn1gZWWFjRs3YsSIEdDr9Rg5ciQAYP78+Xj//fdha2uLyZMnAwBcXV0LPefy5csxZMgQNG/eHDNnzkRcXBwWLFiAffv24cSJE3B0dDTW1el0CAkJQcuWLTF79mzs2LEDc+bMgZ+fH957770ivYcBAwZg6tSp2LVrFzp27AjAcNm3U6dOcHFxyVc/Li4OrVq1MiZY1apVwz///IO3334bqampGDt2LPz9/TF9+nRMmTIF77zzDtq2bQsAaN26tfE89+7dQ/fu3dG/f3+88cYbhcZEp9OhV69eiIiIQP/+/TFmzBikpaUhPDwcZ86cgZ+fH8LDw/H666+jU6dO+OqrrwAA58+fx759+zBmzJhC3/vGjRsBAIMHDy5wv5WVFQYMGIBp06Zh37596Ny5c5E/g3q9Hn369MHevXvxzjvvwN/fH6dPn8a8efNw6dIlbNiwwaStf//9F2vWrMGoUaPg7Oz81DcIhYeH49q1axgyZAjc3Nxw9uxZ/PDDDzh79iwOHjxo/MNr+PDhWLt2LUaNGoX69evj3r172Lt3L86fP4+mTZti8uTJSElJwa1btzBv3jwAgK2t7RPbF0Lg3r17yM3NxeXLlzFx4kTIZDKTP0ROnDgBGxsb+Pv7mxybN8p/4sQJPPfcc4W2sX37drz88suoX78+Zs6ciXv37mHIkCHw9PTMV3fBggXo06cPBg4ciJycHPz+++949dVXsWnTJvTs2RMAMGjQIAwbNgxnzpwxmRN85MgRXLp0CZ988okxtiX5vBFVGIKI8lm2bJkAUOCmVCqN9bZt2yYAiI0bN5oc36NHD1GzZk3j69zcXKHRaEzqJCUlCVdXV/HWW2+ZlAMQYWFhxtehoaHCx8cnXx/DwsLEo7/CmZmZ+eqFhISY9EUIIRo0aCDat2+fr+7OnTsFALFz504hhBA5OTnCxcVFNGzYUGRlZRnrbdq0SQAQU6ZMMeknADF9+nSTczZp0kQEBQXla+tR7du3Fw0aNBBCCNGsWTPx9ttvCyEMcVIoFGLFihXG/v3xxx/G495++23h7u4u7t69a3K+/v37CwcHB2NMjhw5IgCIZcuWFdg2ALFkyZIC9z0cq6VLlwoAYu7cufnq6vV6IYQQY8aMEfb29iI3N/eJ7/thL7zwggAgkpKSCq2zfv16AUB88803QoiifwZ/+eUXIZVKxZ49e0zqLVmyRAAQ+/btM5YBEFKpVJw9e7ZI/X74Z1eYgj6bv/32mwAg/vvvP2OZg4ODGDly5GPP1bNnzwJ/Jx4nJibG5PfY09NTrF69Ot95H/1dEUKIjIwMAUBMnDjxsW0EBgYKd3d3kZycbCzbvn27AJCvv4/GIycnRzRs2FB07NjRWJacnCxUKpWYMGGCSd3Ro0cLGxsbkZ6eLoQo+eeNqKLgNACix/juu+8QHh5usv3zzz/G/R07doSzszNWr15tLEtKSkJ4eDj69etnLJPJZMZRL71ej8TEROTm5qJZs2bGy5ulQa1WG7/PGxVu3749rl27hpSUlGKf7+jRo4iPj8eIESNM5iv27NkT9erVw+bNm/MdM3z4cJPXbdu2xbVr14rV7oABA7B+/Xrk5ORg7dq1kMlkePHFF/PVE0Jg3bp16N27N4QQuHv3rnELCQlBSkpKkeOrVCoLvJz+qHXr1sHZ2Rnvv/9+vn15o4OOjo7IyMhAeHh4kdrOk5aWBgCws7MrtE7evtTUVABF/wz+8ccf8Pf3R7169UzilDd6/eiUlPbt26N+/frF6v/jPPzZzM7Oxt27d9GqVSsAMPkZOTo64tChQ7hz506ptQ0AVapUQXh4ODZu3Ijp06fD2dkZ6enpJnWysrKgVCrzHZv32c/Kyir0/DExMYiMjERoaCgcHByM5V26dCkwjg/HIykpCSkpKWjbtq1JLBwcHNC3b1/89ttvxlUgdDodVq9ejRdeeMF4Y1hJP29EFQWnARA9RosWLR57g5WVlRVefvllrFq1ChqNBkqlEuvXr4dWqzVJFABgxYoVmDNnDi5cuACtVmssL2i1gZLat28fwsLCcODAAWRmZprsS0lJMflPtCiioqIAGObHPapevXrYu3evSZlKpUK1atVMypycnJCUlFSsdvv3748PP/wQ//zzD1auXIlevXoVmMAlJCQgOTkZP/zwA3744YcCzxUfH1+kNj08PIp0M9XVq1dRt25dWFkV/s/niBEjsGbNGnTv3h0eHh7o2rUrXnvtNXTr1u2x5857j2lpaSbTKx72aEJb1M/g5cuXcf78+Xw/nzyPxqk0P5cAkJiYiGnTpuH333/P19bDf0h9/fXXCA0NhZeXF4KCgtCjRw8MHjwYNWvWfKr2FQoFOnfuDADo1asXOnXqhDZt2sDFxQW9evUCYEggH52/C8C4AsXDCeaj8n5XClp6rW7duvn+aNq0aRM+//xzREZGmrT56Dz0wYMHY/Xq1dizZw/atWuHHTt2IC4uDoMGDTLWKennjaii4Mgq0VPq378/0tLSjCOua9asQb169UxujPj111/x5ptvws/PDz/99BO2bt2K8PBwdOzY0eRGooI8+p9XnkdvWrp69So6deqEu3fvYu7cudi8eTPCw8PxwQcfAMAT2ykNpXWntLu7Ozp06IA5c+bgv//+w4ABAwqsl/ee3njjjXwj4HlbmzZtitTm4xKR4nJxcUFkZCT+/vtv9OnTBzt37kT37t0RGhr62OPy5kqeOnWq0Dp5+x4erSvKZ1Cv16NRo0aFxunhm4qA0o0HALz22mv48ccfMXz4cKxfvx7bt2833rT18Gfztddew7Vr17Bw4UJUr14ds2bNQoMGDUyuaJSG1q1bw93dHStXrjSWubu7IzY2Nt9atjExMQCA6tWrl0rbe/bsQZ8+faBSqbBo0SJs2bIF4eHhGDBgQL62Q0JC4Orqil9//RWA4d8SNzc3Y+INlPzzRlRRcGSV6Cm1a9cO7u7uWL16NZ577jn8+++/xhuX8qxduxY1a9bE+vXrTZLPsLCwJ57fycnJ5I7lPHkjOXk2btwIjUaDv//+G97e3sbyglYcKCwBfpSPjw8A4OLFi8bLxXkuXrxo3F8WBgwYgKFDh8LR0RE9evQosE61atVgZ2cHnU5n8p93QYr6np/Ez88Phw4dglarhVwuL7SeQqFA79690bt3b+j1eowYMQLff/89Pv3000LX6e3VqxdmzpyJn3/+Ge3atcu3X6fTYdWqVXBycjJJwovyGfTz88PJkyfRqVOnUotFUSUlJSEiIgLTpk3DlClTjOWXL18usL67uztGjBiBESNGID4+Hk2bNsUXX3yB7t27Ayi9n2V2drbJqG5gYCD+97//4fz58yZ/DBw6dMi4vzB5vwsFvaeLFy+avF63bh1UKhW2bdtmMu1g2bJl+Y6VyWQYMGAAli9fjq+++gobNmzAsGHD8v1hWJLPG1FFwZFVoqcklUqN6zX+8ssvyM3NzTcFIO8/lodHTQ4dOlSkpXD8/PyQkpJiMtoWExOTb0mcgtpISUkp8D9AGxubAhPgRzVr1gwuLi5YsmSJyaXKf/75B+fPnzfetVwWXnnlFYSFhWHRokWFXp6XyWR4+eWXsW7dOpw5cybf/oefUJQ3v68o7/txXn75Zdy9e9e4DufD8mL/6HJMUqkUjRs3BoACLzPnad26NTp37oxly5Zh06ZN+fZPnjwZly5dwvjx401GPovyGXzttddw+/Zt/Pjjj/nOm5WVhYyMjMe866dT0GcTQL6nUOl0unxzq11cXFC9enWTuNnY2BR5DnZGRka+KTGAIWFMSkoymebTt29fyOVyLFq0yFgmhMCSJUvg4eFhsnrEo9zd3REYGIgVK1aY9C08PBznzp0zqSuTySCRSEyujty4cSPfigx5Bg0ahKSkJLz77rtIT0/HG2+8YbK/pJ83ooqCI6tEj/HPP//gwoUL+cpbt25tMoeuX79+WLhwIcLCwtCoUaN8S9/06tUL69evx4svvoiePXvi+vXrWLJkCerXr5/vJo9H9e/fHxMmTMCLL76I0aNHIzMzE4sXL0adOnVM5sF17drVOLqS95/ajz/+CBcXF+NlzDxBQUFYvHgxPv/8c9SqVQsuLi75Rk4BQC6X46uvvsKQIUPQvn17vP7668alq3x9fY1TDMqCg4MDpk6d+sR6X375JXbu3ImWLVti2LBhqF+/PhITE3H8+HHs2LEDiYmJAAxJv6OjI5YsWQI7OzvY2NigZcuWxZ6bOXjwYPz8888YN24cDh8+jLZt2yIjIwM7duzAiBEj0LdvXwwdOhSJiYno2LEjPD09ERUVhYULFyIwMDDfZ+NRP//8Mzp16oS+fftiwIABaNu2LTQaDdavX49du3ahX79++Oijj/Id96TP4KBBg7BmzRoMHz4cO3fuRJs2baDT6XDhwgWsWbMG27Zte6oHYCQkJODzzz/PV16jRg0MHDgQ7dq1w9dffw2tVgsPDw9s374d169fN6mblpYGT09PvPLKKwgICICtrS127NiBI0eOYM6cOcZ6QUFBWL16NcaNG4fmzZvD1tYWvXv3LrBfly9fRufOndGvXz/Uq1cPUqkUR48exa+//gpfX1+TpZ08PT0xduxYzJo1C1qtFs2bN8eGDRuwZ8+eIj01bObMmejZsyeee+45vPXWW0hMTMTChQvRoEEDk9/znj17Yu7cuejWrRsGDBiA+Ph4fPfdd6hVq1aBU0CaNGmChg0bGm+Sa9q0qcn+p/m8EVUI5lqGgMiSPW7pKhSw/JFerxdeXl4CgPj888/znU+v14sZM2YIHx8foVQqRZMmTcSmTZsKXJYKjyxdJYRh+ZuGDRsKhUIh6tatK3799dcCl676+++/RePGjYVKpRK+vr7iq6++Mi61dP36dWO92NhY0bNnT2FnZycAGJdmenTpqjyrV68WTZo0EUqlUlSpUkUMHDhQ3Lp1y6ROaGiosLGxyffeC+pnQYqy/FFBS1cJIURcXJwYOXKk8PLyEnK5XLi5uYlOnTqJH374waTeX3/9JerXry+srKxMfo6Pa/vRpauEMCw7NHnyZFGjRg1je6+88oq4evWqEEKItWvXiq5duwoXFxehUCiEt7e3ePfdd0VMTMwT4yCEEGlpaWLq1KmiQYMGQq1WCzs7O9GmTRuxfPly4/JYj3rSZ1AIw/JIX331lWjQoIFQKpXCyclJBAUFiWnTpomUlBRjPQBPXD7qYXlLfxW0derUSQghxK1bt8SLL74oHB0dhYODg3j11VfFnTt3TD7vGo1GfPTRRyIgIEDY2dkJGxsbERAQIBYtWmTSXnp6uhgwYIBwdHQscFmohyUkJIh33nlH1KtXT9jY2AiFQiFq164txo4dKxISEvLV1+l0xt9VhUIhGjRoIH799dcix2LdunXC399fKJVKUb9+fbF+/foCf89/+uknUbt2baFUKkW9evXEsmXLHvu78vXXXwsAYsaMGfn2Pe3njcjSSYR45LoMERERWZQFCxbggw8+wI0bN0zmpBNVBkxWiYiILJgQAgEBAahatWqRHtFM9KzhnFUiIiILlJGRgb///hs7d+7E6dOn8ddff5m7S0RmwZFVIiIiC3Tjxg3UqFEDjo6OGDFiBL744gtzd4nILJisEhEREZHF4jqrRERERGSxmKwSERERkcWqEDdY6fV63LlzB3Z2duX+mEAiIiIiejIhBNLS0lC9enVIpaU4HlrchVl3794tevXqJdzd3QUA8eeffz7xmJ07d4omTZoIhUIh/Pz88i2o/iQ3b9587ALt3Lhx48aNGzdu3Cxju3nzZnHTy8cq9shqRkYGAgIC8NZbb+Gll156Yv3r16+jZ8+eGD58OFauXImIiAgMHToU7u7uCAkJKVKbdnZ2AICbN2/C3t6+uF0uNq1Wi+3bt6Nr166Qy+Vl3h4ZMO7mwbibB+NuHoy7eTDu5lHecU9NTYWXl5cxbystxU5Wu3fvju7duxe5/pIlS1CjRg3jc539/f2xd+9ezJs3r8jJat6lf3t7+3JLVq2trWFvb89fqnLEuJsH424ejLt5MO7mwbibh7niXtpTNst8zuqBAwfQuXNnk7KQkBCMHTu20GM0Gg00Go3xdWpqKgBD0LVabZn082F5bZRHW/QA424ejLt5MO7mwbibB+NuHuUd97Jqp8yT1djYWLi6upqUubq6IjU1FVlZWVCr1fmOmTlzJqZNm5avfPv27bC2ti6zvj4qPDy83NqiBxh382DczYNxNw/G3TwYd/Mor7hnZmaWyXktcjWASZMmYdy4ccbXeXMgunbtWm7TAMLDw9GlSxderihHjLt5MO7mwbibB+NuHoy7eZR33POuhJe2Mk9W3dzcEBcXZ1IWFxcHe3v7AkdVAUCpVEKpVOYrl8vl5fohL+/2yIBxNw/G3TwYd/Ng3M2DcTeP8op7WbVR5g8FCA4ORkREhElZeHg4goODy7ppIiIiIqrgip2spqenIzIyEpGRkQAMS1NFRkYiOjoagOES/uDBg431hw8fjmvXrmH8+PG4cOECFi1ahDVr1uCDDz4onXdARERERM+sYk8DOHr0KJ5//nnj67y5paGhoVi+fDliYmKMiSsA1KhRA5s3b8YHH3yABQsWwNPTE//73/+KvGxVebuZdhMv/fUSdDodPl/9ubm7U+mUVtwlEgnqOtVFS/eWaOneEgHVAqCQKUqhh0RERFSeip2sdujQAUKIQvcvX768wGNOnDhR3KbMQwDZumwAgFbHJTbMobTiHpkQiciESHx/6nuoZCo0dW1qTF7rOdWDTCorlXaIiIio7FjkagDm5Gbrhk19NmHnzp14/vnnYSVniMpLrja31OKuydUgMiESB2MO4lDMISRmJ2L/nf3Yf2c/AMBeYY8Wbi2MyauvvW+pL2JMRERET4+Z2CPkUjmq21aHk8wJ1W2r867FcqTVaks17jUda+Kl2i9BCIHLyZdxKOYQDsUcwtG4o0jNScWO6B3YEb0DAOBi7YJW7q0MyatbS7jauD7h7ERERFQemKw+KlcDxJ6BQ+YNIPYUYMUQlZvc3FKMuwSoUgNQ2kEikaCOUx3UcaqDQfUHIVefizN3zxiS19hDiIyPRHxmPP6++jf+vvo3AMDX3hct3VuilXsrNHVtCrVVwcusmZNKpuJoMBERPfOYiT0q9TbkP3VEBwC4aOa+VDJyoJTjLgGq1QM8mgLVmwAeQYBrQ1hZKRDoEohAl0C8G/AusnKzcCL+hHHk9dy9c7iRegM3Um9g9cXVpdWZUudi7YKB/gPxap1XYaewM3d3iIiIygST1UdJZBB27sjOzoZKpYIEHLkqLwKi9OKuywEy7wIJ5w1b5EpDuUwBuDY0JLAeQUD1plA710br6q3RunprAECKJgVHY48a5rvGHsL1lOtP+c7KRnxmPOYdm4cfTv2AV2q/gjfqvwE3Gzdzd4uIiKhUMVl9lJMPckefxvYtW9CjRw/OWS1HuVpt6cY9LQ64cxy4fQy4fdzwfVaS4eud48CR/xnqKWwNI6/VmwAeTeHgEYRO3h3RyacTACArN+uxK2CYg07osCNqB1acXYGrKVex4twKrDy/Et1qdMObDd5E3Sp1zd1FIiKiUsFklZ5ddq5A3e6GDQCEAJKu309cTxi+xkQCOenAjT2GLY+18/3pA02hdmsEWKnM8hYe50Xnpujbty/23t6LFWdX4HDsYWy6tgmbrm1CsHsw3mzwJoKrB3NeKxERVWhMVqnykEiAKjUNW6NXDGW6XODuRUPievuYYcQ17qxhCsHl7YbNUkmtIA0ehXbtJ6CdZzucvXsWK86uwLaobTgQcwAHYg6gjlMdvNngTXSr0Q1yKa8SEBFRxcNklSo3mRXg2sCwNR1kKNNmA7Gn708hOG5IZoXevP18VK4GSLgA7JsPnF0P9JyLBrW74Ov2X2NM+hj8eu5XrLu8DpeSLuHjvR9jwfEFeMP/DbxS5xXYKmzN3XsiIqIiY7JK9Ci5CvBqbtgs2YUtwJaPgORoYOUrQIMXgW5fwsPOAxNaTMDwgOH449IfWHl+JeIy4zDn2Bx8f+p7vFLnFQz0H8ibsYiIqEKQmrsDRFRC9XoAIw8BwaMAiRQ4+yfwbXPDjWN6PRyUDhjaaCi2vbwN01tPR02HmkjXpmP52eXovq47Pt7zMS4mcn02IiKybBxZJarIlLZAyBdA49eAjWMNUxc2/x8Q+RvQez7g1ggKmQIv1n4RfWsZbsZadmYZjsYdxcZrG7Hx2ka0cmsF62xr3D59G1IZ/34tL3qdHpeyLzHu5aw0466QKdDBswNqOtYspd4RUUGYrBI9C9wDgKE7gCM/ARHTgdtHge/bA8EjgQ4TAYUNpBIp2nm2QzvPdjhz9wxWnF2B7VHbcTD2IADg39P/mvlNVE4RpyPM3YVKqbTiPu/YPARWC8RLtV9CiG8IrOXWpXJeInqAySrRs0IqA1q+A/j3Av6ZAJz/G9j/DXB2A9BzDlCnq7FqQ+eGmNV+FsakjcHai2tx6vIpeHt7QyrlCF950ev1iI6OZtzLWWnGPS4zDvtu70NkQiQiEyLx5eEv0b1Gd7xc+2U0dG7IZeOISgmTVaJnjX11oN8vwMWtwJYPgZRoYNWrQP2+QLevAHt3Y1VPO0+MDBiJLbe3oEcLPgSjPGm1Wmy5y7iXt9KOe0JmAv66+hf+vPwnotOise7yOqy7vA61nWrjpVovoVfNXnBUOT59x4kqMf45T/SsqtvNcANW6/cBiQw495fhBqzDPwJ6nbl7R/RMqGZdDUMbDcWmFzdhachS9KrZC0qZEpeTLuOrI1+h4x8dMX73eBy4cwB6S1sCj6iCYLJK9CxT2ABdPwfe2QV4BAE5aYbR1v91BmJOmbt3RM8MiUSC5m7NMbPtTPz72r+Y3HIy6lWpB61ei39u/IN3wt9Bj/U98P3J7xGbEWvu7hJVKExWiSoD98bA2+FAj9mA0t6wasAPHYBtkw2PmyWiUmOvsEf/ev3xR+8/sLrXavSr2w92cjvcTr+NbyO/Rci6EIzYMQIRURHQ6rXm7i6RxeOcVaLKQioDWgwD6vUCtk4Ezm0ADnwLq7Mb0FBZH9LwfYY6VC6keh0a3rrBuJezUo273Nrw5Dsn30Kr1K9aH/Wr1sf/Nfs/7IjagXWX1+FY3DHsub0He27vQRVVFfT164suPl2gslI9XX/us1fYw8XahTd40TODySpRZWPvDry2Ari0Hdj8f5CkRMMPt4AEc3escpEB8AMY93JW6nE/uAjo+AnQcvhjk1+1lRq9/Xqjt19v3Ei5gT+v/Im/rvyFe9n3sOzsMiw7u6yUOmTgoHRAHac6Jpufox/UVupSbYeoPDBZJaqs6nQFfA9Cd3Q5rp48AD8/P8i4hFK50en1uHr1KuNezko17tEHgej9wLaPgdNrgT7fAG6NnniYr4MvPgj6AKOajMKeW3uw/vJ6nLl7BgLi6fpzX4omBSmaFByJPYIjsUeM5RJI4GPvg9pOtU2SWA9bD47CkkVjskpUmSlsoG/+Ds4neKLG8z0g4xJK5Uav1eJ81hbGvZyVatz1euDEL8D2Tx/MA289Gmg/HpA/eQRTLpWjo3dHdPTu+HT9eIRGp8G15Gu4lHTJZEvMTsSN1Bu4kXoD4VHhxvo2cpt8o7C1HGvBVmFbqv0iKqkSJavfffcdZs2ahdjYWAQEBGDhwoVo0aJFgXW1Wi1mzpyJFStW4Pbt26hbty6++uordOvW7ak6TkREZFZSKRAUCtQJAbZ8ZHgQx965hmXi+nwD+D5nlm4pZUr4V/WHf1V/k/K7WXdxKekSLiddNiawV5OvIkObgRPxJ3Ai/oRJfQ9bDzgqHUulT0IIpKSlYNXWVRY1iqu2UsPXwRc1HWqihkMN1HCoAXcbd0glvNphSYqdrK5evRrjxo3DkiVL0LJlS8yfPx8hISG4ePEiXFxc8tX/5JNP8Ouvv+LHH39EvXr1sG3bNrz44ovYv38/mjRpUipvgoiIyGzs3AwP4ji/ybA0XOJVYHlPoGko0GU6oHY0dw8BAM5qZzirndG6emtjmVavRVRKFC4lXcLFpIvGJDY+Mx6302/jdvrtUu3D7cTSPV9pOBp31OS1SqaCr4MvatjXQA1HQwJbw74GfB18oZQpzdTLyq3YyercuXMxbNgwDBkyBACwZMkSbN68GUuXLsXEiRPz1f/ll18wefJk9OjRAwDw3nvvYceOHZgzZw5+/fXXp+w+ERGRhfDvBdRoC+yYChxdChxfAVzaalgyrn4fc/euQHKpHLWcaqGWUy30QA9jeXJ2Mi4nX0ZWblaptJObm4ujR4+iWbNmsLKynBmIKZoU3Ei9gesp13E95TpupN5Ati4bFxIv4ELiBZO6EkjgYeuBGg41jCOxNR1rooZ9DT6lrIwV6xOTk5ODY8eOYdKkScYyqVSKzp0748CBAwUeo9FooFKZLsehVquxd+/eQtvRaDTQaDTG16mpqQAMUwq02rJfky6vjfJoix5g3M2DcTcPxt08yjzuMmsg5GtI/F+EbMsHkNy7AqwZBH3dntCFfAnYuT/5HBbARmaDwKqBpXY+rVaLNHkaWrm0sujHC+fqc3En/Q6up17H9dTrxjm+11OuI02bhlvpt3Ar/Rb23N5jcpyj0hFVVFUggeVMcQAM0y9q5NRAF22XcmmvrH6vJEKIIt9+eOfOHXh4eGD//v0IDg42lo8fPx67d+/GoUOH8h0zYMAAnDx5Ehs2bICfnx8iIiLQt29f6HQ6k4T0YVOnTsW0adPyla9atQrW1tZF7S4REZHZSPU5qBP7N2rHbYYUOmilapz16I+oqu0BzomsUIQQyBAZiNfF467+LhJ0CUjQJyBBl4AUkWLu7j1WS0VL9LbuXS5tZWZmYsCAAUhJSYG9vX2pnbfMx+IXLFiAYcOGoV69epBIJPDz88OQIUOwdOnSQo+ZNGkSxo0bZ3ydmpoKLy8vdO3atVTffGG0Wi3Cw8PRpUsXi/4L8FnDuJsH424ejLt5lH/cX4Au/hyweSzkd44j8OYyNJZcgK7HXKBq7XJo3zI8y5/3rNws3Ei9gbScNHN3JR+dTodLxy+VW9zzroSXtmIlq87OzpDJZIiLizMpj4uLg5ubW4HHVKtWDRs2bEB2djbu3buH6tWrY+LEiahZs2ah7SiVSiiV+Scxy+Xycv2Ql3d7ZMC4mwfjbh6Mu3mUa9w9AoChO4DDPwARn0EafQDSHzsA7T8CWo8BrBTl0w8L8Cx+3uVyORqrG5u7GwXSarVIOplUbnEvqzaKdR1CoVAgKCgIERERxjK9Xo+IiAiTaQEFUalU8PDwQG5uLtatW4e+ffuWrMdEREQVjVQGtHoPGHEAqNUZ0GmAfz83rM1665i5e0dk0Yo9DWDcuHEIDQ1Fs2bN0KJFC8yfPx8ZGRnG1QEGDx4MDw8PzJw5EwBw6NAh3L59G4GBgbh9+zamTp0KvV6P8ePHl+47ISIisnROPsDAtYYnXm2dAMSfBf7XCQjoD9i6lk4baifDk7TcAwAb59I5J5EZFTtZ7devHxISEjBlyhTExsYiMDAQW7duhaur4ZcsOjoa0oceYZednY1PPvkE165dg62tLXr06IFffvkFjo6OpfYmiIiIKgyJBGj8KuDX0fCo1lO/Ayd/K5u27D0At8aAe2ND8urWGHDwNPSBqIIo0Q1Wo0aNwqhRowrct2vXLpPX7du3x7lz50rSDBER0bPLpirw0vdA4ADgSjhQ9MV5CicEkHYHiDlleDhB6m3DdumfB3XUTvcT2IAHCWxVP8NUBSILZDkr8xIREVVGNdsbttKWnQrEnTEkrrGngJiTQMIFICsJuL7bsOWR2wBuDU1HYav5V6qbv8hyMVklIiJ6FqnsAZ/Whi2PNhtIOG+awMaeAbQZwM1Dhi2PVG44RymwAtAtJwdWFy0s+bVxAVoMBQLfAOSqJ9cns2CySkREVFnIVUD1JoYtj14H3LtiSGBjIu8nsaeA7GQg816pNCsBoASA3FI5XenJvAds/j9g11dA8Aig2dullqBT6WGySkREVJlJZUC1uoat8auGMiEMc1016aXShDY3F3v++w9t27WD3MpSUg8BXP8P2L8QSLkJ7JgK7JkHNH8baDUCsK1m7g7SfZbyiSEiIiJLIZEYVg0oLVot0tRXDQmxJT0UwMUfaPYWcPoPYO984O5FYO9c4OAioMkgoPX7huXGyKz4cGIiIiKqvGRyw4oMIw4C/VYCHkFAbjZw5EfgmybA+neAOK5qZE5MVomIiIikUsC/FzA0AgjdCNR8HhA64NRqYHEwsKo/cPOwuXtZKTFZJSIiIsojkQA12gGDNwDDdgL1+wKQGNaq/akLsKwncHlH6ayLS0XCZJWIiIioIB5Ngdd+BkYdAZq8YVjOK2ovsPJl4Pt2wJn1htUUqEzxBisiIiKix3GuDfT9DujwMXDgO+DYMsMSX2uHAFVqAm3GAP59AImFjQHm5kKm15i7F0+NySoRERFRUTh4AN1mAO0+BA59Dxz+Hki8BmwcY9gsjBxAA+dOAF40d1eeioX9CUBERERk4ayrAM9PAsaeAUJmAPaluMwX5cORVSIiIqKSUNoCwSMNDxHQW9rjuQCtVotTW7ehoqfSTFaJiIiInoZEYliv1dLoYXnzaEug4r8DIiIiInpmMVklIiIiIovFZJWIiIiILBaTVSIiIiKyWExWiYiIiMhiMVklIiIiIovFZJWIiIiILBaTVSIiIiKyWCVKVr/77jv4+vpCpVKhZcuWOHz48GPrz58/H3Xr1oVarYaXlxc++OADZGdnl6jDRERERFR5FDtZXb16NcaNG4ewsDAcP34cAQEBCAkJQXx8fIH1V61ahYkTJyIsLAznz5/HTz/9hNWrV+Pjjz9+6s4TERER0bOt2Mnq3LlzMWzYMAwZMgT169fHkiVLYG1tjaVLlxZYf//+/WjTpg0GDBgAX19fdO3aFa+//voTR2OJiIiIiKyKUzknJwfHjh3DpEmTjGVSqRSdO3fGgQMHCjymdevW+PXXX3H48GG0aNEC165dw5YtWzBo0KBC29FoNNBoNMbXqampAACtVgutVlucLpdIXhvl0RY9wLibB+NuHoy7eTDu5sG4m0d5x72s2pEIIURRK9+5cwceHh7Yv38/goODjeXjx4/H7t27cejQoQKP++abb/Dhhx9CCIHc3FwMHz4cixcvLrSdqVOnYtq0afnKV61aBWtr66J2l4iIiIjKSWZmJgYMGICUlBTY29uX2nmLNbJaErt27cKMGTOwaNEitGzZEleuXMGYMWPw2Wef4dNPPy3wmEmTJmHcuHHG16mpqfDy8kLXrl1L9c0XRqvVIjw8HF26dIFcLi/z9siAcTcPxt08GHfzYNzNg3E3j/KOe96V8NJWrGTV2dkZMpkMcXFxJuVxcXFwc3Mr8JhPP/0UgwYNwtChQwEAjRo1QkZGBt555x1MnjwZUmn+abNKpRJKpTJfuVwuL9cPeXm3RwaMu3kw7ubBuJsH424ejLt5lFfcy6qNYt1gpVAoEBQUhIiICGOZXq9HRESEybSAh2VmZuZLSGUyGQCgGDMQiIiIiKgSKvY0gHHjxiE0NBTNmjVDixYtMH/+fGRkZGDIkCEAgMGDB8PDwwMzZ84EAPTu3Rtz585FkyZNjNMAPv30U/Tu3duYtBIRERERFaTYyWq/fv2QkJCAKVOmIDY2FoGBgdi6dStcXV0BANHR0SYjqZ988gkkEgk++eQT3L59G9WqVUPv3r3xxRdflN67ICIiIqJnUolusBo1ahRGjRpV4L5du3aZNmBlhbCwMISFhZWkKSIiIiKqxEr0uFUiIiIiovLAZJWIiIiILBaTVSIiIiKyWExWiYiIiMhiMVklIiIiIovFZJWIiIiILBaTVSIiIiKyWExWiYiIiMhiMVklIiIiIovFZJWIiIiILBaTVSIiIiKyWExWiYiIiMhiMVklIiIiIovFZJWIiIiILBaTVSIiIiKyWExWiYiIiMhiMVklIiIiIovFZJWIiIiILBaTVSIiIiKyWExWiYiIiMhiMVklIiIiIovFZJWIiIiILFaJktXvvvsOvr6+UKlUaNmyJQ4fPlxo3Q4dOkAikeTbevbsWeJOExEREVHlUOxkdfXq1Rg3bhzCwsJw/PhxBAQEICQkBPHx8QXWX79+PWJiYozbmTNnIJPJ8Oqrrz5154mIiIjo2VbsZHXu3LkYNmwYhgwZgvr162PJkiWwtrbG0qVLC6xfpUoVuLm5Gbfw8HBYW1szWSUiIiKiJ7IqTuWcnBwcO3YMkyZNMpZJpVJ07twZBw4cKNI5fvrpJ/Tv3x82NjaF1tFoNNBoNMbXqampAACtVgutVlucLpdIXhvl0RY9wLibB+NuHoy7eTDu5sG4m0d5x72s2pEIIURRK9+5cwceHh7Yv38/goODjeXjx4/H7t27cejQoccef/jwYbRs2RKHDh1CixYtCq03depUTJs2LV/5qlWrYG1tXdTuEhEREVE5yczMxIABA5CSkgJ7e/tSO2+xRlaf1k8//YRGjRo9NlEFgEmTJmHcuHHG16mpqfDy8kLXrl1L9c0XRqvVIjw8HF26dIFcLi/z9siAcTcPxt08GHfzYNzNg3E3j/KOe96V8NJWrGTV2dkZMpkMcXFxJuVxcXFwc3N77LEZGRn4/fffMX369Ce2o1QqoVQq85XL5fJy/ZCXd3tkwLibB+NuHoy7eTDu5sG4m0d5xb2s2ijWDVYKhQJBQUGIiIgwlun1ekRERJhMCyjIH3/8AY1GgzfeeKNkPSUiIiKiSqfY0wDGjRuH0NBQNGvWDC1atMD8+fORkZGBIUOGAAAGDx4MDw8PzJw50+S4n376CS+88AKqVq1aOj0nIiIiomdesZPVfv36ISEhAVOmTEFsbCwCAwOxdetWuLq6AgCio6MhlZoO2F68eBF79+7F9u3bS6fXRERERFQplOgGq1GjRmHUqFEF7tu1a1e+srp166IYiw4QEREREQEo4eNWiYiIiIjKA5NVIiIiIrJYTFaJiIiIyGIxWSUiIiIii8VklYiIiIgsFpNVIiIiIrJYTFaJiIiIyGIxWSUiIiIii8VklYiIiIgsFpNVIiIiIrJYTFaJiIiIyGIxWSUiIiIii8VklYiIiIgsFpNVIiIiIrJYTFaJiIiIyGIxWSUiIiIii2Vl7g4QERERUckIIZCmyUVKphYpWVokZ2qRnJWDlCwtEtOykZEsQQ9zd/IpMVklIiIiegq5Oj0ycnSAAPRCQC8EBAzfG8rwoEwvYCgWD8qFIenMe51+P/lMzjIkoCmZOcbvk++Xp2ZpkZxpSEr1ovC+tXWVlFcYygyTVSIiIqJiuJuuwfGoJByPTsbxqCScup2MbK3erH1SyaVwVCvgoJbDwVoOB7Uc9ioZrFNvmrVfpYHJKhEREVEhdHqBi7FpOBadhBNRSTgWnYSoe5lFOlYiAaQSCaQSQAKJ8fXDXyUApFKJ4TUAiUQCW6XsftKpgKPakHg63k9ADd8rjGWOajns1XKo5LJ87Wu1WmzZEl26ATEDJqtERERE96VkanH8ZtL9kdMkREYnGy7xP6K2iy2CfJzQ1NsJTX2c4FVFDZlEAkleciqp+JffLQWTVSIiIqqU9HqBa3fTcSwqCcejknEsOglX4tPz1bNVWiHQyxFNfZzQ1NsRTbyc4GAtN0OPK6cSJavfffcdZs2ahdjYWAQEBGDhwoVo0aJFofWTk5MxefJkrF+/HomJifDx8cH8+fPRo0dFvz+NiIiILFG2Voe76RokpN3fHv7+/utrCRlIydLmO7aGsw2aeDsaR07ruNpBJuVIqbkUO1ldvXo1xo0bhyVLlqBly5aYP38+QkJCcPHiRbi4uOSrn5OTgy5dusDFxQVr166Fh4cHoqKi4OjoWBr9JyIiokpCpxdIzMjB3XQN4tPyJ58JadnG16nZuUU6p0ouRYCnYdQ0yNsJTbwdUdVWWcbvhIqj2Mnq3LlzMWzYMAwZMgQAsGTJEmzevBlLly7FxIkT89VfunQpEhMTsX//fsjlhiFzX1/fx7ah0Wig0WiMr1NTUwEYJgprtfn/AipteW2UR1v0AONuHoy7eTDu5sG4m0dhcdfpBZIzc5CYocW9jBwkZuQgMTMH99INX03KMwzLN4nHLNP0KLlMAhc7JZxtlahmq4Cz3f2vtkpUs1WiuqMKdVxtIZeZPiPpWfl8lPfnvazakQhR9B97Tk4OrK2tsXbtWrzwwgvG8tDQUCQnJ+Ovv/7Kd0yPHj1QpUoVWFtb46+//kK1atUwYMAATJgwATJZ/jvXAGDq1KmYNm1avvJVq1bB2tq6qN0lIiIiM8rMBW6kSXAzA0jJkSBDC6RpJUjPBdK1hv0Cxbu8LoGAjRywlwP2cgE7heF7O7mA/SPfq2WGO/KpfGRmZmLAgAFISUmBvb19qZ23WCOrd+/ehU6ng6urq0m5q6srLly4UOAx165dw7///ouBAwdiy5YtuHLlCkaMGAGtVouwsLACj5k0aRLGjRtnfJ2amgovLy907dq1VN98YbRaLcLDw9GlSxfjaDCVPcbdPBh382DczYNxLzt6vcDVuxk4EZ2MEzdTcOJmMq4mZBTpWCdrOZysFahiI0dVGwWq3N+qmnyVo4qNYSknKxmfFl8U5f15z7sSXtrKfDUAvV4PFxcX/PDDD5DJZAgKCsLt27cxa9asQpNVpVIJpTL/fBG5XF6u/7iUd3tkwLibB+NuHoy7eTDuTy8tW4vIm8nGu+gjo5MKnCdaw9kGAR72yLp7Gy0C68PFXo2qNgpUtVWiio0CTtZMPstaeX3ey6qNYiWrzs7OkMlkiIuLMymPi4uDm5tbgce4u7tDLpebXPL39/dHbGwscnJyoFAoStBtIiIiKi9CCFy/m2FY4ik6GSeik3AxLi3f/FG1XIYALwc09XZCkI8Tmng7oYqN4v7i9DfRo5U3/0igYitWsqpQKBAUFISIiAjjnFW9Xo+IiAiMGjWqwGPatGmDVatWQa/XQyo1/OV06dIluLu7M1ElIiIyMyEEsrV6ZOTkIkOTiwyNDhk5uUjL1uJ8TBqORSXhRHQSkjLz3zzjVUWNoPuL4jf1dkI9NzuOklKpK/Y0gHHjxiE0NBTNmjVDixYtMH/+fGRkZBhXBxg8eDA8PDwwc+ZMAMB7772Hb7/9FmPGjMH777+Py5cvY8aMGRg9enTpvhMiIqJnXK5Oj0ytDlk5OmTmGL5maXOR+dDrzBwdMjS5SNfkIjMnF+kaHTIfSUTTNbnI1BjqZeTkQl+EW62VVlI09nQwPrGpibcjXOxUZf+mqdIrdrLar18/JCQkYMqUKYiNjUVgYCC2bt1qvOkqOjraOIIKAF5eXti2bRs++OADNG7cGB4eHhgzZgwmTJhQeu+CiIiogkjX5CImOQt3UrKNX++la4yJpiEZzTVJPrPuJ6g5On2Z9s1GIYO10gq2SivYKGXwqWpjHDmt724PhRVHTan8legGq1GjRhV62X/Xrl35yoKDg3Hw4MGSNEVERFRhZGt1iHkoCTV+TclCTHI27qRkIa2Ii9U/jlQCWCusoFbIoJbLYK2QQa24/1VuBVulDDZKK8OmMCSeD14bvrdVWsFaIbufmFpBLZdByqc0kQUq89UAiIiIKjK9XiA1W4vEjBwk3V+oPikjB/cychCTkoU7yfeT0ZRsJGbkFOmcdiorVHdQw91RBXcHNVzslLBRyqBWWMFabkg81QoZrOWyB0np/ddqhQxKKykkXECUKgkmq0REVGkIIZCdC0QnZiItRyDp/pORDEnoQ18ztEjMzEHS/bKizOnMY62Qwd1BheqOarg7qODmoEZ1BxXcHR98tVXyv1+iouJvCxERPXOEEIhP0+BCbBouxqbiYmw6Lsal4kp8OrK1VsCRvcU+p53SCk42CjjZKFDFWg4nGwXcHQwjo9Xvj5BWd1DDXm3FUU+iUsRklYiIKrTUbC0uxabhYlwaLsam4UJsGi7FpSG5gKWW8lgrZHCyVsDJJu/JSYoHX20UqHJ/X5X73ztaK3hzEZGZMFklIqIKQZOrw7WEDGNCejE2FZfi0nE7OavA+lKJ4elJ9dzsUdfNDnVc7eDnrELk/t14oXdXLk5PVEEwWSUiIouUlaPD7ksJ2H4uFqdvpeD63QzkFjJ51N1Bhbpudqjramf46mYHv2q2UMllJvW0Wi3OyQo8BRFZKCarRERkMVKztdh5IR5bz8Ri18UEZGl1JvvtVFao55aXkNobklNXOzhYc5SU6FnFZJWIiMzqXroGO87H4Z8zsdh35S60ugejp55OanRr4IY2tZxRz90ObvYq3rxEVMkwWSUionIXk5KFbWdisfVsLA5fTzRZGqqWiy26NXBDt4ZuaFDdnskpUSXHZJWIiMrFjbsZ2Ho2FlvPxCLyZrLJvoYe9sYEtZaLnXk6SEQWickqERGVCSEELsalYesZQ4J6ITbNuE8iAYK8ndCtoRtCGrjBq4q1GXtKRJaMySoRERVKCIFsrR4ZObnI1OiQqc1FhkaHzJyHvubokPXI60xNLk7ev4M/j0wqQXDNqghp6IaQ+q5wsVeZ8Z0RUUXBZJWIqBLL0OTizO0UnLqVgpO3knH9bgYyc3TI0OQavubkQhTjUaOPUlhJ0a62M0IauKGzvyucbBSl13mq0HQ6HbTawh/cQE9Pq9XCysoK2dnZ0Ol0Tz7gCeRyOWSy8l/7jckqEVElocnV4XxMGk7dSsapWyk4dSsZV+LTi/zce2uFDNYKK9goZVDLZbBRWsFaIYONwgrWStmD7+/Xqe6oRrs61WCr5H819IAQArGxsUhOTjZ3V555Qgi4ubnh5s2bpXajoqOjI9zc3Mr1xkf+C0JE9AzS6QUux6fh1E3DiOmpWym4EJtqsixUHncHFRp7OqCxpyP83e1gr5IbE05rhSEhVctlkEp5Vz49vbxE1cXFBdbW1lztoQzp9Xqkp6fD1tYWUunTPS5YCIHMzEzEx8cDANzd3Uuji0XCZJWIqIITQiDqXqYxKT11KxlnbqfmW1AfAKrYKIyJaWMPBzT2coCLHeeOUvnQ6XTGRLVq1arm7s4zT6/XIycnByqV6qmTVQBQq9UAgPj4eLi4uJTblAAmq0REFUC6Jhe3k7JwKykTt0y+ZuHGvQykZefmO8ZWaYWGHvYI8HQ0JKeeDvB0UnMki8wmN9fwObW25uoPFVXez06r1TJZJSKqTB6XjN5KykRS5uNvRFFYSVHf3R4B90dNA7wcUNPZlpfuyaKI+3fr8Q+misscPzsmq0RE5eheugZHo5Jw9Po9HL4oxY9RB3E7OeuJySgAOFrL4emkhqejteGrkxqeTtbwrKKGXzVbyGVPf5mPiMjSMFklIiojQgjcuJeJozcScfRGEo5EJeJaQsZDNaQAUo2vHpeMejiqYaeSl/t7ICIyNyarRESlRKvT49ydVBy5n5wejUrE3fScfPVqu9iiqbcDcu9Go0ubIPhWs2MySmThOnTogMDAQMyfP7/M25o6dSo2bNiAyMjIMm+rImCySkRUQmnZWpyITsbRG4k4ciMJkTeT892Br5BJ0djTAc18q6C5rxOCfJzgaK2AVqvFli1R6FTPBXI5k1QiosKUKFn97rvvMGvWLMTGxiIgIAALFy5EixYtCqy7fPlyDBkyxKRMqVQiOzu7JE0TEZlFZk4u4lI1OHM7BceiknDkRiLOx6TmW1DfQS1HMx8nBPk6oblvFTTycIBKXv5PfCEielYUO1ldvXo1xo0bhyVLlqBly5aYP38+QkJCcPHiRbi4uBR4jL29PS5evGh8zbsAicgSCCGQnKlFQroG8akaxKdlIz7N9PuENA3iU7ORkVPwowq9qqjRzKcKmt1PTmtV4x34RMUhhChwTeCyppbLSpyPJCUlYcyYMdi4cSM0Gg3at2+Pb775BrVr1zbW+fHHHzF9+nTcu3cPISEhaNu2LaZPn16iJ3fp9Xp8/vnn+OGHH5CQkAB/f398+eWX6NatGwAgJycH48aNw7p165CUlARXV1cMHz4cEyZMgBAC06ZNw7JlyxAXF4eqVavilVdewTfffFOi924OxU5W586di2HDhhlHS5csWYLNmzdj6dKlmDhxYoHHSCQSuLm5FbkNjUYDjUZjfJ2aargBQavVlstzhPPa4DOLyxfjbh7PctxzdXqcvpOKy3HphsQzXYOEtBxjEpqQrinwiU6FsVbI4FvVGkE+Tmjm7YimPo5wszddUF+ny0VRHsH9LMfdkjHu5pEX79zcXAghoNfrodfrARiuWjScGl7ufToztQusFcVLg/L6HhoaiitXrmDDhg2wt7fHxIkT0aNHD5w5cwZyuRz79u3D8OHD8eWXX6J3796IiIjAlClTAMD4vp/UzsN158+fjzlz5mDx4sVo0qQJli1bhj59+uD06dOoXbs2FixYgL///hu///47vL29cfPmTdy8eRNCCPz999+YP38+Vq1ahQYNGiA2NhYnT54sUj8KotfrIYQocJ3Vsvq9KtZPKScnB8eOHcOkSZOMZVKpFJ07d8aBAwcKPS49PR0+Pj7Q6/Vo2rQpZsyYgQYNGhRaf+bMmZg2bVq+8u3bt5frQsLh4eX/y0OMu7k8C3HXC+BOJnApRYLLKRJcTZVAo3/yyIm1lYC9HLBX5H0F7OUCDve/2isMZSpZLgANgCTgJnD85tP3+VmIe0XEuJvH/v374ebmhvT0dOTkGG4+zCrkqkVZS0tNQ66i6FN0cnNzkZOTgxMnTmDjxo3YunUrAgICAACLFy9Gw4YN8dtvv+GFF17AvHnz0LlzZwwbNgwAMHDgQOzevRvbtm0zDsA9jkajgU6nM9adPXs2Ro8ejR49egAAPv74Y0RERGDWrFmYPXs2rly5gho1aqBx48aQSCRwcnJC48aNkZaWhlu3bsHFxQUtWrSAXC6Ho6Mj6tWrV6R+FCQnJwdZWVn477//jA95yJOZmVmicz5JsZLVu3fvQqfTwdXV1aTc1dUVFy5cKPCYunXrYunSpWjcuDFSUlIwe/ZstG7dGmfPnoWnp2eBx0yaNAnjxo0zvk5NTYWXlxe6du0Ke3v74nS5RLRaLcLDw9GlSxfe+FCOGHfzqMhxF0LgSkIGDl5LxIFrhpuckrNM/7J3UFshwNMBrvYqVLNVwsVOgWp2SlSzU8LFTglnWyWUVuW/PmlFjntFxribR17cW7dujZiYGNja2kKlMlyVsBMCZ6Z2Kfc+FXcagJWVFRQKBW7evAkrKyt07NjROLJob2+PunXrIioqCvb29rh+/TpeeOEFk5ylTZs22L59e5HyGKVSCZlMBnt7e6SmpiImJgYdO3Y0ObZt27Y4deoU7O3tMWzYMISEhKBly5YICQlBz5490bVrVwgh0LdvX3z//fdo2rQpQkJC0L17d/Tu3RtWViW7xz47OxtqtRrt2rUz/gzzlDQBfpIyXw0gODgYwcHBxtetW7eGv78/vv/+e3z22WcFHqNUKqFUKvOVy+Xycv3HpbzbIwPG3TwqQtyFEIhOzMT+q/ew/+o9HLh6D3fTNSZ1bBQytKxZFa39qqJVzaqo725v0XNIK0Lcn0WMu3lYWVlBIpFAKpWaPKvetpwe2/m08voOIN97eHT/w9/nvc47rijtPNrGo+09XKdZs2a4fv06/vnnH+zYsQP9+/dH586dsWbNGnh6euL8+fP4999/ER4ejlGjRmHOnDnYvXt3iX4HpFIpJBJJgb9DZfU7Vaxk1dnZGTKZDHFxcSblcXFxRZ6TKpfL0aRJE1y5cqU4TRNRJRWTkoX9V+7hwDVDcno7Octkv9JKiua+VRDsVxXBflXRyMOBT3IiojLj7++P3NxcHDp0CK1btwYA3Lt3DxcvXkT9+vUBGK4qHzlyxOS4R18Xlb29PapXr459+/ahffv2xvJ9+/aZrMRkb2+Pfv36oV+/fnjllVfQrVs3JCYmwsrKCmq1Gr1790bv3r0xcuRI1KtXD6dPn0bTpk1L1KfyVqxkVaFQICgoCBEREXjhhRcAGCbaRkREYNSoUUU6h06nw+nTp43zLoiIAMOC+reTsnDjXgaiEzNxPiYNB6/dw/W7GSb15DIJmng5GZPTJt6OUFpVjFEZIqr4ateujb59+2LYsGH4/vvvYWdnh4kTJ8LDwwN9+/YFALz//vto164d5s6di969e+Pff//FP//8U+LVBz766COEhYXBz88PgYGBWLZsGSIjI7Fy5UoAhpvf3d3d0aRJE0ilUvzxxx9wc3ODo6MjfvjhB8jlcgQHB8Pa2hq//vor1Go1fHx8Si0mZa3Y0wDGjRuH0NBQNGvWDC1atMD8+fORkZFhXB1g8ODB8PDwwMyZMwEA06dPR6tWrVCrVi0kJydj1qxZiIqKwtChQ0v3nRCRxcvQ5CI6MRNR9zIQdS8TUYmZiL6XiajEDNxJzobu0UVLAUglQCNPR7T2M1zaD/JxKvYdvEREpWnZsmUYM2YMevXqhZycHLRr1w5btmwxXgZv06YNlixZgmnTpuGTTz5BSEgIPvjgA3z77bclam/06NFISUnB//3f/yE+Ph7169fH33//bVwqy87ODl9//TUuX74MmUyG5s2bY8uWLZBKpXBwcMDChQvx4YcfQqfToVGjRti4cSOqVq1aavEoa8X+F79fv35ISEjAlClTEBsbi8DAQGzdutV401V0dLTJnIqkpCQMGzYMsbGxcHJyQlBQEPbv328cKieiZ4cQAokZOcYk9Ma9jPvJaCai7mXmm1/6KJVcCp8qNvCuao2azjZoUaMKmteoAns+hpSIzGzXrl3G752cnPDzzz8/tv6wYcOMqwHkva5Vq1aR2po6dSqmTp1qfC2VShEWFoawsLAitZVHr9ejZ8+eeP3114s0V9ZSlWh4YtSoUYVe9n/4hwkA8+bNw7x580rSDBFZCL1eIDlLi/i07PuL5GuMC+kbvmYbX6drch97LkdrOXyqWMOnqg18qlrD+6HvXeyUfGgIET0TZs+ejS5dusDGxgb//PMPVqxYgUWLFpm7WxUSr6URVXL3MnJwIw0IPxePe1m5hsXy85LSvMXz0zTILeASfWHcHVT3k9AHiWjeiKmDmqOkRPTsO3z4ML7++mukpaWhZs2a+Oabb4xTIBs0aICoqKgCj/v+++8xcODA8uyqxWOySlQJCSFw8Foilu27jh3n46AXVsCZyCceV8VGYVir1F6JarZKVLv/1SVvDVN7JTwc1VDJecMTEVVua9asKXTfli1bCn3a06Nr2ROTVaJKJVurw1+Rt7Fs3w1ciE0zljspBLxdHOFir4KLccF8lXHhfBd7JaraKKEww+L5RETPmop0J74lYLJKVAnEpmTjl4M3sOpQNJIyDX/Nq+UyvNTUAwNbeOLy0f/Qo0dLLpJOREQWh8kq0TNKCIHj0clYtu86tp6JNc459XBUY3CwD/o394aDtRxarRaXzdxXIiKiwjBZJXrG5OTqseV0DJbtu46Tt1KM5S1qVMFbbXzR2d8VVnzCExERVRBMVomeEQlpGqw6FI1fD0UhIc2wnqlCJkWfwOp4s7UvGno4mLmHRERExcdklaiCO3M7Bcv23cDGk3eQo9MDAFzslBjUygevt/SGs63SzD0kIiIqOSarRBVQrk6P7efisGzfdRy5kWQsD/ByxFttfNG9oTvv3CciKkVCCLz77rtYu3YtkpKScOLECQQGBpq7W5UCk1UiCyeEQGxqNk7eTMGpW8k4dcvwNTXb8KQoK6kEPRq5Y0gbXzTxdjJzb4mInk1bt27F8uXLsWvXLtSsWROXLl1C7969cezYMcTExODPP//ECy+8YO5uPpOYrBJZmMSMHJy8lYxTecnp7RTjHNSHVbFRYEALb7zRygduDioz9JSIqPK4evUq3N3d0bp1awDAiRMnEBAQgLfeegsvvfSSmXv3eDk5OVCpKu7/E0xWicwoLVuL07dTcOpWCk7fSsHJW8m4lZSVr55MKkEdVzs09nBAYy8HBHg6oq6bHeS8q5+IKjohAG1m+bcrtwYkkiJVffPNN7FixQoAgEQigY+PD27cuIHu3buXuPlFixZh3rx5uHnzJhwcHNC2bVusXbsWAKDX6zF79mz88MMPuHnzJlxdXfHuu+9i8uTJAIDTp09jzJgxOHDgAKytrfHyyy9j7ty5sLW1NfY3OTkZzZo1w3fffQeVSoXr16/j5s2b+L//+z9s374dUqkUbdu2xYIFC+Dr61vi91EemKwSlZOcXD1O307B6fuX8k/eSsa1uxkQIn/dms42aOzpgMaejgjwckB9dweoFXyEKRE9g7SZwIzq5d/ux3cAhU2Rqi5YsAB+fn744YcfcOTIEchkT/fv8dGjRzF69Gj88ssvaN26NRITE7Fnzx7j/kmTJuHHH3/EvHnz8NxzzyEmJgYXLlwAAGRkZCAkJATBwcE4cuQI4uPjMXToUIwaNQrLly83niMiIgJ2dnZYv349bG1todVqjcft2bMHVlZW+Pzzz9GtWzecOnUKCoXiqd5TWWKySlTGbidnYdWhKKw+chN303Py7fdwVD9ITD0d0NDTAfYqPkmKiMhSODg4wM7ODjKZDG5ubk99vujoaNjY2KBXr16ws7ODj48PmjRpAgBIS0vDggUL8O233yI0NBQA4Ofnh+eeew4AsGrVKmRnZ+Pnn3+GjY0h2f7222/Ru3dvfPXVV3B1dQUA2NjY4Mcff0R2djbs7e2xatUq6PV6/O9//4Pk/ojysmXL4OjoiF27dqFr165P/b7KCpNVojKg1wv8dzkBvx6Mwr8X4nH/4VFwspajibcTGnsaLuU38nTg0lJEVLnJrQ2jnOZo10y6dOkCHx8f1KxZE926dUO3bt3w4osvwtraGufPn4dGo0GnTp0KPPb8+fMICAgwJqoA0KZNG+j1ely8eNGYrDZq1AgKhQLZ2dkAgJMnT+LKlSuws7MzOV92djauXr1aRu+0dDBZJSpFiRk5+OPoTaw8FI3oxAdzsNrUqopBrXzQyd+V80yJiB4mkRT5cvyzws7ODsePH8euXbuwfft2TJkyBVOnTsWRI0egVqtLpY2Hk1kASE9PR1BQEFauXJmvbrVq1UqlzbLCZJXoKQkhcOJmMn49EIVNp2OQk2tYmN9OZYVXg7wwsJU3/KrZmrmXRERkSaysrNC5c2d07twZYWFhcHR0xL///osePXpArVYjIiICQ4cOzXecv78/li9fjoyMDGNCum/fPkilUtStW7fQ9po2bYrVq1fDxcUF9vb2Zfa+ygKTVaISyszJxV+Rd/DLgSici0k1ljfycMCgVj7oHVCdN0URET2j0tPTceXKFePr69evIzIyElWqVIG3t/djj920aROuXbuGdu3awcnJCVu2bIFer0fdunWhUqkwYcIEjB8/HgqFAm3atEFCQgLOnj2Lt99+GwMHDkRYWBhCQ0MxdepUJCQk4P3338egQYOMUwAKMnDgQMyaNQt9+/bF9OnT4enpiaioKKxfvx7jx4+Hp6dnqcWmtDFZJSqmK/Fp+PVgNNYdu4U0jWFhfqWVFL0DqmNQKx8EeDmat4NERFTmjh49iueff974ety4cQCA0NBQk7vyC+Lo6Ij169dj6tSpyM7ORu3atfHbb7+hQYMGAIBPP/0UVlZWmDJlCu7cuQN3d3cMHz4cAGBtbY1t27ZhzJgxaN68ucnSVY9jbW2N//77DxMmTMBLL72EtLQ0eHh4oFOnThY/0spklagIcnL12H4uFr8ejMLBa4nGct+q1nijlQ9eCfKEo7XlLvtBRERPZ+zYsRg7dqzxdYcOHSAKWnuwCJ577jns2rWr0P1SqRSTJ082rqv6qEaNGuHff/8t9Pi8ZFmv15uUu7m5GdeLrUiYrBI9xpX4dPwVeRu/H7lpfIqUVAJ09nfFoGAftPFzhlRatEWliYiIqPhKdFvyd999B19fX6hUKrRs2RKHDx8u0nG///47JBIJn51LFkuvFzgenYQv/7mAjnN2ofPc3Vj47xUkpGlQzU6J0R1rYe+EjvhhcDO0rV2NiSoREeWzZ88e2NraFrpR8RR7ZHX16tUYN24clixZgpYtW2L+/PkICQnBxYsX4eLiUuhxN27cwIcffoi2bds+VYeJSpsmV4f9V+8h/Fwcws/FGUdQAUAukyDYzxmvNfNE1/puUFhx2SkiInq8Zs2aITIy0tzdeGYUO1mdO3cuhg0bhiFDhgAAlixZgs2bN2Pp0qWYOHFigcfodDoMHDgQ06ZNw549e5CcnPxUnSZ6WqnZWuy8EI/t5+Kw+2IC0u/fKAUAtkorPF/PBV3ru6J93Wp8mhQRERWLWq1GrVq1zN2NZ0axktWcnBwcO3YMkyZNMpZJpVJ07twZBw4cKPS46dOnw8XFBW+//bbJs28Lo9FooNE8GN1KTTUsC6TVaqHVaovT5RLJa6M82qIHyjruMSnZ+PdCPMLPJ+DwjURodQ8mxrvYKdGpXjV08XdBixpVoHxoBPVZ/xzw824ejLt5MO7mkRfv3NxcCCGg1+vz3fxDpS/vBrC8mJcGvV4PIQS0Wi1kMtPlGcvq96pYyerdu3eh0+nyrePl6uqKCxcuFHjM3r178dNPPxVrOHzmzJmYNm1avvLt27fD2rr8Ho8WHh5ebm3RA6UVdyGA2CzgdKIEpxOliM4wnV/qqhZoVEWgsZMeXra5kEoykHb5BiIul0rzFQ4/7+bBuJsH424e+/fvh5ubG9LT05GTk2Pu7lQaaWlppXaunJwcZGVl4b///kNubq7JvszMzEKOejpluhpAWloaBg0ahB9//BHOzs5FPm7SpEnG9coAw8iql5cXunbtWi5rgWm1WoSHh6NLly6Qy3kJuLyURtzvZeQgMjoZh28kIeJiAqIeeuSpRAI08XJEZ/9q6FzPBTWcK9fj/QrDz7t5MO7mwbibR17cW7dujZiYGNja2kKlUpm7W888IQTS0tJgZ2cHiaR0bgjOzs6GWq1Gu3bt8v0M866El7ZiJavOzs6QyWSIi4szKY+Li4Obm1u++levXsWNGzfQu3dvY1neMLSVlRUuXrwIPz+/fMcplUoolcp85XK5vFz/cSnv9sigqHHX6wWuJKTjWFSScbt+N8OkjsJKiudqOaNLfVd08neBix3/cSwMP+/mwbibB+NuHlZWVpBIJJBKpZBKecNqWcvLufJiXhqkUikkEkmBv0Nl9TtVrGRVoVAgKCgIERERxuWn9Ho9IiIiMGrUqHz169Wrh9OnT5uUffLJJ0hLS8OCBQvg5eVV8p5TpZOhycXJm8k4ej8xPR6dhLTs3Hz1arvYoqm3E9rXrYZ2darBVsnlhImIiCqqYv8vPm7cOISGhqJZs2Zo0aIF5s+fj4yMDOPqAIMHD4aHhwdmzpwJlUqFhg0bmhzv6OgIAPnKiR4mhMCtpCwcj34wano+JhX6Rx4WopbLEOjliCAfJwT5OqGplxMcrDlaQkREFd+uXbvw/PPPIykpyZg/lUbdiqbYyWq/fv2QkJCAKVOmIDY2FoGBgdi6davxpqvo6GgO7VOx5er0OHkrBTvvSPDP7ydx4mYy4lI1+ep5OKrR1McJQd6OaOZbBfXc7GAl4+eNiIiePXlzfB0cHEq1bkVTouujo0aNKvCyP4DHPusWePC8Wqrc9HqBczGpOHjtHvZfvYfD1xPvr3UqA2CYE20llaBBdXs09XFCM58qaOrjCHcHtVn7TUREVBQ5OTlQKBRPdQ6FQlHgPUFPW7ei4ZAUlQshBC7GpmH5vut495ejaPJZOHot3IvPN5/Hvxfika7Jhb3KCg2c9PiwS22sfqcVTk8NwV+jnkNY7wbo2didiSoR0TNICIFMbWa5b3lrkBZVhw4djIN1Dg4OcHZ2xqeffmo8j6+vLz777DMMHjwY9vb2eOeddwAYlvBs27Yt1Go1vLy8MHr0aGRkPLgZWKPRYMKECfDy8oJSqUStWrXw008/ATAMAEokEuPDlKKiotC7d284OTnBxsYGDRo0wJYtWwqsCwDr1q1DcHAw1Go1fH19MWfOHJP35OvrixkzZuCtt96CnZ0dvL298cMPPxQrLuWBd55QmRBC4PrdDOy/eg8Hrt3DoWv3cDfddE09G4UMLWpUQbBfVbT2c0YtZzW2bf0HPdrV4F26RESVRFZuFlqualnu7R4acAjW8uKt3b5ixQq8/fbbOHz4MI4ePYp33nkH3t7eGDZsGABg9uzZmDJlCsLCwgAYVkXq1q0bPv/8cyxduhQJCQnGhHfZsmUADPf6HDhwAN988w0CAgJw/fp13L17t8D2R44ciZycHPz333+wsbHBuXPnYGtrW2DdY8eOoX///pg4cSIGDRqEgwcPYsSIEahatSrefPNNY705c+bgs88+w8cff4y1a9fivffeQ/v27VG3bt1ixaYsMVmlUnMzMRMH7ien+6/ezTfnVCWXopmPITkN9quKRh4OkMsqz5OiiIioYvPy8sK8efMgkUhQt25dnD59GvPmzTMmqx07dsT//d//GesPHToUAwcOxNixYwEAtWvXxjfffIP27dtj8eLFiI6Oxpo1axAeHo7OnTsDAGrWrFlo+9HR0Xj55ZfRqFGjJ9adO3cuOnbsiI8++gj29vaoV68ezp07h1mzZpkkqz169MCIESMAABMmTMC8efOwc+dOJqtU8QkhcONeJo5FJeHQNUOCeispy6SOQiZFE29H48hpgJcDlFayQs5IRESVkdpKjUMDDpml3eJq1aqVyeL6wcHBmDNnDnQ6HQCgWbNmJvVPnjyJU6dOYeXKlcayvEefXr9+HadPn4ZMJkP79u2L1P7o0aPx3nvvYfv27ejcuTNefvllNG7cuMC658+fR58+fUzK2rRpg/nz50On0xkflfrw8RKJBG5uboiPjy9Sf8oLk1UqkmytDqdupRiXkToenYTEDNPL+lZSCRp7OqC1nzOC/aoiyMcJKjmTUyIiKpxEIin25XhLZWNj+mTE9PR0vPvuuxg9enS+ut7e3rhy5Uqxzj906FCEhIRg8+bN2L59O2bOnIk5c+bg/fffL3GfH512J5FIjA8TsBRMVqlAsSnZD54MFZ2Ec3dSoNWZTkZXyKRo5OmAZj5OCPariua+VWDDBfiJiOgZdeiQ6QjwwYMHUbt2beMo5aOaNm2Kc+fOoVatWgXub9SoEfR6PXbv3m2cBvAkXl5eGD58OIYPH45Jkybhxx9/LDBZ9ff3x759+4xTEABg3759qFOnTqH9tVTMLAi5Oj3Ox6ThWFQijkUn43hUEm4nZ+WrV81OiSBvJwT5OKGpjxMaetjzsj4REVUa0dHRGDduHN59910cP34cCxcuzHeH/cMmTJiAVq1aYdSoURg6dKjxpqjw8HB8++238PX1RWhoKN566y3jDVZRUVGIj4/Ha6+9lu98Y8eORffu3VGnTh0kJSVh586d8Pf3L7Dt//u//0Pz5s0xa9YsDBo0CIcOHcK3336LRYsWlVo8yguT1UpECIE0TS7upmlw414Gjkcl42hUIk7eTEGWVmdSVyoB6rnZG54MdX/zdFKbzNUhIiKqTAYPHoysrCy0aNECMpkMY8aMMS5RVZDGjRtj9+7dmDx5Mtq2bQshBPz8/NCvXz9jncWLF+Pjjz/GiBEjcO/ePXh7e+Pjjz8u8Hw6nQ4jR47ErVu3YG9vj27dumHevHkF1m3atCl+//13TJkyBbNmzYK7uzumT59ucnNVRcFktYLT6wWSMnNwNz0Hd9M197f736dpcC/jwfd3M3KQk1vwPBR7ldX9J0MZRk0DvBxhy0v6RERERnK5HPPnz8fixYvz7btx40aBxzRv3hzbt28v9JwqlQpz587F3Llz8+3r0KGDyXqwCxcuLPQ8j9YFgJdffhldunSBvb19gU8XLajPkZGRhbZhLsxGLIwQAqnZuUjKyMG9jBwkZuQgMUODxAwtEjMeJKIJ9xPRxIwc6PTFW9jYVmkFNwcVmng5GkdN/arZQirlqCkRERFZFiarpUgIgbw/asT913oBpGRpkZSZg3vp+ZPPBwmpYUvKzMl3I1NROFnLUdVWCWdbBZxtlfe3B99Xfeh7tYLzTImIiKhiYLL6iKh7GQiZ/x90uTJ8dGSHIesEIGBIRPPSSCHE/YS0bPpho5Chiq0CVawVqGKjQBUbQ8JZ1eZ+0mn3IBmtYqMwWVyfiIiISteuXbvM3YVKi8nqI4QAsrV6ABKgkPmdxSWRAI5q+f2k86Hk00YBJxtDAlrlkY3rkxIRERExWc3Hw0mNXf/XFjt37kTH55+HXC5H3g3wEkggkQDGmZ2SB2X3X0IikRj3S+7vt1HKYMWRTyIiquTyVpR59EYgqjjM8bNjsvoIuUwKD0c1qiiB6o7qfE92ICIiopKxsjKkHZmZmVCri/+4UzK/zMxMAPmffFWWmKwSERFRuZDJZHB0dDQ+e97a2prrd5chvV6PnJwcZGdnF7h0VXEIIZCZmYn4+Hg4OjqW61OwmKwSERFRuXFzcwMAY8JKZUcIgaysLKjVpfdQH0dHR+PPsLwwWSUiIqJyI5FI4O7uDhcXF2i1WnN355mm1Wrx33//oV27dqVy2V4ul5friGoeJqtERERU7mQymVkSn8pEJpMhNzcXKpWqQt+Dw1vUiYiIiMhiMVklIiIiIovFZJWIiIiILFaFmLOatwBtampqubSn1WqRmZmJ1NTUCj3Ho6Jh3M2DcTcPxt08GHfzYNzNo7zjnpenlfaDAypEspqWlgYA8PLyMnNPiIiIiOhx0tLS4ODgUGrnk4gK8MwzvV6PO3fuwM7OrlwWD05NTYWXlxdu3rwJe3v7Mm+PDBh382DczYNxNw/G3TwYd/Mo77gLIZCWlobq1as/9UMIHlYhRlalUik8PT3LvV17e3v+UpkB424ejLt5MO7mwbibB+NuHuUZ99IcUc3DG6yIiIiIyGIxWSUiIiIii8VktQBKpRJhYWFQKpXm7kqlwribB+NuHoy7eTDu5sG4m8ezEvcKcYMVEREREVVOHFklIiIiIovFZJWIiIiILBaTVSIiIiKyWExWiYiIiMhimTVZnTlzJpo3bw47Ozu4uLjghRdewMWLF03qZGdnY+TIkahatSpsbW3x8ssvIy4uzrj/5MmTeP311+Hl5QW1Wg1/f38sWLCg0Db37dsHKysrBAYGPrF/QghMmTIF7u7uUKvV6Ny5My5fvmxSp0+fPvD29oZKpYK7uzsGDRqEO3fuPPa8MTExGDBgAOrUqQOpVIqxY8fmq6PVajF9+nT4+flBpVIhICAAW7dufWKfi6Kyxn39+vXo0qULqlWrBnt7ewQHB2Pbtm2F1v/yyy8hkUgK/PmUBONeeNzT0tIwduxY+Pj4QK1Wo3Xr1jhy5MgT+1wUz0Lc82g0GgQGBkIikSAyMvKJ5961axeaNm0KpVKJWrVqYfny5Sb7Fy9ejMaNGxsXDA8ODsY///zzxPMWBeNeeNx1Oh0+/fRT1KhRA2q1Gn5+fvjss89K5XnqlTXuRfl/df369WjWrBkcHR1hY2ODwMBA/PLLL0/sc1Ew7oXHHQDmz5+PunXrQq1Ww8vLCx988AGys7Of2O+H34DZhISEiGXLlokzZ86IyMhI0aNHD+Ht7S3S09ONdYYPHy68vLxERESEOHr0qGjVqpVo3bq1cf9PP/0kRo8eLXbt2iWuXr0qfvnlF6FWq8XChQvztZeUlCRq1qwpunbtKgICAp7Yvy+//FI4ODiIDRs2iJMnT4o+ffqIGjVqiKysLGOduXPnigMHDogbN26Iffv2ieDgYBEcHPzY816/fl2MHj1arFixQgQGBooxY8bkqzN+/HhRvXp1sXnzZnH16lWxaNEioVKpxPHjx5/Y7yeprHEfM2aM+Oqrr8Thw4fFpUuXxKRJk4RcLi8wpocPHxa+vr6icePGBf58SoJxLzzur732mqhfv77YvXu3uHz5sggLCxP29vbi1q1bT+z3kzwLcc8zevRo0b17dwFAnDhx4rHnvXbtmrC2thbjxo0T586dEwsXLhQymUxs3brVWOfvv/8WmzdvFpcuXRIXL14UH3/8sZDL5eLMmTNP7PeTMO6Fx/2LL74QVatWFZs2bRLXr18Xf/zxh7C1tRULFix4Yr+fpLLGvSj/r+7cuVOsX79enDt3Tly5ckXMnz8/38+mpBj3wuO+cuVKoVQqxcqVK8X169fFtm3bhLu7u/jggw+e2O88Zk1WHxUfHy8AiN27dwshhEhOThZyuVz88ccfxjrnz58XAMSBAwcKPc+IESPE888/n6+8X79+4pNPPhFhYWFP/OHq9Xrh5uYmZs2aZSxLTk4WSqVS/Pbbb4Ue99dffwmJRCJycnIee/487du3L/CH6+7uLr799luTspdeekkMHDiwSOctjsoY9zz169cX06ZNMylLS0sTtWvXFuHh4YX+fEoD426Ie2ZmppDJZGLTpk0mdZo2bSomT55crPMWRUWN+5YtW0S9evXE2bNni/SfyPjx40WDBg3y9S0kJOSxxzk5OYn//e9/j61TEoz7g7j37NlTvPXWWyZ1+O/708X9YcX5d7tJkybik08+KfK5i4pxf2DkyJGiY8eOJmXjxo0Tbdq0KfK5LWrOakpKCgCgSpUqAIBjx45Bq9Wic+fOxjr16tWDt7c3Dhw48Njz5J0jz7Jly3Dt2jWEhYUVqS/Xr19HbGysSdsODg5o2bJloW0nJiZi5cqVaN26NeRyeZHaKYxGo4FKpTIpU6vV2Lt371OdtyCVNe56vR5paWn5+jxy5Ej07NnTpA9lgXE39Dk3Nxc6nY6f9/sKintcXByGDRuGX375BdbW1kU694EDB/J9hkNCQgp9TzqdDr///jsyMjIQHBxcpDaKg3F/cN7WrVsjIiICly5dAmC4/Lt371507969SG0UR2WJe3EJIRAREYGLFy+iXbt2pX5+xv2B1q1b49ixYzh8+DAA4Nq1a9iyZQt69OhR5HNYlVpvnpJer8fYsWPRpk0bNGzYEAAQGxsLhUIBR0dHk7qurq6IjY0t8Dz79+/H6tWrsXnzZmPZ5cuXMXHiROzZswdWVkV7y3nnd3V1fWLbEyZMwLfffovMzEy0atUKmzZtKlIbjxMSEoK5c+eiXbt28PPzQ0REBNavXw+dTvfU535YZY777NmzkZ6ejtdee81Y9vvvv+P48eOlNl+yMIz7g7jb2dkhODgYn332Gfz9/eHq6orffvsNBw4cQK1atYp17iepiHEXQuDNN9/E8OHD0axZM9y4caPI5y7ovKmpqcjKyoJarQYAnD59GsHBwcjOzoatrS3+/PNP1K9fv0htFBXjbhr3iRMnIjU1FfXq1YNMJoNOp8MXX3yBgQMHFqmNoqpMcS+qlJQUeHh4QKPRQCaTYdGiRejSpUuptsG4mxowYADu3r2L5557DkII5ObmYvjw4fj444+LfA6LGVkdOXIkzpw5g99//73E5zhz5gz69u2LsLAwdO3aFYBhtGDAgAGYNm0a6tSpU+BxK1euhK2trXHbs2dPsdr96KOPcOLECWzfvh0ymQyDBw82TpR/+LzDhw8v8jkXLFiA2rVro169elAoFBg1ahSGDBkCqbR0f2SVNe6rVq3CtGnTsGbNGri4uAAAbt68iTFjxmDlypX5RvlKG+P+IO4A8Msvv0AIAQ8PDyiVSnzzzTd4/fXX+XkHsHDhQqSlpWHSpEmF1inpvzMAULduXURGRuLQoUN47733EBoainPnzhXrHE/CuJtas2YNVq5ciVWrVuH48eNYsWIFZs+ejRUrVhT5HEXBuOdnZ2eHyMhIHDlyBF988QXGjRuHXbt2FescT8K4m9q1axdmzJiBRYsW4fjx41i/fj02b96Mzz77rMjnsIg5qyNHjhSenp7i2rVrJuURERECgEhKSjIp9/b2FnPnzjUpO3v2rHBxcREff/yxSXlSUpIAIGQymXGTSCTGsoiICJGamiouX75s3DIzM8XVq1cLnK/Rrl07MXr06ELfy82bNwUAsX//fiGEMDlvXFxcvvpPmluTlZUlbt26JfR6vRg/fryoX79+oXWLq7LG/bfffhNqtTrfHMk///wzX58BCIlEImQymcjNzS20/eJg3DcVdCohhBDp6enizp07QgjDTVc9evQotG5xVdS49+3bV0il0nyfS5lMJgYPHiyEKDjubdu2zfdvy9KlS4W9vf1j49SpUyfxzjvvPLZOcTDu+ePu6emZ756Ezz77TNStW/fJAS2iyhb3hxVnzurbb78tunbtWqS6RcG4j8lX/txzz4kPP/zQpCzv5jGdTldoLB9m1mRVr9eLkSNHiurVq4tLly7l2583IXnt2rXGsgsXLuSbkHzmzBnh4uIiPvroo3zn0Ol04vTp0ybbe++9J+rWrStOnz5tcqfeo31zc3MTs2fPNpalpKQ88YaTqKgoAUDs3LmzKCEo8i9VTk6O8PPzE5MmTSrSeR+nMsd91apVQqVSiQ0bNuTbl5qamq/PzZo1E2+88YY4ffr0Y89bFIx7wXEvSGJionBwcBDff/99keo/TkWPe1RUlMl5t23bJgCItWvXips3bxb6vsePHy8aNmxoUvb6668/8Qar559/XoSGhj62TlEw7g88GvcqVaqIRYsWmdSZMWOGqF27dqHnLarKGveHFSdZHTJkiGjfvn2R6j4O41543Js2bSrGjx9vUrZq1SqhVquLPAhk1mT1vffeEw4ODmLXrl0iJibGuGVmZhrrDB8+XHh7e4t///1XHD16NN9SOadPnxbVqlUTb7zxhsk54uPjC223KHfPCWFY6sHR0VH89ddf4tSpU6Jv374mSz0cPHhQLFy4UJw4cULcuHFDREREiNatWws/Pz+RnZ392HOfOHFCnDhxQgQFBYkBAwaIEydOiLNnzxr3Hzx4UKxbt05cvXpV/Pfff6Jjx46iRo0a+f4qK4nKGveVK1cKKysr8d1335n0OTk5udBjSnM1AMa98Lhv3bpV/PPPP+LatWti+/btIiAgQLRs2bLYqwwUpKLH/VHXr18v1hJKH330kTh//rz47rvv8i3TM3HiRLF7925x/fp1cerUKTFx4kQhkUjE9u3bn9jvJ2HcC497aGio8PDwMC5dtX79euHs7JzvP/SSqKxxF+LJ/6/OmDFDbN++XVy9elWcO3dOzJ49W1hZWYkff/zxied+Esa98LiHhYUJOzs78dtvvxn/jffz8xOvvfbaE8+dx6zJKoACt2XLlhnrZGVliREjRggnJydhbW0tXnzxRRETE2PcHxYWVuA5fHx8Cm23qD9cvV4vPv30U+Hq6iqUSqXo1KmTuHjxonH/qVOnxPPPPy+qVKkilEql8PX1FcOHDy/S2pBP6vOuXbuEv7+/UCqVomrVqmLQoEHi9u3bTzxvUVTWuLdv377APj9uFKk0k1XGvfC4r169WtSsWVMoFArh5uYmRo4c+dg/Ioqjosf9UcX5T2Tnzp0iMDBQKBQKUbNmTZP3LIQQb731lvDx8REKhUJUq1ZNdOrUqVQSVSEY98fFPTU1VYwZM0Z4e3sLlUolatasKSZPniw0Gs0Tz/0klTnuT+rz5MmTRa1atYRKpRJOTk4iODhY/P777088b1Ew7oX3WavViqlTpwo/Pz+hUqmEl5eXGDFiRLEG3yT3GyIiIiIisjgWsxoAEREREdGjmKwSERERkcViskpEREREFovJKhERERFZLCarRERERGSxmKwSERERkcViskpEREREFovJKhERERFZLCarRERlZPny5XB0dCzx8VOnTkVgYGCp9YeIqCJiskpE9AyRSCTYsGGDubtBRFRqmKwSERERkcViskpElZZer8fMmTNRo0YNqNVqBAQEYO3atdDr9fD09MTixYtN6p84cQJSqRRRUVEAgLlz56JRo0awsbGBl5cXRowYgfT09BL358svv4Srqyvs7Ozw9ttvIzs722T/kSNH0KVLFzg7O8PBwQHt27fH8ePHjft9fX0BAC+++CIkEonxNQD89ddfaNq0KVQqFWrWrIlp06YhNze3xH0lIiovTFaJqNKaOXMmfv75ZyxZsgRnz57FBx98gDfeeAN79uzB66+/jlWrVpnUX7lyJdq0aQMfHx8AgFQqxTfffIOzZ89ixYoV+PfffzF+/PgS9WXNmjWYOnUqZsyYgaNHj8Ld3R2LFi0yqZOWlobQ0FDs3bsXBw8eRO3atdGjRw+kpaUBMCSzALBs2TLExMQYX+/ZsweDBw/GmDFjcO7cOXz//fdYvnw5vvjiixL1lYioXAkiokooOztbWFtbi/3795uUv/322+L1118XJ06cEBKJRERFRQkhhNDpdMLDw0MsXry40HP+8ccfomrVqsbXy5YtEw4ODkXqT3BwsBgxYoRJWcuWLUVAQEChx+h0OmFnZyc2btxoLAMg/vzzT5N6nTp1EjNmzDAp++WXX4S7u3uR+kZEZE4cWSWiSunKlSvIzMxEly5dYGtra9x+/vlnXL16FYGBgfD39zeOru7evRvx8fF49dVXjefYsWMHOnXqBA8PD9jZ2WHQoEG4d+8eMjMzi92f8+fPo2XLliZlwcHBJq/j4uIwbNgw1K5dGw4ODrC3t0d6ejqio6Mfe+6TJ09i+vTpJu9z2LBhiImJKVFfiYjKk5W5O0BEZA55c0s3b94MDw8Pk31KpRIAMHDgQKxatQoTJ07EqlWr0K1bN1StWhUAcOPGDfTq1QvvvfcevvjiC1SpUgV79+7F22+/jZycHFhbW5d6n0NDQ3Hv3j0sWLAAPj4+UCqVCA4ORk5OzhPf67Rp0/DSSy/l26dSqUq9n0REpYnJKhFVSvXr14dSqUR0dDTat29fYJ0BAwbgk08+wbFjx7B27VosWbLEuO/YsWPQ6/WYM2cOpFLDRao1a9aUuD/+/v44dOgQBg8ebCw7ePCgSZ19+/Zh0aJF6NGjBwDg5s2buHv3rkkduVwOnU5nUta0aVNcvHgRtWrVKnH/iIjMhckqEVVKdnZ2+PDDD/HBBx9Ar9fjueeeQ0pKCvbt2wd7e3uEhobC19cXrVu3xttvvw2dToc+ffoYj69Vqxa0Wi0WLlyI3r17Y9++fSbJbHGNGTMGb775Jpo1a4Y2bdpg5cqVOHv2LGrWrGmsU7t2bfzyyy9o1qwZUlNT8dFHH0GtVpucx9fXFxEREWjTpg2USiWcnJwwZcoU9OrVC97e3njllVcglUpx8uRJnDlzBp9//nmJ+0xEVB44Z5WIKq3PPvsMn376KWbOnAl/f39069YNmzdvRo0aNYx1Bg4ciJMnT+LFF180SQwDAgIwd+5cfPXVV2jYsCFWrlyJmTNnlrgv/fr1w6efforx48cjKCgIUVFReO+990zq/PTTT0hKSkLTpk0xaNAgjB49Gi4uLiZ15syZg/DwcHh5eaFJkyYAgJCQEGzatAnbt29H8+bN0apVK8ybN8+4qgERkSWTCCGEuTtBRERERFQQjqwSERERkcViskpEVA4aNGhgsnTUw9vKlSvN3T0iIovFaQBEROUgKioKWq22wH15j1glIqL8mKwSERERkcXiNAAiIiIislhMVomIiIjIYjFZJSIiIiKLxWSViIiIiCwWk1UiIiIislhMVomIiIjIYjFZJSIiIiKL9f/0/CjpiDrIFgAAAABJRU5ErkJggg==",
            "text/plain": [
              "\u003cFigure size 800x300 with 1 Axes\u003e"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "evaluations.plot(y = ['log_loss', 'f1_score', 'precision'], legend = True, title = 'Evaluation Metrics Over Last 30 days', figsize = (8, 3), grid = True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "06zUgdOSxLVM"
      },
      "source": [
        "---\n",
        "## **Model Monitoring**\n",
        "\n",
        "Reviewing model evaluations over time does show a pattern of deterioration in the example above.  But at what point is it actionable?  And what if each days evaluations are relying on actual values that took hours, days, weeks, or even months to be available?  \n",
        "\n",
        "Model monitoring takes a different approach.  It goes back to the source of the model, each input feature.  These features form a distribution at the input to the model (the training data) and this can be used to compare to over time.  By setting thresholds on these comparisons to the original training data, alerts can be used to notify that a models accuracy *could be* in question.  This comparison to the training data is called **skew** detection.\n",
        "\n",
        "Similarly, each days values could be compared to the previous days, or weeks values.  These comparisons are referred to as **drift** detection.  \n",
        "\n",
        "Both of these types of comparisons can be done directly in BigQuery with two functions built for these tasks:\n",
        "- [`ML.VALIDATE_DATA_SKEW`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-validate-data-skew)\n",
        "- [`ML.VALIDATE_DATA_DRIFT`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-validate-data-drift)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1lACvkTJxLVM"
      },
      "source": [
        "### Monitoring Skew (Change from training) With `ML.VALIDATE_DATA_SKEW`\n",
        "\n",
        "Compare today, or any data, to the training data by directly comparing to the the model created in BigQuery."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "tags": [],
        "id": "NPuam-xjxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7a138f013e844d69a7fbff00b39f419f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3412ed003f824fab93ce7550140e16c9",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003einput\u003c/th\u003e\n",
              "      \u003cth\u003emetric\u003c/th\u003e\n",
              "      \u003cth\u003ethreshold\u003c/th\u003e\n",
              "      \u003cth\u003evalue\u003c/th\u003e\n",
              "      \u003cth\u003eis_anomaly\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003ebody_mass_g\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.335210\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_depth_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.023026\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.061987\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eflipper_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.950948\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.042504\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e5\u003c/th\u003e\n",
              "      \u003ctd\u003esex\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.112512\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "               input                     metric  threshold     value  \\\n",
              "0        body_mass_g  JENSEN_SHANNON_DIVERGENCE        0.3  0.335210   \n",
              "1    culmen_depth_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.023026   \n",
              "2   culmen_length_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.061987   \n",
              "3  flipper_length_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.950948   \n",
              "4             island                    L_INFTY        0.3  0.042504   \n",
              "5                sex                    L_INFTY        0.3  0.112512   \n",
              "\n",
              "   is_anomaly  \n",
              "0        True  \n",
              "1       False  \n",
              "2       False  \n",
              "3        True  \n",
              "4       False  \n",
              "5       False  "
            ]
          },
          "execution_count": 35,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.VALIDATE_DATA_SKEW(\n",
        "    MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "        WHERE DATE(instance_timestamp) \u003e= CURRENT_DATE()\n",
        "    )\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EV6yWpulxLVM"
      },
      "source": [
        "### Monitoring Drift (Change Over Time) With `ML.VALIDATE_DATA_DRIFT`\n",
        "\n",
        "Compare today to 5 days ago and set the a lower, more sensitive, threshold of `0.03` (default is `0.3`) for all features."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "tags": [],
        "id": "XViVZouyxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a069784d2b5f4cb1925cb95b5c3cf016",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "052a2e5686764173a28727c7a6f3b999",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003einput\u003c/th\u003e\n",
              "      \u003cth\u003emetric\u003c/th\u003e\n",
              "      \u003cth\u003ethreshold\u003c/th\u003e\n",
              "      \u003cth\u003evalue\u003c/th\u003e\n",
              "      \u003cth\u003eis_anomaly\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003ebody_mass_g\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.040365\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_depth_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000000\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000000\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eflipper_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.118338\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000000\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e5\u003c/th\u003e\n",
              "      \u003ctd\u003esex\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000000\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "               input                     metric  threshold     value  \\\n",
              "0        body_mass_g  JENSEN_SHANNON_DIVERGENCE       0.03  0.040365   \n",
              "1    culmen_depth_mm  JENSEN_SHANNON_DIVERGENCE       0.03  0.000000   \n",
              "2   culmen_length_mm  JENSEN_SHANNON_DIVERGENCE       0.03  0.000000   \n",
              "3  flipper_length_mm  JENSEN_SHANNON_DIVERGENCE       0.03  0.118338   \n",
              "4             island                    L_INFTY       0.03  0.000000   \n",
              "5                sex                    L_INFTY       0.03  0.000000   \n",
              "\n",
              "   is_anomaly  \n",
              "0        True  \n",
              "1       False  \n",
              "2       False  \n",
              "3        True  \n",
              "4       False  \n",
              "5       False  "
            ]
          },
          "execution_count": 36,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.VALIDATE_DATA_DRIFT(\n",
        "    (\n",
        "        SELECT * EXCEPT(instance_timestamp, species)\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "        WHERE DATE(instance_timestamp) = DATE_SUB(CURRENT_DATE(), INTERVAL 5 DAY)\n",
        "    ),\n",
        "    (\n",
        "        SELECT * EXCEPT(instance_timestamp, species)\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "        WHERE DATE(instance_timestamp) = CURRENT_DATE() \n",
        "    ),\n",
        "    STRUCT(\n",
        "        0.03 AS categorical_default_threshold,\n",
        "        0.03 AS numerical_default_threshold\n",
        "    )\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "etL0IyVqxLVM"
      },
      "source": [
        "### Monitoring Skew (Manually) - As Drift from Training Data\n",
        "\n",
        "Since the `ML.VALIDATE_DATA_DRIFT` is comparing two tables, it could also be used for **skew** by comparing the training data to new data, like todays data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "tags": [],
        "id": "MqJ7c8cdxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "500c1285bfee447fadb5d466877ee1b0",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d2675d30f46b4d73988df5b55877a547",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003einput\u003c/th\u003e\n",
              "      \u003cth\u003emetric\u003c/th\u003e\n",
              "      \u003cth\u003ethreshold\u003c/th\u003e\n",
              "      \u003cth\u003evalue\u003c/th\u003e\n",
              "      \u003cth\u003eis_anomaly\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003ebody_mass_g\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.335210\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_depth_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.023026\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.061987\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eflipper_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.950948\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.042504\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e5\u003c/th\u003e\n",
              "      \u003ctd\u003esex\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.112512\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "               input                     metric  threshold     value  \\\n",
              "0        body_mass_g  JENSEN_SHANNON_DIVERGENCE        0.3  0.335210   \n",
              "1    culmen_depth_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.023026   \n",
              "2   culmen_length_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.061987   \n",
              "3  flipper_length_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.950948   \n",
              "4             island                    L_INFTY        0.3  0.042504   \n",
              "5                sex                    L_INFTY        0.3  0.112512   \n",
              "\n",
              "   is_anomaly  \n",
              "0        True  \n",
              "1       False  \n",
              "2       False  \n",
              "3        True  \n",
              "4       False  \n",
              "5       False  "
            ]
          },
          "execution_count": 37,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.VALIDATE_DATA_DRIFT(\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "        WHERE splits = 'TRAIN'\n",
        "    ),\n",
        "    (\n",
        "        SELECT * EXCEPT(instance_timestamp, species)\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "        WHERE DATE(instance_timestamp) = CURRENT_DATE() \n",
        "    )\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dqDsuu2uxLVM"
      },
      "source": [
        "---\n",
        "### TensorFlow Data Validation (TFDV) In BigQuery\n",
        "\n",
        "BigQuery offers two functions that bring the power of the TensorFlow's [TFX platform](https://www.tensorflow.org/tfx) with built-in [TensorFlow Data Validation (TFDV)](https://www.tensorflow.org/tfx/data_validation/get_started) module directly inside of BigQuery.\n",
        "- [`ML.TFDV_DESCRIBE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-tfdv-describe) to generate descriptive statistics of any table\n",
        "- [`ML.TFDV_VALIDATE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-tfdv-validate) to compare statistics of two tables"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xgJXmr6rxLVM"
      },
      "source": [
        "#### Generate Statistics with `ML.TFDV_DESCRIBE`\n",
        "\n",
        "Generate statistics for any table, in this case the training data features or a timespan of serving data features.  This is like using the TensorFlow's [TFX platform](https://www.tensorflow.org/tfx) with built-in [TensorFlow Data Validation (TFDV)](https://www.tensorflow.org/tfx/data_validation/get_started) module's [`tfdv.generate_statistics_from_csv()`](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/generate_statistics_from_csv) method.\n",
        "\n",
        "The descriptions can be used in comparisons for skew or drift detection with the `ML.TFDV_VALIDATE` function, covered next, as well as in the visualization tool `tfdv.visualize_statistics` also covered below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "tags": [],
        "id": "SYsyK-zsxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e23cc6c2b94c406ea10df2bb575d266f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c583d4e139ca4965949028791ccfee86",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "%%bigquery train_describe\n",
        "SELECT *\n",
        "FROM ML.TFDV_DESCRIBE(\n",
        "    (\n",
        "        SELECT * EXCEPT(splits, species)\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "        WHERE splits = 'TRAIN'\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "tags": [],
        "id": "jBKAR3WMxLVM"
      },
      "outputs": [],
      "source": [
        "train_describe = json.loads(train_describe['dataset_feature_statistics_list'].iloc[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "tags": [],
        "id": "og7io97YxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'name': 'body_mass_g',\n",
              " 'num_stats': {'common_stats': {'avg_num_values': 1,\n",
              "   'max_num_values': '1',\n",
              "   'min_num_values': '1',\n",
              "   'num_non_missing': '237',\n",
              "   'num_values_histogram': {'buckets': [{'high_value': 1,\n",
              "      'low_value': 1,\n",
              "      'sample_count': 23.8},\n",
              "     {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "     {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "     {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "     {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "     {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "     {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "     {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "     {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "     {'high_value': 1, 'low_value': 1, 'sample_count': 23.8}],\n",
              "    'type': 'QUANTILES'},\n",
              "   'tot_num_values': '237'},\n",
              "  'histograms': [{'buckets': [{'high_value': 3275,\n",
              "      'low_value': 2850,\n",
              "      'sample_count': 23.8},\n",
              "     {'high_value': 3500, 'low_value': 3275, 'sample_count': 23.8},\n",
              "     {'high_value': 3650, 'low_value': 3500, 'sample_count': 23.8},\n",
              "     {'high_value': 3800, 'low_value': 3650, 'sample_count': 23.8},\n",
              "     {'high_value': 4050, 'low_value': 3800, 'sample_count': 23.8},\n",
              "     {'high_value': 4375, 'low_value': 4050, 'sample_count': 23.8},\n",
              "     {'high_value': 4650, 'low_value': 4375, 'sample_count': 23.8},\n",
              "     {'high_value': 4925, 'low_value': 4650, 'sample_count': 23.8},\n",
              "     {'high_value': 5400, 'low_value': 4925, 'sample_count': 23.8},\n",
              "     {'high_value': 6300, 'low_value': 5400, 'sample_count': 23.8}],\n",
              "    'type': 'QUANTILES'},\n",
              "   {'buckets': [{'high_value': 3195,\n",
              "      'low_value': 2850,\n",
              "      'sample_count': 16.136399999999977},\n",
              "     {'high_value': 3540,\n",
              "      'low_value': 3195,\n",
              "      'sample_count': 39.222399999999936},\n",
              "     {'high_value': 3885,\n",
              "      'low_value': 3540,\n",
              "      'sample_count': 48.02839999999992},\n",
              "     {'high_value': 4230,\n",
              "      'low_value': 3885,\n",
              "      'sample_count': 29.083599999999954},\n",
              "     {'high_value': 4575,\n",
              "      'low_value': 4230,\n",
              "      'sample_count': 26.03719999999996},\n",
              "     {'high_value': 4920,\n",
              "      'low_value': 4575,\n",
              "      'sample_count': 31.368399999999948},\n",
              "     {'high_value': 5265,\n",
              "      'low_value': 4920,\n",
              "      'sample_count': 17.016999999999978},\n",
              "     {'high_value': 5610,\n",
              "      'low_value': 5265,\n",
              "      'sample_count': 18.064199999999975},\n",
              "     {'high_value': 5955,\n",
              "      'low_value': 5610,\n",
              "      'sample_count': 9.972199999999988},\n",
              "     {'high_value': 6300, 'low_value': 5955, 'sample_count': 3.0702}]}],\n",
              "  'max': 6300,\n",
              "  'mean': 4197.78481012658,\n",
              "  'median': 4050,\n",
              "  'min': 2850,\n",
              "  'std_dev': 794.4280165783061},\n",
              " 'type': 'FLOAT'}"
            ]
          },
          "execution_count": 40,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "train_describe['datasets'][0]['features'][0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "tags": [],
        "id": "McnPP37UxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6c487ab6fd204bfc9a5099ef9d9c2378",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2c7202d1927644b8960308075687bf76",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "%%bigquery today_describe\n",
        "SELECT *\n",
        "FROM ML.TFDV_DESCRIBE(\n",
        "    (\n",
        "        SELECT * EXCEPT(instance_timestamp, species)\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "        WHERE DATE(instance_timestamp) = CURRENT_DATE()\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "tags": [],
        "id": "0rL8Hpv3xLVM"
      },
      "outputs": [],
      "source": [
        "today_describe = json.loads(today_describe['dataset_feature_statistics_list'].iloc[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "tags": [],
        "id": "uCyYhkT7xLVM"
      },
      "outputs": [],
      "source": [
        "#today_describe['datasets'][0]['features'][0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "At4240OVxLVM"
      },
      "source": [
        "#### Visualize Statistics with `tfdv.visualize_statistics`\n",
        "\n",
        "Use the [`tfdv.visualize_statistics`](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/visualize_statistics) tool to visualize the generated descriptions and directly compare them.  This presents visuals using [Facets Overview](https://pair-code.github.io/facets/)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "tags": [],
        "id": "lQOrV_q2xLVM"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
        "import tensorflow_data_validation as tfdv\n",
        "import tensorflow_metadata as tfmd\n",
        "from google.protobuf import json_format"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "tags": [],
        "id": "ynrRtJ97xLVM"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\u003ciframe id='facets-iframe' width=\"100%\" height=\"500px\"\u003e\u003c/iframe\u003e\n",
              "        \u003cscript\u003e\n",
              "        facets_iframe = document.getElementById('facets-iframe');\n",
              "        facets_html = '\u003cscript src=\"https://cdnjs.cloudflare.com/ajax/libs/webcomponentsjs/1.3.3/webcomponents-lite.js\"\u003e\u003c\\/script\u003e\u003clink rel=\"import\" href=\"https://raw.githubusercontent.com/PAIR-code/facets/master/facets-dist/facets-jupyter.html\"\u003e\u003cfacets-overview proto-input=\"Cq4lChNUcmFpbmluZyBEYXRhIFN0YXRzEO4BGsQHCgtib2R5X21hc3NfZxABGrIHCrYCCO0BGAEgAS0AAIA/MqQCGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AgAUDtARE7A1HpyGWwQBnEr/STbNOIQCkAAAAAAESmQDEAAAAAAKSvQDkAAAAAAJy4QEKkAhobCQAAAAAARKZAEQAAAAAAlqlAIc3MzMzMzDdAGhsJAAAAAACWqUARAAAAAABYq0AhzczMzMzMN0AaGwkAAAAAAFirQBEAAAAAAISsQCHNzMzMzMw3QBobCQAAAAAAhKxAEQAAAAAAsK1AIc3MzMzMzDdAGhsJAAAAAACwrUARAAAAAACkr0AhzczMzMzMN0AaGwkAAAAAAKSvQBEAAAAAABexQCHNzMzMzMw3QBobCQAAAAAAF7FAEQAAAAAAKrJAIc3MzMzMzDdAGhsJAAAAAAAqskARAAAAAAA9s0AhzczMzMzMN0AaGwkAAAAAAD2zQBEAAAAAABi1QCHNzMzMzMw3QBobCQAAAAAAGLVAEQAAAAAAnLhAIc3MzMzMzDdAIAFCogIaGwkAAAAAAESmQBEAAAAAAPaoQCGfLEMc6yIwQBobCQAAAAAA9qhAEQAAAAAAqKtAIahQa5p3nENAGhsJAAAAAACoq0ARAAAAAABarkAhYJp3nKIDSEAaGwkAAAAAAFquQBEAAAAAAIawQCEG8kHPZhU9QBobCQAAAAAAhrBAEQAAAAAA37FAITlpb/CFCTpAGhsJAAAAAADfsUARAAAAAAA4s0Ahn9hfdk9eP0AaGwkAAAAAADizQBEAAAAAAJG0QCErCKwcWgQxQBobCQAAAAAAkbRAEQAAAAAA6rVAITFnRGlvEDJAGhsJAAAAAADqtUARAAAAAABDt0AhoVfKMsTxI0AaGwkAAAAAAEO3QBEAAAAAAJy4QCEAb4EExY8IQBrIBwoPY3VsbWVuX2RlcHRoX21tEAEasgcKtgII7QEYASABLQAAgD8ypAIaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QCABQO0BEXuSGmWsIjFAGRlfmzt0rP8/KTMzMzMzMypAMc3MzMzMTDFAOTMzMzMzMzVAQqQCGhsJAAAAAAAAKkARZmZmZmZmLEAhzczMzMzMN0AaGwkAAAAAAAAsQBEAAAAAAAAuQCHNzMzMzMw3QBobCQAAAAAAAC5AEc3MzMzMzC9AIc3MzMzMzDdAGhsJAAAAAAAALkARzczMzMzMMEAhzczMzMzMN0AaGwkAAAAAAAAwQBHNzMzMzEwxQCHNzMzMzMw3QBobCQAAAAAAADFAEWZmZmZm5jFAIc3MzMzMzDdAGhsJAAAAAAAAMUARAAAAAACAMkAhzczMzMzMN0AaGwkAAAAAAAAyQBFmZmZmZuYyQCHNzMzMzMw3QBobCQAAAAAAADJAEZqZmZmZmTNAIc3MzMzMzDdAGhsJAAAAAAAAM0ARMzMzMzMzNUAhzczMzMzMN0AgAUKiAhobCQAAAAAAACpAEaRwPQrXoytAIdrMOpQ/IypAGhsJpHA9CtejK0ARSOF6FK5HLUAhS4fcDDdAOEAaGwlI4XoUrkctQBHsUbgehesuQCFXCc3qbKorQBobCexRuB6F6y5AEUjhehSuRzBAIZkv/H712StAGhsJSOF6FK5HMEARmpmZmZkZMUAhhwQd/CZpM0AaGwmamZmZmRkxQBHsUbgehesxQCG10MpuBJ06QBobCexRuB6F6zFAET4K16NwvTJAIelG5pE/hENAGhsJPgrXo3C9MkARkML1KFyPM0Ah0ZviwA19NUAaGwmQwvUoXI8zQBHiehSuR2E0QCEP9SboayUiQBobCeJ6FK5HYTRAETQzMzMzMzVAISK6rZ1aJRhAGskHChBjdWxtZW5fbGVuZ3RoX21tEAEasgcKtgII7QEYASABLQAAgD8ypAIaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QCABQO0BEX807fbp5UVAGSHxchUpNBVAKc3MzMzMDEBAMWZmZmZm5kVAOQAAAAAAAE1AQqQCGhsJAAAAAAAAQEARAAAAAABAQkAhzczMzMzMN0AaGwkAAAAAAABCQBEAAAAAAEBDQCHNzMzMzMw3QBobCQAAAAAAAENAEQAAAAAAQERAIc3MzMzMzDdAGhsJAAAAAAAAREARmpmZmZnZREAhzczMzMzMN0AaGwkAAAAAAIBEQBEAAAAAAABGQCHNzMzMzMw3QBobCQAAAAAAAEZAEWZmZmZm5kZAIc3MzMzMzDdAGhsJAAAAAACARkARMzMzMzNzR0AhzczMzMzMN0AaGwkAAAAAAABHQBHNzMzMzIxIQCHNzMzMzMw3QBobCQAAAAAAgEhAEc3MzMzMTElAIc3MzMzMzDdAGhsJAAAAAAAASUARAAAAAAAATUAhzczMzMzMN0AgAUKiAhobCQAAAAAAAEBAEc3MzMzMTEFAIW9GlPYGXxhAGhsJzczMzMxMQUARmpmZmZmZQkAhr7VfUSQyOkAaGwmamZmZmZlCQBFnZmZmZuZDQCEhujhLmfc+QBobCWdmZmZm5kNAETQzMzMzM0VAIU56gAGXGkJAGhsJNDMzMzMzRUARAQAAAACARkAhmdsIUYpzMEAaGwkBAAAAAIBGQBHOzMzMzMxHQCGjWy5ZfyxKQBobCc7MzMzMzEdAEZuZmZmZGUlAITzoFevEVz9AGhsJm5mZmZkZSUARaGZmZmZmSkAh7LGd76cGNEAaGwloZmZmZmZKQBE1MzMzM7NLQCFy1Z7OZGEQQBobCTUzMzMzs0tAEQIAAAAAAE1AIZZtokcqF/k/GsoHChFmbGlwcGVyX2xlbmd0aF9tbRABGrIHCrYCCO0BGAEgAS0AAIA/MqQCGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AgAUDtARFOs4LDFxJpQBn+xnalCogrQCkAAAAAAIBlQDEAAAAAAKBoQDkAAAAAAOBsQEKkAhobCQAAAAAAgGVAEQAAAAAAIGdAIc3MzMzMzDdAGhsJAAAAAAAgZ0ARAAAAAACAZ0AhzczMzMzMN0AaGwkAAAAAAIBnQBEAAAAAAOBnQCHNzMzMzMw3QBobCQAAAAAA4GdAEQAAAAAAQGhAIc3MzMzMzDdAGhsJAAAAAABAaEARAAAAAACgaEAhzczMzMzMN0AaGwkAAAAAAKBoQBEAAAAAAEBpQCHNzMzMzMw3QBobCQAAAAAAQGlAEQAAAAAAQGpAIc3MzMzMzDdAGhsJAAAAAABAakARAAAAAADgakAhzczMzMzMN0AaGwkAAAAAAOBqQBEAAAAAAIBrQCHNzMzMzMw3QBobCQAAAAAAgGtAEQAAAAAA4GxAIc3MzMzMzDdAIAFCogIaGwkAAAAAAIBlQBHNzMzMzDxmQCFIA3gLJKgIQBobCc3MzMzMPGZAEZqZmZmZ+WZAIXGDL0ym6i1AGhsJmpmZmZn5ZkARZ2ZmZma2Z0Ah0T15WKglQ0AaGwlnZmZmZrZnQBE0MzMzM3NoQCHBO07RkQxKQBobCTQzMzMzc2hAEQEAAAAAMGlAIQELtaZ5Jz9AGhsJAQAAAAAwaUARzszMzMzsaUAhgv1l9+QhKkAaGwnOzMzMzOxpQBGbmZmZmalqQCGSoiO5/Cc6QBobCZuZmZmZqWpAEWhmZmZmZmtAIVGYTBWM6j5AGhsJaGZmZmZma0ARNTMzMzMjbEAhE8wT9W8kNUAaGwk1MzMzMyNsQBECAAAAAOBsQCFs5vh7PIccQBrNAwoGaXNsYW5kEAIiwAMKtgII7gEYASABLQAAgD8ypAIaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QCABQO4BEAMaERIGQmlzY29lGQAAAAAAAF1AGhASBURyZWFtGQAAAAAAgFVAGhQSCVRvcmdlcnNlbhkAAAAAAABCQCU+9cJAKkMKESIGQmlzY29lKQAAAAAAAF1AChQIARABIgVEcmVhbSkAAAAAAIBVQAoYCAIQAiIJVG9yZ2Vyc2VuKQAAAAAAAEJAGpgDCgNzZXgQAiKOAwq2AgjpARgBIAEtAACAPzKkAhobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAGhsJAAAAAAAA8D8RAAAAAAAA8D8hzczMzMzMN0AaGwkAAAAAAADwPxEAAAAAAADwPyHNzMzMzMw3QBobCQAAAAAAAPA/EQAAAAAAAPA/Ic3MzMzMzDdAIAFA6QEQAhoREgZGRU1BTEUZAAAAAADAXkAaDxIETUFMRRkAAAAAAIBbQCUQyaFAKigKESIGRkVNQUxFKQAAAAAAwF5AChMIARABIgRNQUxFKQAAAAAAgFtACqwlCh5TZXJ2aW5nIERhdGEgU3RhdHMgLSBGb3IgVG9kYXkQRRrCBwoLYm9keV9tYXNzX2cQARqwBwq0AghEGAEgAS0AAIA/MqQCGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AgAUBEEQTIj0b0wbRAGby8ALG05otAKXS9iuYYIK5AMadQnGC2G7RAOXj8A3a4cb5AQqQCGhsJAAAAAAAgrkARZjFx78SlsEAhmpmZmZmZG0AaGwkAAAAAAKWwQBE4rFBuypKxQCGamZmZmZkbQBobCQAAAAAAkrFAEbDbPsEzoLJAIZqZmZmZmRtAGhsJAAAAAACgskARRS+CsBFms0AhmpmZmZmZG0AaGwkAAAAAAGazQBGnUJxgthu0QCGamZmZmZkbQBobCQAAAAAAG7RAETdxvpyIiLVAIZqZmZmZmRtAGhsJAAAAAACItUARkngVgMYotkAhmpmZmZmZG0AaGwkAAAAAACi2QBFRunxhPry3QCGamZmZmZkbQBobCQAAAAAAvLdAEWuhlffMW7pAIZqZmZmZmRtAGhsJAAAAAABbukARePwDdrhxvkAhmpmZmZmZG0AgAUKiAhobCQAAAAAAIK5AEXLMZqXFmbBAIap1rpPjhRhAGhsJcsxmpcWZsEAR5JjNSosjskAhSxTkSM5MJkAaGwnkmM1KiyOyQBFWZTTwUK2zQCGmiLnRx0QsQBobCVZlNPBQrbNAEcgxm5UWN7VAIa4/VF8pYSJAGhsJyDGblRY3tUAROv4BO9zAtkAh+3dZgPNDJkAaGwk6/gE73MC2QBGsymjgoUq4QCHjFd7lF2AUQBobCazKaOChSrhAER6Xz4Vn1LlAIbv7VXDTLBBAGhsJHpfPhWfUuUARkGM2Ky1eu0Ah2xV8aPEyFEAaGwmQYzYrLV67QBECMJ3Q8ue8QCF0kJlnqJIAQBobCQIwndDy57xAEXT8A3a4cb5AISvGGU81K6M/GsYHCg9jdWxtZW5fZGVwdGhfbW0QARqwBwq0AghEGAEgAS0AAIA/MqQCGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AgAUBEEYqKioqKajFAGSTEjB990/8/KTMzMzMzMypAMWZmZmZm5jFAOZqZmZmZGTVAQqQCGhsJAAAAAAAAKkARZmZmZmZmLEAhmpmZmZmZG0AaGwkAAAAAAAAsQBFmZmZmZmYuQCGamZmZmZkbQBobCQAAAAAAAC5AEZqZmZmZGTBAIZqZmZmZmRtAGhsJAAAAAAAAMEARmpmZmZmZMUAhmpmZmZmZG0AaGwkAAAAAAAAxQBFmZmZmZuYxQCGamZmZmZkbQBobCQAAAAAAADFAETMzMzMzMzJAIZqZmZmZmRtAGhsJAAAAAAAAMkARmpmZmZmZMkAhmpmZmZmZG0AaGwkAAAAAAAAyQBEAAAAAAAAzQCGamZmZmZkbQBobCQAAAAAAADNAEc3MzMzMzDNAIZqZmZmZmRtAGhsJAAAAAAAAM0ARmpmZmZkZNUAhmpmZmZmZG0AgAUKiAhobCQAAAAAAACpAER+F61G4nitAIZfmHafoyAhAGhsJH4XrUbieK0ARPgrXo3A9LUAhhXPyriggEUAaGwk+CtejcD0tQBFdj8L1KNwuQCHUI5PRxHoQQBobCV2PwvUo3C5AET4K16NwPTBAIfEFAIpQyhRAGhsJPgrXo3A9MEARzczMzMwMMUAhBE1nJ4PjB0AaGwnNzMzMzAwxQBFcj8L1KNwxQCHwePSCDqgVQBobCVyPwvUo3DFAEetRuB6FqzJAISf50e5TlCxAGhsJ61G4HoWrMkARehSuR+F6M0AhhCY4Nr3QE0AaGwl6FK5H4XozQBEJ16NwPUo0QCH36hAT7/QMQBobCQnXo3A9SjRAEZiZmZmZGTVAIcIEbu6yd/E/GscHChBjdWxtZW5fbGVuZ3RoX21tEAEasAcKtAIIRBgBIAEtAACAPzKkAhobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAIAFARBGioqKiosJFQBlRNmo9JYAWQCnNzMzMzAxBQDHNzMzMzMxEQDnNzMzMzMxNQEKkAhobCQAAAAAAAEFAEWZmZmZmZkJAIZqZmZmZmRtAGhsJAAAAAAAAQkARMzMzMzPzQkAhmpmZmZmZG0AaGwkAAAAAAIBCQBGamZmZmdlDQCGamZmZmZkbQBobCQAAAAAAgENAEc3MzMzMTERAIZqZmZmZmRtAGhsJAAAAAAAAREARzczMzMzMREAhmpmZmZmZG0AaGwkAAAAAAIBEQBEAAAAAAEBGQCGamZmZmZkbQBobCQAAAAAAAEZAEWZmZmZm5kdAIZqZmZmZmRtAGhsJAAAAAACAR0ARZmZmZmamSEAhmpmZmZmZG0AaGwkAAAAAAIBIQBFmZmZmZmZJQCGamZmZmZkbQBobCQAAAAAAAElAEc3MzMzMzE1AIZqZmZmZmRtAIAFCogIaGwkAAAAAAABBQBFI4XoUrkdCQCHHw0KtaV4UQBobCUjhehSuR0JAEZDC9Shcj0NAIUmQgxJmWiNAGhsJkML1KFyPQ0AR2KNwPQrXREAh0A4mhIyTMEAaGwnYo3A9CtdEQBEghetRuB5GQCHYY+lDF1QYQBobCSCF61G4HkZAEWhmZmZmZkdAITAjmlj4khhAGhsJaGZmZmZmR0ARsEfhehSuSEAh2uwiXxwvIEAaGwmwR+F6FK5IQBH4KFyPwvVJQCFa66sVTl0fQBobCfgoXI/C9UlAEUAK16NwPUtAIXBbY1KA6QdAGhsJQArXo3A9S0ARiOtRuB6FTEAhirKzp2ImoD8aGwmI61G4HoVMQBHQzMzMzMxNQCHaLnn/3uvvPxrIBwoRZmxpcHBlcl9sZW5ndGhfbW0QARqwBwq0AghEGAEgAS0AAIA/MqQCGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AgAUBEESiouxO6AHBAGSn+acGLhzJAKaHPGJr2PmxAMYQ9zfXqdm9AOViF53/DKHNAQqQCGhsJAAAAAAAgbEAR1r6Fh+oYbUAhmpmZmZmZG0AaGwkAAAAAAABtQBEVlsB7RrxtQCGamZmZmZkbQBobCQAAAAAAoG1AEdzwY9TIj25AIZqZmZmZmRtAGhsJAAAAAACAbkAREc3Ri9oMb0AhmpmZmZmZG0AaGwkAAAAAAABvQBGEPc316nZvQCGamZmZmZkbQBobCQAAAAAAYG9AEcOgUGU9FnBAIZqZmZmZmRtAGhsJAAAAAAAQcEARlNCgC1u5cEAhmpmZmZmZG0AaGwkAAAAAALBwQBHdxBs+Eg9xQCGamZmZmZkbQBobCQAAAAAAAHFAEdyOCOGMznFAIZqZmZmZmRtAGhsJAAAAAADAcUARWIXnf8Moc0AhmpmZmZmZG0AgAUKiAhobCQAAAAAAIGxAERKBYebzJG1AIeAij6wWNiBAGhsJEoFh5vMkbUARJALDzOcpbkAh9+V2FGxbIkAaGwkkAsPM5yluQBE2gySz2y5vQCG9eMLP2EwoQBobCTaDJLPbLm9AESQCw8znGXBAIcxBYfPcPShAGhsJJALDzOcZcEARrcLzv2GccEAhCMsNXtGhGEAaGwmtwvO/YZxwQBE2gySz2x5xQCFHdIXEezIgQBobCTaDJLPbHnFAEb9DVaZVoXFAIW85EkBeXRRAGhsJv0NVplWhcUARSASGmc8jckAhA/NIJu48GEAaGwlIBIaZzyNyQBHRxLaMSaZyQCESL0fO7ZDwPxobCdHEtoxJpnJAEVqF53/DKHNAIUA3m9r4CPA/GssDCgZpc2xhbmQQAiK+Awq0AghFGAEgAS0AAIA/MqQCGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AgAUBFEAMaERIGQmlzY29lGQAAAAAAAEFAGhASBURyZWFtGQAAAAAAADZAGhQSCVRvcmdlcnNlbhkAAAAAAAAqQCVS4sdAKkMKESIGQmlzY29lKQAAAAAAAEFAChQIARABIgVEcmVhbSkAAAAAAAA2QAoYCAIQAiIJVG9yZ2Vyc2VuKQAAAAAAACpAGpYDCgNzZXgQAiKMAwq0AghBGAEgAS0AAIA/MqQCGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AaGwkAAAAAAADwPxEAAAAAAADwPyGamZmZmZkbQBobCQAAAAAAAPA/EQAAAAAAAPA/IZqZmZmZmRtAGhsJAAAAAAAA8D8RAAAAAAAA8D8hmpmZmZmZG0AgAUBBEAIaDxIETUFMRRkAAAAAAABDQBoREgZGRU1BTEUZAAAAAAAAO0AlqZWaQCooCg8iBE1BTEUpAAAAAAAAQ0AKFQgBEAEiBkZFTUFMRSkAAAAAAAA7QA==\"\u003e\u003c/facets-overview\u003e';\n",
              "        facets_iframe.srcdoc = facets_html;\n",
              "         facets_iframe.id = \"\";\n",
              "         setTimeout(() =\u003e {\n",
              "           facets_iframe.setAttribute('height', facets_iframe.contentWindow.document.body.offsetHeight + 'px')\n",
              "         }, 1500)\n",
              "         \u003c/script\u003e"
            ],
            "text/plain": [
              "\u003cIPython.core.display.HTML object\u003e"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "tfdv.visualize_statistics(\n",
        "    lhs_statistics = json_format.ParseDict(train_describe, tfmd.proto.statistics_pb2.DatasetFeatureStatisticsList()),\n",
        "    rhs_statistics = json_format.ParseDict(today_describe, tfmd.proto.statistics_pb2.DatasetFeatureStatisticsList()),\n",
        "    lhs_name = 'Training Data Stats',\n",
        "    rhs_name = 'Serving Data Stats - For Today'\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "flmSZdZIxLVM"
      },
      "source": [
        "#### Compare And Detect Anomalies with `ML.TFDV_VALIDATE`\n",
        "\n",
        "Compare the distribution of any table to a base table and compute distance statistics to detect anomalies.  This is like using the TensorFlow's [TFX platform](https://www.tensorflow.org/tfx) with built-in [TensorFlow Data Validation (TFDV)](https://www.tensorflow.org/tfx/data_validation/get_started) module's [validate_statistics api](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/validate_statistics).\n",
        "\n",
        "These comparison can be made to training data for **skew** detection or previous serving data for **drift** detection."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {
        "tags": [],
        "id": "MfNNlhT2xLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b38707ad9b8a49dca9cc30948fb34f1d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "0b46380263d14ce4bf0e215e13dadae9",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "%%bigquery validate\n",
        "WITH\n",
        "    TRAIN AS (\n",
        "        SELECT * EXCEPT(splits, species)\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "        WHERE splits = 'TRAIN'\n",
        "    ),\n",
        "    SERVE AS (\n",
        "        SELECT * EXCEPT(instance_timestamp, species)\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "        WHERE DATE(instance_timestamp) = CURRENT_DATE()\n",
        "    )\n",
        "SELECT ML.TFDV_VALIDATE(\n",
        "    (SELECT * FROM ML.TFDV_DESCRIBE(TABLE TRAIN)),\n",
        "    (SELECT * FROM ML.TFDV_DESCRIBE(TABLE SERVE)),\n",
        "    'SKEW'\n",
        ") as validate"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 47,
      "metadata": {
        "tags": [],
        "id": "FXSFaJyexLVM"
      },
      "outputs": [],
      "source": [
        "validate = json.loads(validate['validate'].iloc[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 48,
      "metadata": {
        "tags": [],
        "id": "hdTKJFERxLVM"
      },
      "outputs": [],
      "source": [
        "#validate['anomaly_info']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2mcMluFlxLVM"
      },
      "source": [
        "#### Display Anomalies with `tfdv.display_anomalies`\n",
        "\n",
        "Use the [`tfdv.display_anomalies`](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/display_anomalies) tool to display a table of anomaly detections with descriptions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {
        "tags": [],
        "id": "_XsqcG1yxLVM"
      },
      "outputs": [],
      "source": [
        "if 'generate_legacy_feature_spec' in validate['baseline'].keys():\n",
        "    del validate['baseline']['generate_legacy_feature_spec']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {
        "tags": [],
        "id": "gO9o8ttKxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eAnomaly short description\u003c/th\u003e\n",
              "      \u003cth\u003eAnomaly long description\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003eFeature name\u003c/th\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e'flipper_length_mm'\u003c/th\u003e\n",
              "      \u003ctd\u003eHigh approximate Jensen-Shannon divergence between training and serving\u003c/td\u003e\n",
              "      \u003ctd\u003eThe approximate Jensen-Shannon divergence between training and serving is 0.950948 (up to six significant digits), above the threshold 0.3.\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e'body_mass_g'\u003c/th\u003e\n",
              "      \u003ctd\u003eHigh approximate Jensen-Shannon divergence between training and serving\u003c/td\u003e\n",
              "      \u003ctd\u003eThe approximate Jensen-Shannon divergence between training and serving is 0.33521 (up to six significant digits), above the threshold 0.3.\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                                                                   Anomaly short description  \\\n",
              "Feature name                                                                                   \n",
              "'flipper_length_mm'  High approximate Jensen-Shannon divergence between training and serving   \n",
              "'body_mass_g'        High approximate Jensen-Shannon divergence between training and serving   \n",
              "\n",
              "                                                                                                                                        Anomaly long description  \n",
              "Feature name                                                                                                                                                      \n",
              "'flipper_length_mm'  The approximate Jensen-Shannon divergence between training and serving is 0.950948 (up to six significant digits), above the threshold 0.3.  \n",
              "'body_mass_g'         The approximate Jensen-Shannon divergence between training and serving is 0.33521 (up to six significant digits), above the threshold 0.3.  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "tfdv.display_anomalies(\n",
        "    anomalies = json_format.ParseDict(validate, tfmd.proto.anomalies_pb2.Anomalies())\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7g3yFYYwxLVM"
      },
      "source": [
        "---\n",
        "## Understanding Monitoring Thresholds\n",
        "\n",
        "The `ML.VALIDATE_DATA_DRIFT`, `ML.VALIDATE_DATA_SKEW`, and `ML.TFDV_VALIDATE` functions use distance scores to quantify the change in the statistical distribution of features.  The methods of calculating the distance scores are:\n",
        "- [L-infinity distance](https://en.wikipedia.org/wiki/Chebyshev_distance) for categorical features\n",
        "- [Jensen-Shannon divergence](https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence) for numeric features.  Can also be use for categorical features.\n",
        "\n",
        "These methods are the same ones used by [TensorFlow Data Validation](https://www.tensorflow.org/tfx/guide/tfdv#training-serving_skew_detection) which is also used in [Vertex AI Model Monitoring](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview#calculating-skew-and-drift).\n",
        "\n",
        "This section aims to provide intuition on how these two metrics work.  In the data used for this workflow the feature `island` has three possible values: `Biscoe`, `Dream`, and `Torgersen`.  The proportion of data in each of these categories forms a distribution.  This will be used to visually show how both methods of calculating distance scores work."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "metadata": {
        "tags": [],
        "id": "-4PGw_zmxLVM"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jJNe8w9wxLVM"
      },
      "source": [
        "### Retrieve Distributions: Training \u0026 Serving\n",
        "\n",
        "Use the `ML.TFDV_DESCRIBE` function to get the statistics for the distribution of the `island` feature at both training and for serving."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {
        "tags": [],
        "id": "ibkgYFgbxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "04b40806f8a3473da989efbf395c0bdd",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "f84ae2ba2179483fbdf82debdf820b10",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "%%bigquery island_train\n",
        "SELECT *\n",
        "FROM ML.TFDV_DESCRIBE(\n",
        "    (\n",
        "        SELECT island\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "        WHERE splits = 'TRAIN'\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "metadata": {
        "tags": [],
        "id": "yOngMcvoxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "619fbcc9be6a4b88a4c8c7ecdadfae9a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ff5848c69d49492a81213a1565f5865c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "%%bigquery island_serve\n",
        "SELECT *\n",
        "FROM ML.TFDV_DESCRIBE(\n",
        "    (\n",
        "        SELECT island\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "metadata": {
        "tags": [],
        "id": "X739pzEXxLVM"
      },
      "outputs": [],
      "source": [
        "island_train = json.loads(island_train['dataset_feature_statistics_list'].iloc[0])\n",
        "island_serve = json.loads(island_serve['dataset_feature_statistics_list'].iloc[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "metadata": {
        "tags": [],
        "id": "ucfMedEExLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'avg_length': 6.092437,\n",
              " 'common_stats': {'avg_num_values': 1,\n",
              "  'max_num_values': '1',\n",
              "  'min_num_values': '1',\n",
              "  'num_non_missing': '238',\n",
              "  'num_values_histogram': {'buckets': [{'high_value': 1,\n",
              "     'low_value': 1,\n",
              "     'sample_count': 23.8},\n",
              "    {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "    {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "    {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "    {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "    {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "    {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "    {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "    {'high_value': 1, 'low_value': 1, 'sample_count': 23.8},\n",
              "    {'high_value': 1, 'low_value': 1, 'sample_count': 23.8}],\n",
              "   'type': 'QUANTILES'},\n",
              "  'tot_num_values': '238'},\n",
              " 'rank_histogram': {'buckets': [{'label': 'Biscoe', 'sample_count': 116},\n",
              "   {'high_rank': '1', 'label': 'Dream', 'low_rank': '1', 'sample_count': 86},\n",
              "   {'high_rank': '2',\n",
              "    'label': 'Torgersen',\n",
              "    'low_rank': '2',\n",
              "    'sample_count': 36}]},\n",
              " 'top_values': [{'frequency': 116, 'value': 'Biscoe'},\n",
              "  {'frequency': 86, 'value': 'Dream'},\n",
              "  {'frequency': 36, 'value': 'Torgersen'}],\n",
              " 'unique': '3'}"
            ]
          },
          "execution_count": 55,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "island_train['datasets'][0]['features'][0]['string_stats']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "metadata": {
        "tags": [],
        "id": "YbNGJsYXxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'label': 'Biscoe', 'sample_count': 116},\n",
              " {'high_rank': '1', 'label': 'Dream', 'low_rank': '1', 'sample_count': 86},\n",
              " {'high_rank': '2', 'label': 'Torgersen', 'low_rank': '2', 'sample_count': 36}]"
            ]
          },
          "execution_count": 56,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "island_train_stats = island_train['datasets'][0]['features'][0]['string_stats']['rank_histogram']['buckets']\n",
        "island_train_stats"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "metadata": {
        "tags": [],
        "id": "vX9osX7DxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'label': 'Biscoe', 'sample_count': 1054},\n",
              " {'high_rank': '1', 'label': 'Dream', 'low_rank': '1', 'sample_count': 682},\n",
              " {'high_rank': '2',\n",
              "  'label': 'Torgersen',\n",
              "  'low_rank': '2',\n",
              "  'sample_count': 403}]"
            ]
          },
          "execution_count": 57,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "island_serve_stats = island_serve['datasets'][0]['features'][0]['string_stats']['rank_histogram']['buckets']\n",
        "island_serve_stats"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p65FpXdMxLVM"
      },
      "source": [
        "### Combine Distributions In DataFrame"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "metadata": {
        "tags": [],
        "id": "h7UjSA8ZxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003etrain_n\u003c/th\u003e\n",
              "      \u003cth\u003eserve_n\u003c/th\u003e\n",
              "      \u003cth\u003etrain_pct\u003c/th\u003e\n",
              "      \u003cth\u003eserve_pct\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003eBiscoe\u003c/th\u003e\n",
              "      \u003ctd\u003e116\u003c/td\u003e\n",
              "      \u003ctd\u003e1054\u003c/td\u003e\n",
              "      \u003ctd\u003e0.487395\u003c/td\u003e\n",
              "      \u003ctd\u003e0.492754\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003eDream\u003c/th\u003e\n",
              "      \u003ctd\u003e86\u003c/td\u003e\n",
              "      \u003ctd\u003e682\u003c/td\u003e\n",
              "      \u003ctd\u003e0.361345\u003c/td\u003e\n",
              "      \u003ctd\u003e0.318841\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003eTorgersen\u003c/th\u003e\n",
              "      \u003ctd\u003e36\u003c/td\u003e\n",
              "      \u003ctd\u003e403\u003c/td\u003e\n",
              "      \u003ctd\u003e0.151261\u003c/td\u003e\n",
              "      \u003ctd\u003e0.188406\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "           train_n  serve_n  train_pct  serve_pct\n",
              "Biscoe         116     1054   0.487395   0.492754\n",
              "Dream           86      682   0.361345   0.318841\n",
              "Torgersen       36      403   0.151261   0.188406"
            ]
          },
          "execution_count": 58,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "index = [i['label'] for i in island_train_stats]\n",
        "train = [i['sample_count'] for i in island_train_stats]\n",
        "serve = [island_serve_stats[index.index(k)]['sample_count'] for k in [i['label'] for i in island_serve_stats]]\n",
        "stats = pd.DataFrame(\n",
        "    {\n",
        "        'train_n': train,\n",
        "        'serve_n': serve,\n",
        "        'train_pct': [x/sum(train) for x in train],\n",
        "        'serve_pct': [x/sum(serve) for x in serve]\n",
        "    },\n",
        "    index = index\n",
        ")\n",
        "stats"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fEMCienaxLVM"
      },
      "source": [
        "### Visualize Distributions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "metadata": {
        "tags": [],
        "id": "edBQnK5QxLVM"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABLy0lEQVR4nO3de1yP9/8/8Me78/FdlI46IIdyKEp9yvkjyjBsJocpMZ+Z8/ryEbZyXD5baxkmc8gwH2xz2tAHWZvThshZJhJScioVxfv9+v3h17W9VfQmrrLH/XZ73269r+t1va7ndXXp/fC6Dm+FEEKAiIiISCY6chdAREREf28MI0RERCQrhhEiIiKSFcMIERERyYphhIiIiGTFMEJERESyYhghIiIiWTGMEBERkaz05C6gKtRqNbKzs2Fubg6FQiF3OURERFQFQgjcu3cPDg4O0NGpfPyjVoSR7OxsODk5yV0GERERPYcrV66gfv36lc6vFWHE3NwcwOONUSqVMldDREREVVFQUAAnJyfpc7wytSKMlJ2aUSqVDCNERES1zLMuseAFrERERCQrhhEiIiKSFcPIa2TRokVwdXWFkZER/Pz8cOjQoSott27dOigUCvTt21djem5uLoYNGwYHBweYmJggODgYf/zxhzT/9u3bGDduHJo2bQpjY2M4Oztj/PjxyM/Pl9qsXLkSCoWiwteNGzcAACkpKRXOz8nJqbDeefPmQaFQYOLEidrtICIiqpFqxTUj9Gzr169HREQEEhIS4Ofnh/j4eAQFBSE9PR02NjaVLpeZmYlJkyahQ4cOGtOFEOjbty/09fWxZcsWKJVKxMXFITAwEGfOnIGpqSmys7ORnZ2N2NhYeHh44PLlyxg1ahSys7Px/fffAwBCQkIQHBys0fewYcPw4MGDcnWlp6drXBNUUd2HDx/GkiVL0KpVK633ERHVHCqVCg8fPpS7DHpBurq60NPTe+HHbiiEEKKaanppCgoKYGFhgfz8fF7AWgk/Pz+0bdsWCxcuBPD42SxOTk4YN24cIiMjK1xGpVKhY8eOGD58OPbu3Yu7d+9i8+bNAIDz58+jadOmOHXqFJo3by71aWdnh08++QTvvfdehX1+9913ePfdd1FUVAQ9vfJZNy8vD46Ojli+fDmGDh0K4PHISJcuXXDnzh1YWlpWuo2FhYVo06YNvvrqK8yZMwdeXl6Ij4+v4h4iopqisLAQV69eRS34+KEqMDExgb29PQwMDMrNq+rn93ONjCxatAifffYZcnJy4OnpiQULFsDX17fCtitXrkR4eLjGNENDQzx48OB5Vi0bbbb5r9atW4dBgwahT58+0gc98PgfY2RkJDZv3oxbt26hQYMGGD9+PEaNGgXg8YhFgwYNKuxzw4YNeOedd3Dr1i0MGTIEx48fR05ODjIyMgAAn3zyCZRKJQIDA3Hw4EEsWrQICxcuRGZmJpydnTF9+nSEhoZi1qxZsLGxga6uLr755hsAf17xrK+vDwAwMjKS1qujowNDQ0Ps27ev0jBSdsBVFEQAYNWqVTAxMUH//v3LzfPy8kJJSQlatGiBGTNmoF27dhrzx4wZg549eyIwMBBz5sypsH8iqtlUKhWuXr0KExMT1KtXjw+yrMWEECgtLUVeXh4uXbqExo0bP/XBZs/qTCvr1q0TBgYGYsWKFeL06dNi5MiRwtLSUuTm5lbYPjExUSiVSnH9+nXplZOTo9U68/PzBQCRn5+vbbnVQtttLnPp0iXh6OgoOnToIPr06aMxb+TIkaJRo0bi559/FpcuXRJLliwRurq6YsuWLUIIIR49eqSxz65fvy5mzpwpzMzMxL1794QQQty+fVt89dVXYvv27QKA+PLLL0XTpk3FoEGDhBBCTJ48Wbi6ugpzc3Oxbt06kZGRIf773/8KMzMzMW/ePOHo6Cjy8vJEYmKi0NfXF0FBQdK6rly5IpydncU777wjbt++LUpKSsS8efMEANG9e/cKtzcvL084OzuLadOmVbpP3N3dxQcffKAx7dy5cyIhIUEcOXJE7N+/X4SHhws9PT2Rmpoqtfnvf/8rWrRoIe7fvy+EEKJTp05iwoQJT93/RFTz3L9/X5w5c0YUFxfLXQpVk6KiInHmzBnp7/NfVfXzW+sw4uvrK8aMGSO9V6lUwsHBQcTExFTYPjExUVhYWGi7Gg1yhxFtt1mIx2EiICBALFu2TISFhZULI82bNxezZs3SmNamTRsxffr0Svv08vISw4cPLzf92rVrAoA4cOCAmD9/vqhfv74Q4nEYMTMzE5MmTdJoP3bsWGFoaCi2b98uhBBSGHmyxiNHjghPT08BQOjq6oqgoCDRo0cPERwcXK6G/Px84evrK4KDg0VpaWmF9R84cEAAEEeOHKl0G8t07NhRvPvuu0IIIbKysoSNjY04fvy4NJ9hhKh2KgsjFX1wUe30tN9pVT+/tRpPKS0tRWpqKgIDA6VpOjo60umAyhQWFsLFxQVOTk7o06cPTp8+/dT1lJSUoKCgQOMll+fd5rJTICNGjKhwfkBAALZu3Ypr165BCIGff/4Z58+fR/fu3Stsn5qairS0tAr7s7a2hq6uLs6ePYuNGzeiU6dOAB7fDaOrq6txqgUA7t+/j5KSEvTu3Rt6enoYMWIEHj58iC1btkChUCAwMBCnT5+Gt7c30tLScPfuXVy/fh1JSUm4desWGjZsqNHfvXv3EBwcDHNzc2zatEk6xfOkZcuWwcvLC97e3pXutzK+vr64cOGCtO03btxAmzZtoKenBz09Pfzyyy/48ssvoaenB5VK9cz+iIio5tIqjNy8eRMqlQq2trYa021tbSu9DbNp06ZYsWIFtmzZgjVr1kCtViMgIABXr16tdD0xMTGwsLCQXnJ+L83zbPO+ffuwfPlyLF26tNJ+FyxYAA8PD9SvXx8GBgYIDg7GokWL0LFjxwrbL1++HO7u7ggICCg3LywsDGq1GiNGjIBSqcSyZcugVquRnJyM1q1bY9myZUhNTYUQAkeOHMGPP/4IANi1axfS0tLwzTffoE2bNvD19cWKFStgaGio8TuysLBAvXr18Mcff+DIkSPo06ePtO6CggJ0794dBgYG2Lp1a7ngU6awsBAbNmyoNJw9KS0tDfb29gCArl274uTJk0hLS5NePj4+GDJkCNLS0qCrq1ulPomIqGZ66bf2+vv7w9/fX3ofEBAAd3d3LFmyBLNnz65wmalTpyIiIkJ6X/Zs+9rg3r17GDp0KJYuXQpra+tK2y1YsAC//fYbtm7dChcXF/z6668YM2YMHBwcNEZhgMcjGWvXrsXHH39cYV9ffPEF2rZti8jISKSmpiI8PBxKpRJFRUVITEzErFmz0LZtWwCAvb09hg0bhk8//RQeHh6wtbVFixYtsHv3bty9exfh4eF499134e7ujgkTJmDcuHFwdnbGyZMnMWHCBPTt21cavSkLIsXFxVizZo3GKFa9evU0QsL69evx6NEjvPvuu+Xqj4+PR4MGDdC8eXM8ePAAy5Ytw549e7Bz504Aj7+bqEWLFhrLmJqawsrKqtx0IqqdXCO3vdL1Zc7r+UrX91eurq6YOHEin5X0F1qFkbLTAbm5uRrTc3NzYWdnV6U+9PX10bp1a2kIviKGhoYwNDTUprSXRtttzsjIQGZmJnr37i1NU6vVAAA9PT2kp6fDwcEB06ZNw6ZNm9Cz5+N/EK1atUJaWhpiY2PLhZHvv/8excXFCA0NrbBGOzs7REREwMDAAHPmzMG6devQunVrJCUlwdXVFStWrEBGRgZsbGywbt06fP311zA3N0e9evUq7K/sd3Tx4kUMHToUubm5sLe3R2hoqEYgOnr0KH7//XcAgJubm0Yfly5dgqurq/R++fLleOuttyq8dbe0tBT/93//h2vXrsHExAStWrXC7t270aVLlwrrIyJ61Tp37lxtjxM4fPgwTE1NX7yoalKd2/a8tAojBgYG8Pb2RnJysvS0zrLTAWPHjq1SHyqVCidPnsQbb7yhdbFy0HabmzVrhpMnT2pM++ijj3Dv3j3Mnz8fTk5OePDgAR4+fFjuFihdXV0puPzV8uXL8eabb1YaHsqMHTsWrVq1QqdOnbBx40aNMPDLL79IP69btw69evXSWP/KlSuln//6O4qLi6t0fZ07d67ycwIOHDhQ6bx///vf+Pe//12lfsqkpKRo1Z6I6GUSQkClUlX6WIO/etbf8r8jrW8IjoiIwNKlS/HNN9/g7Nmz+OCDD1BUVCQ9SyQ0NBRTp06V2s+aNQs7d+7ExYsXcfToUbz77ru4fPlypc+pqIm02WYjIyO0aNFC42VpaSmdajAwMIBSqUSnTp0wefJkpKSk4NKlS1i5ciVWrVqFfv36aaz7woUL+PXXXyvcX9u3b0diYiJOnTqFzMxMbNu2DaNGjUK7du2kIHL+/HmsWbMGf/zxBw4dOoSBAwfi1KlT+OSTT6R+XoffERHRyzJs2DD88ssvmD9/vvR1FWVfdbFjxw54e3tLz2DKyMhAnz59YGtrCzMzM7Rt2xa7d+/W6M/V1VVjFEKhUGDZsmXo168fTExM0LhxY2zdurVKtZV9nca2bdvQqlUrGBkZ4R//+AdOnTql0W7//v3o3LkzTExMUKdOHQQFBeHOnTsVbltmZuaL7jKtaX3NSEhICPLy8hAVFYWcnBx4eXkhKSlJusAzKytL43/cd+7cwciRI5GTk4M6derA29sbBw4cgIeHR/VtxUum7TZXxbp16zB16lQMGTIEt2/fhouLC+bOnSs99KzMihUrUL9+/QrvsjE2NsbSpUvx4YcfoqSkBE5OTnjrrbc0nriqUqnw+eefIz09Hfr6+uhiX4wDgwzhutJTanMn6QFGxj1ETqFAHSMFvB10cSDUEB4b/Mut85WZkf/sNkREr8D8+fNx/vx5tGjRArNmzQIA6a7QyMhIxMbGomHDhqhTpw6uXLmCN954A3PnzoWhoSFWrVqF3r17Iz09Hc7OzpWuY+bMmfj000/x2WefYcGCBRgyZAguX76MunXrVqnGyZMnY/78+bCzs8O0adPQu3dvnD9/Hvr6+khLS0PXrl0xfPhwzJ8/H3p6evj555+hUqkq3DY5Rm74OPi/mxkWcldQNQwjRK+lBw8e4NKlS2jQoIHG3Xc1/QLWJ6+rKPsai82bN2vcYViRFi1aYNSoUdKp/ScvYFUoFPjoo4+kmzqKiopgZmaGHTt2lPturyeV1bFu3TqEhIQAePwlpvXr18fKlSsxYMAADB48GFlZWdi3b1+Vtk1blf1Ogap/fvNbe4mIiJ6Tj4+PxvvCwkJMmjQJ7u7usLS0hJmZGc6ePYusrKyn9vPXL/80NTWFUqmUvtm8Kv5612rdunXRtGlTnD17FgCkkZGajN/aS0RE9JyevCtm0qRJ2LVrF2JjY+Hm5gZjY2P0798fpaWlT+3nyYdFKhSKCm9oeB7GxsbV0s/LxDBSjV71MOPzyKz4mWRERPQUBgYGVXra8/79+zFs2DDpZoTCwsJXckHob7/9Jl2TcufOHZw/fx7u7u4AHo+6JCcnY+bMmRUuW9Vte5l4moaIiOgZXF1d8fvvvyMzMxM3b96sdNSicePG2LhxI9LS0nD8+HEMHjy42kY4nmbWrFlITk7GqVOnMGzYMFhbW0uPo5g6dSoOHz6M0aNH48SJEzh37hwWL16MmzdvarVtLxNHRoiISHZyPhG1KiZNmoSwsDB4eHjg/v37SExMrLBdXFwchg8fjoCAAFhbW2PKlCmv5PvV5s2bhwkTJuCPP/6Al5cXfvzxRxgYGAAAmjRpgp07d2LatGnw9fWFsbEx/Pz8MGjQoAq37cmHVr4KvJumGtWO0zSD5S6hang3DdFr6Wl3XpD2yu6muXPnToVPuH4VeDcNERER1XoMI0RERDXUqFGjYGZmVuHryYdk1ma8ZoSIiKiGmjVrFiZNmlThPKVSCRsbmyp/R1hNxjBCRERUQ9nY2MDGxkbuMl46nqYhIiIiWTGMEBERkawYRoiIiEhWDCNEREQkK4YRIiIikhXDCBEREcmKt/YSEZH8Zli84vXxKyeeJTMzEw0aNMCxY8fg5eX1UtfFkREiIqJX7OHDh3KXUKMwjBAREVXB999/j5YtW8LY2BhWVlYIDAxEUVERAGDZsmVwd3eHkZERmjVrhq+++kpaLjMzEwqFAuvXr0enTp1gZGSExYsXw9jYGDt27NBYx6ZNm2Bubo7i4mIAwJUrVzBgwABYWlqibt266NOnDzIzM6tU77Bhw9C3b1/MnDkT9erVg1KpxKhRo1BaWiq1UavV+PTTT+Hm5gZDQ0M4Oztj7ty5AIAGDRoAAFq3bg2FQoHOnTs/7657Jp6mISIieobr169j0KBB+PTTT9GvXz/cu3cPe/fuhRAC3377LaKiorBw4UK0bt0ax44dw8iRI2FqaoqwsDCpj8jISHz++edo3bo1jIyMsHfvXqxduxY9evSQ2nz77bfo27cvTExM8PDhQwQFBcHf3x979+6Fnp4e5syZg+DgYJw4cQIGBgbPrDs5ORlGRkZISUlBZmYmwsPDYWVlJQWOqVOnYunSpfjiiy/Qvn17XL9+HefOnQMAHDp0CL6+vti9ezeaN29epfU9L4YRIiKiZ7h+/ToePXqEt956Cy4uLgCAli1bAgCio6Px+eef46233gLweEThzJkzWLJkiUYYmThxotQGAIYMGYKhQ4eiuLgYJiYmKCgowLZt27Bp0yYAwPr166FWq7Fs2TIoFAoAQGJiIiwtLZGSkoLu3bs/s24DAwOsWLECJiYmaN68OWbNmoXJkydj9uzZKCoqwvz587Fw4UKpzkaNGqF9+/YAgHr16gEArKysYGdn90L771kYRoiIiJ7B09MTXbt2RcuWLREUFITu3bujf//+MDAwQEZGBkaMGIGRI0dK7R89egQLC82Lcn18fDTev/HGG9DX18fWrVsxcOBA/PDDD1AqlQgMDAQAHD9+HBcuXIC5ubnGcg8ePEBGRkaV6zYxMZHe+/v7o7CwEFeuXEFubi5KSkrQtWtXrfbFy8AwQkRE9Ay6urrYtWsXDhw4gJ07d2LBggWYPn06fvzxRwDA0qVL4efnV26ZvzI1NdV4b2BggP79+2Pt2rUYOHAg1q5di5CQEOjpPf5oLiwshLe3N7799tty9ZSNWrwIY2PjF+6jujCMEBERVYFCoUC7du3Qrl07REVFwcXFBfv374eDgwMuXryIIUOGaN3nkCFD0K1bN5w+fRp79uzBnDlzpHlt2rTB+vXrYWNjA6VS+Vw1Hz9+HPfv35eCx2+//QYzMzM4OTmhtLQUxsbGSE5OxnvvvVdu2bJrRFQq1XOtWxu8m4aIiOgZfv/9d3zyySc4cuQIsrKysHHjRuTl5cHd3R0zZ85ETEwMvvzyS5w/fx4nT55EYmIi4uLintlvx44dYWdnhyFDhqBBgwYaoytDhgyBtbU1+vTpg7179+LSpUtISUnB+PHjcfXq1SrVXVpaihEjRuDMmTPYvn07oqOjMXbsWOjo6MDIyAhTpkzBv//9b6xatQoZGRn47bffsHz5cgCAjY0NjI2NkZSUhNzcXOTnv7xns3BkhIiI5FfDH0KmVCrx66+/Ij4+HgUFBXBxccHnn38u3QljYmKCzz77DJMnT4apqSlatmyJiRMnPrNfhUIh3aUTFRWlMc/ExAS//vorpkyZgrfeegv37t2Do6MjunbtWuWRkq5du6Jx48bo2LEjSkpKMGjQIMyYMUOa//HHH0NPTw9RUVHIzs6Gvb09Ro0aBQDQ09PDl19+iVmzZiEqKgodOnRASkpKldarLYUQQryUnqtRQUEBLCwskJ+f/9xDVa+Ca+Q2uUt4pkyjwXKXUDU1/A8TET2fBw8e4NKlS2jQoAGMjIzkLue1NmzYMNy9exebN29+qet52u+0qp/fPE1DREREsuJpGiIiolrIzMys0nlPPtm1pmMYISIiqoXS0tIqnefo6IgOHTq8umJeEMMIERFRLeTm5iZ3CdWG14wQEdErVwvunaAqqo7fJcMIERG9MmVPJf3rN8dS7Vb2DcP6+vrP3QdP0xAR0Sujp6cHExMT5OXlQV9fHzo6/D9xbSWEQHFxMW7cuAFLS8tyj7/XBsMIERG9MgqFAvb29rh06RIuX74sdzlUDSwtLV/4W30ZRoiI6JUyMDBA48aNearmNaCvr/9CIyJlGEaIiOiVK/tuFCKAF7ASERGRzBhGiIiISFYMI0RERCQrhhEiIiKSFcMIERERyYphhIiIiGTFMEJERESyYhghIiIiWTGMEBERkawYRoiIiEhWDCNEREQkK4YRIiIikhXDCBEREcmKYYSIiIhkxTBCREREsmIYISIiIlkxjBAREZGsGEaIiIhIVgwjREREJCuGESIiIpIVwwgRERHJimGEiIiIZMUwQkRERLJ6rjCyaNEiuLq6wsjICH5+fjh06FCVllu3bh0UCgX69u37PKslIiKi15DWYWT9+vWIiIhAdHQ0jh49Ck9PTwQFBeHGjRtPXS4zMxOTJk1Chw4dnrtYIiIiev1oHUbi4uIwcuRIhIeHw8PDAwkJCTAxMcGKFSsqXUalUmHIkCGYOXMmGjZs+EIFExER0etFqzBSWlqK1NRUBAYG/tmBjg4CAwNx8ODBSpebNWsWbGxsMGLEiOevlIiIiF5Leto0vnnzJlQqFWxtbTWm29ra4ty5cxUus2/fPixfvhxpaWlVXk9JSQlKSkqk9wUFBdqUSURERLXIS72b5t69exg6dCiWLl0Ka2vrKi8XExMDCwsL6eXk5PQSqyQiIiI5aTUyYm1tDV1dXeTm5mpMz83NhZ2dXbn2GRkZyMzMRO/evaVparX68Yr19JCeno5GjRqVW27q1KmIiIiQ3hcUFDCQEBERvaa0CiMGBgbw9vZGcnKydHuuWq1GcnIyxo4dW659s2bNcPLkSY1pH330Ee7du4f58+dXGjAMDQ1haGioTWlERERUS2kVRgAgIiICYWFh8PHxga+vL+Lj41FUVITw8HAAQGhoKBwdHRETEwMjIyO0aNFCY3lLS0sAKDediIiI/p60DiMhISHIy8tDVFQUcnJy4OXlhaSkJOmi1qysLOjo8MGuREREVDUKIYSQu4hnKSgogIWFBfLz86FUKuUup1KukdvkLuGZMo0Gy11C1czIl7sCIiJ6QVX9/OYQBhEREcmKYYSIiIhkxTBCREREsmIYISIiIlkxjBAREZGsGEaIiIhIVgwjREREJCuGESIiIpIVwwhRLbVo0SK4urrCyMgIfn5+OHToUKVtN27cCB8fH1haWsLU1BReXl5YvXp1uXZnz57Fm2++CQsLC5iamqJt27bIysqS5n/99dfo3LkzlEolFAoF7t69W66PN998E87OzjAyMoK9vT2GDh2K7OzsCuu6cOECzM3Npa+JKLNy5UooFAqNl5GRUdV2DBHVOgwjRLXQ+vXrERERgejoaBw9ehSenp4ICgrCjRs3Kmxft25dTJ8+HQcPHsSJEycQHh6O8PBw/O9//5PaZGRkoH379mjWrBlSUlJw4sQJfPzxxxohoLi4GMHBwZg2bVqltXXp0gUbNmxAeno6fvjhB2RkZKB///7l2j18+BCDBg1Chw4dKuxHqVTi+vXr0uvy5ctV3T1EVMvwcfDViI+Dr0Z8HPxT+fn5oW3btli4cCGAx9+e7eTkhHHjxiEyMrJKfbRp0wY9e/bE7NmzAQADBw6Evr5+hSMmT0pJSUGXLl1w586dcqMaT9q6dSv69u2LkpIS6OvrS9OnTJmC7OxsdO3aFRMnTtQYZVm5cmW5aURU+/Bx8ESvqdLSUqSmpiIwMFCapqOjg8DAQBw8ePCZywshkJycjPT0dHTs2BHA4zCzbds2NGnSBEFBQbCxsYGfnx82b978QrXevn0b3377LQICAjSCyJ49e/Ddd99h0aJFlS5bWFgIFxcXODk5oU+fPjh9+vQL1UJENRfDCFEtc/PmTahUKumbssvY2toiJyen0uXy8/NhZmYGAwMD9OzZEwsWLEC3bt0AADdu3EBhYSHmzZuH4OBg7Ny5E/369cNbb72FX375Resap0yZAlNTU1hZWSErKwtbtmyR5t26dQvDhg3DypUrK/2fUtOmTbFixQps2bIFa9asgVqtRkBAAK5evap1LURU8zGMEP1NmJubIy0tDYcPH8bcuXMRERGBlJQUAI9HRgCgT58++PDDD+Hl5YXIyEj06tULCQkJWq9r8uTJOHbsGHbu3AldXV2Ehoai7IzwyJEjMXjwYGlUpiL+/v4IDQ2Fl5cXOnXqhI0bN6JevXpYsmSJ9htORDWentwFEJF2rK2toauri9zcXI3pubm5sLOzq3Q5HR0duLm5AQC8vLxw9uxZxMTEoHPnzrC2toaenh48PDw0lnF3d8e+ffueq0Zra2s0adIE7u7ucHJywm+//QZ/f3/s2bMHW7duRWxsLIDHp43UajX09PTw9ddfY/jw4eX609fXR+vWrXHhwgWtayGimo8jI0S1jIGBAby9vZGcnCxNU6vVSE5Ohr+/f5X7UavVKCkpkfps27Yt0tPTNdqcP38eLi4uL1Rv2ahL2boOHjyItLQ06TVr1ixp1KZfv34V9qFSqXDy5EnY29u/UC1EVDNxZISoFoqIiEBYWBh8fHzg6+uL+Ph4FBUVITw8HAAQGhoKR0dHxMTEAABiYmLg4+ODRo0aoaSkBNu3b8fq1auxePFiqc/JkycjJCQEHTt2RJcuXZCUlIQff/xROpUDADk5OcjJyZFGKE6ePAlzc3M4Ozujbt26+P3333H48GG0b98ederUQUZGBj7++GM0atRICkru7u4a23LkyBHo6OigRYsW0rRZs2bhH//4B9zc3HD37l189tlnuHz5Mt57772Xsj+JSF4MI0S1UEhICPLy8hAVFYWcnBx4eXkhKSlJuqg1KysLOjp/DnwWFRVh9OjRuHr1KoyNjdGsWTOsWbMGISEhUpt+/fohISEBMTExGD9+PJo2bYoffvgB7du3l9okJCRg5syZ0vuy6z4SExMxbNgwmJiYYOPGjYiOjkZRURHs7e0RHByMjz76CIaGhlXevjt37mDkyJHIyclBnTp14O3tjQMHDpQ7jURErwc+Z6Qa8Tkj1YjPGSEiqvX4nBEiIiKqFRhGiIiISFa8ZoToNVYbTh0CQOa8nnKXQEQy4sgIERERyYphhIiIiGTFMEJERESyYhghIiIiWTGMEBERkawYRoiIiEhWDCNEREQkK4YRIiIikhXDCBEREcmKYYSIiIhkxTBCREREsmIYISIiIlkxjBAREZGsGEaIiIhIVgwjREREJCuGESIiIpIVwwgRERHJimGEiF6ae0d/wtXFw3E5th+ur4pASXZ6pW03btwIHx8fWFpawtTUFF5eXli9enW5Nt27d4eVlRUUCgXS0tLK9ZOTk4OhQ4fCzs4OpqamaNOmDX744QeNNufPn0efPn1gbW0NpVKJ9u3b4+eff66wrlu3bqF+/fpQKBS4e/euNP369esYPHgwmjRpAh0dHUycOLHK+4WINDGMENFLUXT2V9zeswyW7QbBfth8GNg0wI0NUVAV3a2wfd26dTF9+nQcPHgQJ06cQHh4OMLDw/G///3vzz6LitC+fXv85z//qXS9oaGhSE9Px9atW3Hy5Em89dZbGDBgAI4dOya16dWrFx49eoQ9e/YgNTUVnp6e6NWrF3Jycsr1N2LECLRq1arc9JKSEtSrVw8fffQRPD09tdgzRPQkhhEieikKDm+GuWcQzFp1g4G1M+oGjYFC3xCFJ3dV2L5z587o168f3N3d0ahRI0yYMAGtWrXCvn37pDZDhw5FVFQUAgMDK13vgQMHMG7cOPj6+qJhw4b46KOPYGlpidTUVADAzZs38ccffyAyMhKtWrVC48aNMW/ePBQXF+PUqVMafS1evBh3797FpEmTyq3H1dUV8+fPR2hoKCwsLJ5nFxHR/8cwQkTVTqgeojTnAoxcvKRpCoUOjFy9UHLt3LOXFwLJyclIT09Hx44dtVp3QEAA1q9fj9u3b0OtVmPdunV48OABOnfuDACwsrJC06ZNsWrVKhQVFeHRo0dYsmQJbGxs4O3tLfVz5swZzJo1C6tWrYKODv9UEr1MenIXQESvH1VxASDU0DW11Jiua2KJh7euVrpcfn4+HB0dUVJSAl1dXXz11Vfo1q2bVuvesGEDQkJCYGVlBT09PZiYmGDTpk1wc3MDACgUCuzevRt9+/aFubk5dHR0YGNjg6SkJNSpUwfA41MwgwYNwmeffQZnZ2dcvHhRux1ARFphGCGiGsPc3BxpaWkoLCxEcnIyIiIi0LBhQ2lUoyo+/vhj3L17F7t374a1tTU2b96MAQMGYO/evWjZsiWEEBgzZgxsbGywd+9eGBsbY9myZejduzcOHz4Me3t7TJ06Fe7u7nj33Xdf3sYSkYRhhIiqna6JElDolLtYVVV8F7qmdSpdTkdHRxrB8PLywtmzZxETE1PlMJKRkYGFCxfi1KlTaN68OQDA09MTe/fuxaJFi5CQkIA9e/bgp59+wp07d6BUKgEAX331FXbt2oVvvvkGkZGR2LNnD06ePInvv/8ewOPTRgBgbW2N6dOnY+bMmdrsDiJ6BoYRIqp2Cl19GNi54cHl4zBp4g8AEEKNB5nHYe7dq8r9qNVqlJSUVLl9cXExAJS7xkNXVxdqtfqpbXR0dKQ2P/zwA+7fvy/NO3z4MIYPH469e/eiUaNGVa6HiKqGYYSIXgpl2764ue0LGNg1hqF9ExQc2QLx8AHMWj6+E+bmT59D19wKdToNAwDExMTAx8cHjRo1QklJCbZv347Vq1dj8eLFUp+3b99GVlYWsrOzAQDp6Y+fW2JnZwc7Ozs0a9YMbm5ueP/99xEbGwsrKyts3rwZu3btwk8//QQA8Pf3R506dRAWFoaoqCgYGxtj6dKluHTpEnr27AkA5QLHzZs3AQDu7u6wtLSUppc956SwsBB5eXlIS0uDgYEBPDw8qndnEr3mGEaI6KUwde8IVXE+7u5bA1XRHRjYNITNgFnSaZpHBXmA4s/RiaKiIowePRpXr16FsbExmjVrhjVr1iAkJERqs3XrVoSHh0vvBw4cCACIjo7GjBkzoK+vj+3btyMyMhK9e/dGYWEh3Nzc8M033+CNN94A8PhUS1JSEqZPn45//vOfePjwIZo3b44tW7Zo/byQ1q1bSz+npqZi7dq1cHFxQWZmptb7i+jvTCHKTobWYAUFBbCwsEB+fr50jrcmco3cJncJz5RpNFjuEqpmRr7cFbwWasMxCQCZ83rKXQIRvQRV/fzmzfNEREQkK4YRIiIikhWvGSEi+c2oBY9T56lDopeGIyNEREQkK4YRIiIikhXDCBEREcmKYYSIiIhkxTBCREREsmIYISIiIlkxjBAREZGsGEaIiIhIVs8VRhYtWgRXV1cYGRnBz88Phw4dqrTtxo0b4ePjA0tLS5iamsLLywurV69+7oKJiIjo9aJ1GFm/fj0iIiIQHR2No0ePwtPTE0FBQbhx40aF7evWrYvp06fj4MGDOHHiBMLDwxEeHo7//e9/L1w8ERER1X5ah5G4uDiMHDkS4eHh8PDwQEJCAkxMTLBixYoK23fu3Bn9+vWDu7s7GjVqhAkTJqBVq1bYt2/fCxdPREREtZ9WYaS0tBSpqakIDAz8swMdHQQGBuLgwYPPXF4IgeTkZKSnp6Njx46VtispKUFBQYHGi4iIiF5PWoWRmzdvQqVSwdbWVmO6ra0tcnJyKl0uPz8fZmZmMDAwQM+ePbFgwQJ069at0vYxMTGwsLCQXk5OTtqUSURERLXIK7mbxtzcHGlpaTh8+DDmzp2LiIgIpKSkVNp+6tSpyM/Pl15Xrlx5FWUSERGRDPS0aWxtbQ1dXV3k5uZqTM/NzYWdnV2ly+no6MDNzQ0A4OXlhbNnzyImJgadO3eusL2hoSEMDQ21KY2IiIhqKa1GRgwMDODt7Y3k5GRpmlqtRnJyMvz9/avcj1qtRklJiTarJiIioteUViMjABAREYGwsDD4+PjA19cX8fHxKCoqQnh4OAAgNDQUjo6OiImJAfD4+g8fHx80atQIJSUl2L59O1avXo3FixdX75YQERFRraR1GAkJCUFeXh6ioqKQk5MDLy8vJCUlSRe1ZmVlQUfnzwGXoqIijB49GlevXoWxsTGaNWuGNWvWICQkpPq2goiIiGothRBCyF3EsxQUFMDCwgL5+flQKpVyl1Mp18htcpfwTJlGg+UuoWpm5MtdwWuhNhyTQC05LnlMEmmtqp/f/G4aIiIikhXDCBEREcmKYYSIiIhkxTBCREREsmIYISIiIlkxjBAREZGsGEaIiIhIVgwjREREJCuGESIiIpIVwwgRERHJimGEiIiIZMUwQkRERLJiGCEiIiJZMYwQERGRrBhGiIiISFYMI0RERCQrhhEiIiKSFcMIERERyYphhIiIiGTFMEJERESyYhghIiIiWTGMEBERkawYRoiIiEhWDCNEREQkK4YRIiIikhXDCBEREcmKYYSIiIhkxTBCREREsmIYISIiIlkxjBAREZGsGEaIiIhIVgwjREREJCuGESIiIpIVwwgRERHJimGEiIiIZMUwQkRERLJiGCEiIiJZMYwQERGRrBhGiIiISFYMI0RERCQrhhEiIiKSFcMIERERyYphhIiIiGTFMEJERESyYhghIiIiWTGMEBERkawYRoiIiEhWDCNEREQkK4YRIiIikhXDCBEREcmKYYSIiIhkxTBCREREsmIYISIiIlkxjBAREZGsGEaIiIhIVgwjREREJCuGESIiIpIVwwgRERHJimGEiIiIZMUwQkRERLJ6rjCyaNEiuLq6wsjICH5+fjh06FClbZcuXYoOHTqgTp06qFOnDgIDA5/anoiI6GXS5jPs9OnTePvtt+Hq6gqFQoH4+PhybVQqFT7++GM0aNAAxsbGaNSoEWbPng0hhNSmsLAQY8eORf369WFsbAwPDw8kJCRUuE4hBHr06AGFQoHNmzdrzMvKykLPnj1hYmICGxsbTJ48GY8ePdJoU1JSgunTp8PFxQWGhoZwdXXFihUrqr6DZKCn7QLr169HREQEEhIS4Ofnh/j4eAQFBSE9PR02Njbl2qekpGDQoEEICAiAkZER/vOf/6B79+44ffo0HB0dq2UjiIiIqkLbz7Di4mI0bNgQ77zzDj788MMK+/zPf/6DxYsX45tvvkHz5s1x5MgRhIeHw8LCAuPHjwcAREREYM+ePVizZg1cXV2xc+dOjB49Gg4ODnjzzTc1+ouPj4dCoSi3HpVKhZ49e8LOzg4HDhzA9evXERoaCn19fXzyySdSuwEDBiA3NxfLly+Hm5sbrl+/DrVa/SK77aVTiL9Gtyrw8/ND27ZtsXDhQgCAWq2Gk5MTxo0bh8jIyGcur1KpUKdOHSxcuBChoaFVWmdBQQEsLCyQn58PpVKpTbmvlGvkNrlLeKZMo8Fyl1A1M/LlruC1UBuOSaCWHJc8Jl8LL/IZ5urqiokTJ2LixIka03v16gVbW1ssX75cmvb222/D2NgYa9asAQC0aNECISEh+Pjjj6U23t7e6NGjB+bMmSNNS0tLQ69evXDkyBHY29tj06ZN6Nu3LwBgx44d6NWrF7Kzs2FrawsASEhIwJQpU5CXlwcDAwMkJSVh4MCBuHjxIurWrfvc+6m6VPXzW6vTNKWlpUhNTUVgYOCfHejoIDAwEAcPHqxSH8XFxXj48OFTd1JJSQkKCgo0XkRERC+iOj7DKhIQEIDk5GScP38eAHD8+HHs27cPPXr00GizdetWXLt2DUII/Pzzzzh//jy6d+8utSkuLsbgwYOxaNEi2NnZlVvPwYMH0bJlSymIAEBQUBAKCgpw+vRpAMDWrVvh4+ODTz/9FI6OjmjSpAkmTZqE+/fvP/f2vQpanaa5efMmVCqVxo4AAFtbW5w7d65KfUyZMgUODg4aB8OTYmJiMHPmTG1KIyIieqrq+AyrSGRkJAoKCtCsWTPo6upCpVJh7ty5GDJkiNRmwYIF+Ne//oX69etDT08POjo6WLp0KTp27Ci1+fDDDxEQEIA+ffpUuJ6cnJwKay+bBwAXL17Evn37YGRkhE2bNuHmzZsYPXo0bt26hcTExOfexpdN62tGXsS8efOwbt06pKSkwMjIqNJ2U6dORUREhPS+oKAATk5Or6JEIiIirWzYsAHffvst1q5di+bNmyMtLQ0TJ06Eg4MDwsLCADwOI7/99hu2bt0KFxcX/PrrrxgzZoz0n/OtW7diz549OHbs2AvVolaroVAo8O2338LCwgIAEBcXh/79++Orr76CsbHxC2/vy6BVGLG2toauri5yc3M1pufm5lY4pPRXsbGxmDdvHnbv3o1WrVo9ta2hoSEMDQ21KY2IiOipXuQz7GkmT56MyMhIDBw4EADQsmVLXL58GTExMQgLC8P9+/cxbdo0bNq0CT179gQAtGrVCmlpaYiNjUVgYCD27NmDjIwMWFpaavT99ttvo0OHDkhJSYGdnV25O3/KtqWsfnt7ezg6OkpBBADc3d0hhMDVq1fRuHHj597Ol0mra0YMDAzg7e2N5ORkaZparUZycjL8/f0rXe7TTz/F7NmzkZSUBB8fn+evloiI6Dk972fYsxQXF0NHR/PjVFdXV7qD5eHDh3j48OFT20RGRuLEiRNIS0uTXgDwxRdfSKdX/P39cfLkSdy4cUPqY9euXVAqlfDw8AAAtGvXDtnZ2SgsLJTanD9/Hjo6Oqhfv/5zb+PLpvVpmoiICISFhcHHxwe+vr6Ij49HUVERwsPDAQChoaFwdHRETEwMgMe3PEVFRWHt2rVwdXWVzmuZmZnBzMysGjeFiIjo6bT9DCstLcWZM2ekn69du4a0tDSYmZnBzc0NANC7d2/MnTsXzs7OaN68OY4dO4a4uDgMHz4cAKBUKtGpUydMnjwZxsbGcHFxwS+//IJVq1YhLi4OwOORjYpGZ5ydndGgQQMAQPfu3eHh4YGhQ4fi008/RU5ODj766COMGTNGOpswePBgzJ49G+Hh4Zg5cyZu3ryJyZMnY/jw4TX2FA3wHGEkJCQEeXl5iIqKQk5ODry8vJCUlCRdRJOVlaWR/hYvXozS0lL0799fo5/o6GjMmDHjxaonIiLSgrafYdnZ2WjdurX0PjY2FrGxsejUqRNSUlIAPL4e5OOPP8bo0aNx48YNODg44P3330dUVJS03Lp16zB16lQMGTIEt2/fhouLC+bOnYtRo0ZVuXZdXV389NNP+OCDD+Dv7w9TU1OEhYVh1qxZUhszMzPs2rUL48aNg4+PD6ysrDBgwACN24drIq2fMyIHPmek+tSK5zkAfKZDNakNxyRQS45LHpNEWnspzxkhIiIiqm4MI0RERCSrV/qcESIiolphhsWz28jtNTp1yJERIiIikhXDCBEREcmKYYSIiIhkxTBCREREsmIYISIiIlkxjBAREZGsGEaIiIhIVgwjREREJCuGESIiIpIVwwgRERHJimGEiIiIZMUwQkRERLJiGCEiIiJZMYwQEVG1WLRoEVxdXWFkZAQ/Pz8cOnSo0ranT5/G22+/DVdXVygUCsTHx5drM2PGDCgUCo1Xs2bNNNp8/fXX6Ny5M5RKJRQKBe7evasxPzMzEyNGjECDBg1gbGyMRo0aITo6GqWlpRrthBCIjY1FkyZNYGhoCMe4e5j7a4k0//o9NQb/UIwmCwqhM7MAE5MeaL+DqFIMI0RE9MLWr1+PiIgIREdH4+jRo/D09ERQUBBu3LhRYfvi4mI0bNgQ8+bNg52dXaX9Nm/eHNevX5de+/btK9dPcHAwpk2bVuHy586dg1qtxpIlS3D69Gl88cUXSEhIKNd+woQJWLZsGWJjY3Hu3DlsHWgCX0ddaX6JCqhnooOPOhrA044fndVNT+4CiIio9ouLi8PIkSMRHh4OAEhISMC2bduwYsUKREZGlmvftm1btG3bFgAqnF9GT0/vqWFl4sSJAICUlJQK5wcHByM4OFh637BhQ6Snp2Px4sWIjY0FAJw9exaLFy/GqVOn0LRpUwBAAwddjX5cLXUwv4cRAGDFsYeV1kPPh/GOiIheSGlpKVJTUxEYGChN09HRQWBgIA4ePPhCff/xxx9wcHBAw4YNMWTIEGRlZb1oucjPz0fdunWl9z/++CMaNmyIn376CQ0aNICrqyve23oft++LF14XVQ3DCBERvZCbN29CpVLB1tZWY7qtrS1ycnKeu18/Pz+sXLkSSUlJWLx4MS5duoQOHTrg3r17z93nhQsXsGDBArz//vvStIsXL+Ly5cv47rvvsGrVKqxcuRKp11Xov6H4uddD2uFpGiIiqpF69Ogh/dyqVSv4+fnBxcUFGzZswIgRI7Tu79q1awgODsY777yDkSNHStPVajVKSkqwatUqNGnSBACw/E1jeH9dhPSbKjS11q2sS6omHBkhIqIXYm1tDV1dXeTm5mpMz83Nfer1HtqytLREkyZNcOHCBa2Xzc7ORpcuXRAQEICvv/5aY569vT309PSkIAIA7taPPx6z8nmq5lVgGCEiohdiYGAAb29vJCcnS9PUajWSk5Ph7+9fbespLCxERkYG7O3ttVru2rVr6Ny5M7y9vZGYmAgdHc2Pvnbt2uHRo0fIyMiQpp2/pQYAuFgqXrxweiaepiEiohcWERGBsLAw+Pj4wNfXF/Hx8SgqKpLurgkNDYWjoyNiYmIAPL7o9cyZM9LP165dQ1paGszMzODm5gYAmDRpEnr37g0XFxdkZ2cjOjoaurq6GDRokLTenJwc5OTkSKMlJ0+ehLm5OZydnVG3bl0piLi4uCA2NhZ5eXnSsmWjNoGBgWjTpg2GDx+O+Ph4qNVqjPnpAbo11EUTqz9P0aTlqAAAhaUCecVqpOWoYKALeNTjaZwXxTBCREQvLCQkBHl5eYiKikJOTg68vLyQlJQkXdSalZWlMSKRnZ2N1q1bS+9jY2MRGxuLTp06SbfpXr16FYMGDcKtW7dQr149tG/fHr/99hvq1asnLZeQkICZM2dK7zt27AgASExMxLBhw7Br1y5cuHABFy5cQP369TVqFuLxKRgdHR38+OOPGDduHDp27AhTU1P0sNfB592NNNq3XlIk/Zx6XY21Jx/BxUKBzInmL7LrCIBClP02arCCggJYWFggPz8fSqVS7nIq5Rq5Te4SninTaLDcJVTNjHy5K3gt1IZjEqglxyWPyb+XGRZyV/BsteCYrOrnN68ZISIiIlnxNA0REb1StWHELtPo2W2o+nBkhIiIiGTFMEJERESyYhghIiIiWTGMEBERkawYRoiIiEhWDCNEREQkK4YRIiIikhXDCBEREcmKYYSIiIhkxTBCREREsmIYISIiIlkxjBAREZGsGEaIiIhIVgwjREREJCuGESIiIpIVwwgRERHJimGEiIiIZMUwQkRERLJiGCEiIiJZMYwQERGRrBhGiIiISFYMI0RERCQrhhEiIiKSFcMIERERyYphhIiIiGTFMEJERESyYhghIiIiWTGMEBERkawYRoiIiEhWDCNEREQkK4YRIiIikhXDCBEREcnqucLIokWL4OrqCiMjI/j5+eHQoUOVtj19+jTefvttuLq6QqFQID4+/nlrJSIioteQ1mFk/fr1iIiIQHR0NI4ePQpPT08EBQXhxo0bFbYvLi5Gw4YNMW/ePNjZ2b1wwURERPR60TqMxMXFYeTIkQgPD4eHhwcSEhJgYmKCFStWVNi+bdu2+OyzzzBw4EAYGhq+cMFERET0etEqjJSWliI1NRWBgYF/dqCjg8DAQBw8eLDaiiopKUFBQYHGi4iIiF5PWoWRmzdvQqVSwdbWVmO6ra0tcnJyqq2omJgYWFhYSC8nJ6dq65uIiIhqlhp5N83UqVORn58vva5cuSJ3SURERPSS6GnT2NraGrq6usjNzdWYnpubW60XpxoaGvL6EiIior8JrUZGDAwM4O3tjeTkZGmaWq1GcnIy/P39q704IiIiev1pNTICABEREQgLC4OPjw98fX0RHx+PoqIihIeHAwBCQ0Ph6OiImJgYAI8vej1z5oz087Vr15CWlgYzMzO4ublV46YQERFRbaR1GAkJCUFeXh6ioqKQk5MDLy8vJCUlSRe1ZmVlQUfnzwGX7OxstG7dWnofGxuL2NhYdOrUCSkpKS++BURERFSraR1GAGDs2LEYO3ZshfOeDBiurq4QQjzPaoiIiOhvoEbeTUNERER/HwwjREREJCuGESIiIpIVwwgRERHJimGEiIiIZMUwQkRERLJiGCEiIiJZMYwQERGRrBhGiIiISFYMI0RERCQrhhEiIiKSFcMIERERyYphhIiIiGTFMEJERESyYhghIiIiWTGMEBERkawYRoiIiEhWDCNEREQkK4YRIiIikhXDCBEREcmKYYSIiIhkxTBCREREsmIYISIiIlkxjBAREZGsGEaIiIhIVgwjREREJCuGESIiIpIVwwgRERHJimGEiIiIZMUwQkRERLJiGCEiIiJZMYwQERGRrBhGiIiISFYMI0RERCQrhhEiIiKSFcMIERERyYphhIiIiGTFMEJERESyYhghIiIiWTGMEBERkawYRoiIiEhWDCNEREQkK4YRIiIikhXDCBEREcmKYYSIiIhkxTBCREREsmIYISIiIlkxjBAREZGsGEaIiIhIVgwjREREJCuGESIiIpIVwwgRERHJimGEiIiIZMUwQkRERLJiGCEiIiJZMYwQERGRrBhGiIiISFYMI0RERCQrhhEiIiKSFcMIERERyeq5wsiiRYvg6uoKIyMj+Pn54dChQ09t/91336FZs2YwMjJCy5YtsX379ucqloiIiF4/WoeR9evXIyIiAtHR0Th69Cg8PT0RFBSEGzduVNj+wIEDGDRoEEaMGIFjx46hb9++6Nu3L06dOvXCxRMREVHtp3UYiYuLw8iRIxEeHg4PDw8kJCTAxMQEK1asqLD9/PnzERwcjMmTJ8Pd3R2zZ89GmzZtsHDhwhcunoiIiGo/PW0al5aWIjU1FVOnTpWm6ejoIDAwEAcPHqxwmYMHDyIiIkJjWlBQEDZv3lzpekpKSlBSUiK9z8/PBwAUFBRoU+4rpy4plruEZypQCLlLqJoa/ruuLWrDMQnUkuOSx2S1qQ3HJY/J6lH2uS3E0/enVmHk5s2bUKlUsLW11Zhua2uLc+fOVbhMTk5Ohe1zcnIqXU9MTAxmzpxZbrqTk5M25VIFLOQuoKrm1ZpKqRrUit82j8m/lVrx265Fx+S9e/dgYVF5vVqFkVdl6tSpGqMparUat2/fhpWVFRQKhYyV1W4FBQVwcnLClStXoFQq5S6HCACPS6p5eExWHyEE7t27BwcHh6e20yqMWFtbQ1dXF7m5uRrTc3NzYWdnV+EydnZ2WrUHAENDQxgaGmpMs7S01KZUegqlUsl/YFTj8LikmobHZPV42ohIGa0uYDUwMIC3tzeSk5OlaWq1GsnJyfD3969wGX9/f432ALBr165K2xMREdHfi9anaSIiIhAWFgYfHx/4+voiPj4eRUVFCA8PBwCEhobC0dERMTExAIAJEyagU6dO+Pzzz9GzZ0+sW7cOR44cwddff129W0JERES1ktZhJCQkBHl5eYiKikJOTg68vLyQlJQkXaSalZUFHZ0/B1wCAgKwdu1afPTRR5g2bRoaN26MzZs3o0WLFtW3FVQlhoaGiI6OLncKjEhOPC6ppuEx+eopxLPutyEiIiJ6ifjdNERERCQrhhEiIiKSFcMIERERyYphpBbLzMyEQqFAWlqa3KUQERE9N4aRGmzYsGFQKBTSy8rKCsHBwThx4gSAx4/Hv379Ou9Molfmr8ekvr4+bG1t0a1bN6xYsQJqtVru8ug18Ne/eRW9ZsyYIXeJ9BIwjNRwwcHBuH79Oq5fv47k5GTo6emhV69eAABdXV3Y2dlBT69GPtWfXlNlx2RmZiZ27NiBLl26YMKECejVqxcePXpU4TIPHz58xVVSbVX29+769euIj4+HUqnUmDZp0iSt+ntZx54QotLjnbTHMFLDGRoaws7ODnZ2dvDy8kJkZCSuXLmCvLy8cqdp7ty5gyFDhqBevXowNjZG48aNkZiYKPV19epVDBo0CHXr1oWpqSl8fHzw+++/S/MXL16MRo0awcDAAE2bNsXq1as1arl79y7ee+891KtXD0qlEv/85z9x/PjxV7IfqOYoOyYdHR3Rpk0bTJs2DVu2bMGOHTuwcuVKAI//d7t48WK8+eabMDU1xdy5cwEAW7ZsQZs2bWBkZISGDRti5syZGn/Q4+Li0LJlS5iamsLJyQmjR49GYWGhNH/lypWwtLTETz/9hKZNm8LExAT9+/dHcXExvvnmG7i6uqJOnToYP348VCrVK90vVD3K/t7Z2dnBwsICCoVCem9jY4O4uDjUr18fhoaG0nOuypT9TVy/fj06deoEIyMjfPvtt3j06BHGjx8PS0tLWFlZYcqUKQgLC0Pfvn2lZdVqNWJiYtCgQQMYGxvD09MT33//vTQ/JSUFCoUCO3bsgLe3NwwNDbFv3z4cP34cXbp0gbm5OZRKJby9vXHkyBFpuX379qFDhw4wNjaGk5MTxo8fj6KiImm+q6srPvnkEwwfPhzm5uZwdnb+ez4UVFCNFRYWJvr06SO9v3fvnnj//feFm5ubUKlU4tKlSwKAOHbsmBBCiDFjxggvLy9x+PBhcenSJbFr1y6xdetWadmGDRuKDh06iL1794o//vhDrF+/Xhw4cEAIIcTGjRuFvr6+WLRokUhPTxeff/650NXVFXv27JHWHxgYKHr37i0OHz4szp8/L/7v//5PWFlZiVu3br2yfULyevKY/CtPT0/Ro0cPIYQQAISNjY1YsWKFyMjIEJcvXxa//vqrUCqVYuXKlSIjI0Ps3LlTuLq6ihkzZkh9fPHFF2LPnj3i0qVLIjk5WTRt2lR88MEH0vzExEShr68vunXrJo4ePSp++eUXYWVlJbp37y4GDBggTp8+LX788UdhYGAg1q1b91L3Bb18iYmJwsLCQnofFxcnlEql+O9//yvOnTsn/v3vfwt9fX1x/vx5IYSQ/ia6urqKH374QVy8eFFkZ2eLOXPmiLp164qNGzeKs2fPilGjRgmlUqlxLM+ZM0c0a9ZMJCUliYyMDJGYmCgMDQ1FSkqKEEKIn3/+WQAQrVq1Ejt37hQXLlwQt27dEs2bNxfvvvuuOHv2rDh//rzYsGGDSEtLE0IIceHCBWFqaiq++OILcf78ebF//37RunVrMWzYMGm9Li4uom7dumLRokXijz/+EDExMUJHR0ecO3fu5e/gGoRhpAYLCwsTurq6wtTUVJiamgoAwt7eXqSmpgohRLkw0rt3bxEeHl5hX0uWLBHm5uaVBoeAgAAxcuRIjWnvvPOOeOONN4QQQuzdu1colUrx4MEDjTaNGjUSS5YseZHNpFrkaWEkJCREuLu7CyEeh5GJEydqzO/atav45JNPNKatXr1a2NvbV7q+7777TlhZWUnvExMTBQBx4cIFadr7778vTExMxL1796RpQUFB4v3336/ydlHN9GQYcXBwEHPnztVo07ZtWzF69GghxJ9/E+Pj4zXa2Nrais8++0x6/+jRI+Hs7Cwdyw8ePBAmJibSf87KjBgxQgwaNEgI8WcY2bx5s0Ybc3NzsXLlygrrHzFihPjXv/6lMW3v3r1CR0dH3L9/XwjxOIy8++670ny1Wi1sbGzE4sWLK+zzdcWLDWq4Ll26YPHixQAen4b56quv0KNHDxw6dKhc2w8++ABvv/02jh49iu7du6Nv374ICAgAAKSlpaF169aoW7duhes5e/Ys/vWvf2lMa9euHebPnw8AOH78OAoLC2FlZaXR5v79+8jIyHjh7aTaTwgBhUIhvffx8dGYf/z4cezfv186ZQMAKpUKDx48QHFxMUxMTLB7927ExMTg3LlzKCgowKNHjzTmA4CJiQkaNWok9WFrawtXV1eYmZlpTLtx48bL2lSSQUFBAbKzs9GuXTuN6e3atSt3uvivx15+fj5yc3Ph6+srTdPV1YW3t7d00fWFCxdQXFyMbt26afRTWlqK1q1bV9o38Pj72t577z2sXr0agYGBeOedd6Tj8/jx4zhx4gS+/fZbqb0QAmq1GpcuXYK7uzsAoFWrVtL8stNSf7fjl2GkhjM1NYWbm5v0ftmyZbCwsMDSpUvx3nvvabTt0aMHLl++jO3bt2PXrl3o2rUrxowZg9jYWBgbG79QHYWFhbC3t0dKSkq5eZaWli/UN70ezp49iwYNGkjvTU1NNeYXFhZi5syZeOutt8ota2RkhMzMTPTq1QsffPAB5s6di7p162Lfvn0YMWIESktLpTCir6+vsWzZnT1PTuPdPX9fTx57z1J2XdK2bdvg6OioMe/J76d5su8ZM2Zg8ODB2LZtG3bs2IHo6GisW7cO/fr1Q2FhId5//32MHz++3DqdnZ2ln3n8MozUOgqFAjo6Orh//36F8+vVq4ewsDCEhYWhQ4cOmDx5MmJjY9GqVSssW7YMt2/frnB0xN3dHfv370dYWJg0bf/+/fDw8AAAtGnTBjk5OdDT04Orq+tL2Taqvfbs2YOTJ0/iww8/rLRNmzZtkJ6erhGu/yo1NRVqtRqff/659GWbGzZseCn1Uu2jVCrh4OCA/fv3o1OnTtL0/fv3a4x6PMnCwgK2trY4fPgwOnbsCODxiNzRo0fh5eUFAPDw8IChoSGysrI0+q6qJk2aoEmTJvjwww8xaNAgJCYmol+/fmjTpg3OnDlT6TFPf2IYqeFKSkqQk5MD4PFpmoULF6KwsBC9e/cu1zYqKgre3t5o3rw5SkpK8NNPP0nDgIMGDcInn3yCvn37IiYmBvb29jh27BgcHBzg7++PyZMnY8CAAWjdujUCAwPx448/YuPGjdi9ezcAIDAwEP7+/ujbty8+/fRTNGnSBNnZ2di2bRv69etXbuiSXl9lx6RKpUJubi6SkpIQExODXr16ITQ0tNLloqKi0KtXLzg7O6N///7Q0dHB8ePHcerUKcyZMwdubm54+PAhFixYgN69e2P//v1ISEh4hVtGNd3kyZMRHR2NRo0awcvLC4mJiUhLS9M4DVKRcePGISYmBm5ubmjWrBkWLFiAO3fuSKcVzc3NMWnSJHz44YdQq9Vo37498vPzsX//fiiVSo3/pP3V/fv3MXnyZPTv3x8NGjTA1atXcfjwYbz99tsAgClTpuAf//gHxo4di/feew+mpqY4c+YMdu3ahYULF1bvzqnt5L5ohSoXFhYmAEgvc3Nz0bZtW/H9998LIcpfwDp79mzh7u4ujI2NRd26dUWfPn3ExYsXpf4yMzPF22+/LZRKpTAxMRE+Pj7i999/l+Z/9dVXomHDhkJfX180adJErFq1SqOegoICMW7cOOHg4CD09fWFk5OTGDJkiMjKynr5O4NqhL8ek3p6eqJevXoiMDBQrFixQqhUKqkdALFp06ZyyyclJYmAgABhbGwslEql8PX1FV9//bU0Py4uTtjb2wtjY2MRFBQkVq1aJQCIO3fuCCHKX9AohBDR0dHC09OzXJ2VXWhLtceTv2+VSiVmzJghHB0dhb6+vvD09BQ7duyQ5j/5N7HMw4cPxdixY4VSqRR16tQRU6ZMEe+8844YOHCg1EatVov4+HjRtGlToa+vL+rVqyeCgoLEL7/8IoT48wLWsmNRCCFKSkrEwIEDhZOTkzAwMBAODg5i7Nix0sWpQghx6NAh0a1bN2FmZiZMTU1Fq1atNC7CdXFxEV988YVGvZ6eniI6Ovr5d1wtpBBCCLmCEBER0aumVqvh7u6OAQMGYPbs2XKXQ+BpGiIies1dvnwZO3fuRKdOnVBSUoKFCxfi0qVLGDx4sNyl0f/HJ7ASEdFrTUdHBytXrkTbtm3Rrl07nDx5Ert375auqSP58TQNERERyYojI0RERCQrhhEiIiKSFcMIERERyYphhIiIiGTFMEJERESyYhghIiIiWTGMEBERkawYRoiIiEhWDCNEREQkq/8H+Q/2CXwt0pUAAAAASUVORK5CYII=",
            "text/plain": [
              "\u003cFigure size 640x480 with 1 Axes\u003e"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "ax = stats[['train_pct', 'serve_pct']].plot.bar(rot=0)\n",
        "for container in ax.containers:\n",
        "    ax.bar_label(container)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8VzVRNGxxLVM"
      },
      "source": [
        "### L-Infinity Distance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "metadata": {
        "tags": [],
        "id": "dOm0RevHxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003etrain_n\u003c/th\u003e\n",
              "      \u003cth\u003eserve_n\u003c/th\u003e\n",
              "      \u003cth\u003etrain_pct\u003c/th\u003e\n",
              "      \u003cth\u003eserve_pct\u003c/th\u003e\n",
              "      \u003cth\u003eabs_change_pct\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003eBiscoe\u003c/th\u003e\n",
              "      \u003ctd\u003e116\u003c/td\u003e\n",
              "      \u003ctd\u003e1054\u003c/td\u003e\n",
              "      \u003ctd\u003e0.487395\u003c/td\u003e\n",
              "      \u003ctd\u003e0.492754\u003c/td\u003e\n",
              "      \u003ctd\u003e0.005359\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003eDream\u003c/th\u003e\n",
              "      \u003ctd\u003e86\u003c/td\u003e\n",
              "      \u003ctd\u003e682\u003c/td\u003e\n",
              "      \u003ctd\u003e0.361345\u003c/td\u003e\n",
              "      \u003ctd\u003e0.318841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.042504\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003eTorgersen\u003c/th\u003e\n",
              "      \u003ctd\u003e36\u003c/td\u003e\n",
              "      \u003ctd\u003e403\u003c/td\u003e\n",
              "      \u003ctd\u003e0.151261\u003c/td\u003e\n",
              "      \u003ctd\u003e0.188406\u003c/td\u003e\n",
              "      \u003ctd\u003e0.037145\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "           train_n  serve_n  train_pct  serve_pct  abs_change_pct\n",
              "Biscoe         116     1054   0.487395   0.492754        0.005359\n",
              "Dream           86      682   0.361345   0.318841        0.042504\n",
              "Torgersen       36      403   0.151261   0.188406        0.037145"
            ]
          },
          "execution_count": 60,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "stats['abs_change_pct'] = abs(stats['train_pct'] - stats['serve_pct'])\n",
        "stats"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "metadata": {
        "tags": [],
        "id": "C5lrv4ncxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[Text(0, 0, '0.00535867'), Text(0, 0, '0.042504'), Text(0, 0, '0.0371453')]"
            ]
          },
          "execution_count": 61,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjUAAAHSCAYAAAAUvDo/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABIsklEQVR4nO3de1yUdf7//ycDwpgJKiaIkXig8IBgqAjrZn6cFZPbGrsd0Cxdv6T2+SQeaE30p2K1hdVa1Oon0jWrLb+a5ZJLfnEJ29IgDyCppZXlWQclEpQSlJnfH96cdhYwxsjBax73221u5ft6Xe/rNey18fQ6etntdrsAAACucSZ3NwAAANAcCDUAAMAQCDUAAMAQCDUAAMAQCDUAAMAQCDUAAMAQCDUAAMAQfNzdwNVis9l0/PhxtW3bVl5eXu5uBwAANIHdbteZM2cUEhIik+nyx2I8JtQcP35coaGh7m4DAABcgSNHjujGG2+8bI3HhJq2bdtKuvhD8ff3d3M3AACgKaqqqhQaGur4PX45HhNqLp1y8vf3J9QAAHCNacqlI1woDAAADIFQAwAADIFQAwAADIFQA6BZLV26VGFhYTKbzYqNjdW2bdsuW7927VpFRETIbDYrMjJSGzZsaLT2oYcekpeXl7KyshxjBw8eVEpKirp166bWrVurR48eysjIUG1trVONl5dXvc8nn3zSrL0AcC9CDYBms2bNGqWlpSkjI0MlJSWKiopSQkKCTp482WB9YWGhxo4dq5SUFO3cuVNJSUlKSkrSnj176tX+/e9/1yeffKKQkBCn8X379slms+nll1/WZ599pueff17Z2dmaO3duvTnef/99nThxwvGJiYlp1l4AuJndQ1RWVtol2SsrK93dCmBYgwYNsj/88MOOP9fV1dlDQkLsmZmZDdbfe++99sTERKex2NhY+5QpU5zGjh49au/SpYt9z5499q5du9qff/75y/bxzDPP2Lt16+b484EDB+yS7Dt37mx0nV+qFwA/jyu/vzlSA6BZ1NbWqri4WBaLxTFmMplksVhUVFTU4DpFRUVO9ZKUkJDgVG+z2fTAAw9o1qxZ6tOnT5N6qaysVIcOHeqNjx49Wp06ddKQIUO0fv36q9ILgKuHUAOgWZSXl6uurk5BQUFO40FBQbJarQ2uY7Vaf7L+6aeflo+Pj6ZNm9akPvbv36+//OUvmjJlimPs+uuv1+LFi7V27Vq99957GjJkiJKSkpyCzS/RC4Cry2Mevgfg2lNcXKwXXnhBJSUlTXrw1rFjxzRy5Ejdc889mjRpkmO8Y8eOSktLc/x54MCBOn78uJ599lmNHj36F+kFwNXHkRoAzaJjx47y9vZWWVmZ03hZWZmCg4MbXCc4OPiy9Zs3b9bJkyd10003ycfHRz4+Pjp06JAeeeQRhYWFOa13/PhxDRs2TPHx8Vq2bNlP9hsbG6v9+/f/Ir0AcA9CDYBm4evrq5iYGBUUFDjGbDabCgoKFBcX1+A6cXFxTvWSlJ+f76h/4IEHtGvXLpWWljo+ISEhmjVrljZu3OhY59ixY7r99tsVExOjlStX/uSbfCWptLRUnTt3bvZeALgPp58ANJu0tDRNmDBBAwYM0KBBg5SVlaXq6mpNnDhRkjR+/Hh16dJFmZmZkqTp06dr6NChWrx4sRITE7V69Wrt2LHDcaQlMDBQgYGBTtto1aqVgoODdcstt0j6MdB07dpVf/7zn3Xq1ClH7aWjLK+99pp8fX3Vv39/SdK6dev0yiuv6K9//aujtjl6AeBehBoAzSY5OVmnTp3SggULZLVaFR0drby8PMcFuIcPH3Y6ihIfH69Vq1Zp3rx5mjt3rsLDw5WTk6O+ffs2eZv5+fnav3+/9u/frxtvvNFpmd1ud/z7E088oUOHDsnHx0cRERFas2aN7r777mbtBYB7edn//f/1BlZVVaWAgABVVlbylm4AAK4Rrvz+5poaAABgCJx+AvCTwtLfc3cLhnFwUaK7WwAMiyM1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEK4o1CxdulRhYWEym82KjY3Vtm3bLlu/du1aRUREyGw2KzIyUhs2bGi09qGHHpKXl5eysrKcxisqKjRu3Dj5+/urXbt2SklJ0dmzZ6+kfQAAYEAuh5o1a9YoLS1NGRkZKikpUVRUlBISEnTy5MkG6wsLCzV27FilpKRo586dSkpKUlJSkvbs2VOv9u9//7s++eQThYSE1Fs2btw4ffbZZ8rPz1dubq4++ugjTZ482dX2AQCAQXnZ7Xa7KyvExsZq4MCBWrJkiSTJZrMpNDRUqampSk9Pr1efnJys6upq5ebmOsYGDx6s6OhoZWdnO8aOHTum2NhYbdy4UYmJiZoxY4ZmzJghSdq7d6969+6t7du3a8CAAZKkvLw8jRo1SkePHm0wBNXU1Kimpsbx56qqKoWGhqqyslL+/v6ufGXA44Wlv+fuFgzj4KJEd7cAXFOqqqoUEBDQpN/fLh2pqa2tVXFxsSwWy48TmEyyWCwqKipqcJ2ioiKneklKSEhwqrfZbHrggQc0a9Ys9enTp8E52rVr5wg0kmSxWGQymbR169YGt5uZmamAgADHJzQ01JWvCgAArjEuhZry8nLV1dUpKCjIaTwoKEhWq7XBdaxW60/WP/300/Lx8dG0adManaNTp05OYz4+PurQoUOj250zZ44qKysdnyNHjvzk9wMAANcuH3c3UFxcrBdeeEElJSXy8vJqtnn9/Pzk5+fXbPMBAICWzaUjNR07dpS3t7fKysqcxsvKyhQcHNzgOsHBwZet37x5s06ePKmbbrpJPj4+8vHx0aFDh/TII48oLCzMMcd/Xoh84cIFVVRUNLpdAADgWVwKNb6+voqJiVFBQYFjzGazqaCgQHFxcQ2uExcX51QvSfn5+Y76Bx54QLt27VJpaanjExISolmzZmnjxo2OOU6fPq3i4mLHHJs2bZLNZlNsbKwrXwEAABiUy6ef0tLSNGHCBA0YMECDBg1SVlaWqqurNXHiREnS+PHj1aVLF2VmZkqSpk+frqFDh2rx4sVKTEzU6tWrtWPHDi1btkySFBgYqMDAQKdttGrVSsHBwbrlllskSb169dLIkSM1adIkZWdn6/z585o6darGjBnT4J1PAADA87gcapKTk3Xq1CktWLBAVqtV0dHRysvLc1wMfPjwYZlMPx4Aio+P16pVqzRv3jzNnTtX4eHhysnJUd++fV3a7ptvvqmpU6dq+PDhMplMuuuuu/Tiiy+62j4AADAol59Tc61y5T53AM54Tk3z4Tk1gGt+sefUAAAAtFSEGgAAYAiEGgAAYAiEGgAAYAiEGgCAoSxdulRhYWEym82KjY3Vtm3bLlu/du1aRUREyGw2KzIyUhs2bHBavnDhQkVERKhNmzZq3769LBaL03sH//Wvf8nLy6vBz/bt2yVJ586d0x/+8AdFRkbKx8dHSUlJl+3p448/lo+Pj6Kjo+v18p/biIiIcKqZMmWKevToodatW+uGG27QnXfeqX379v3ET80YCDUAAMNYs2aN0tLSlJGRoZKSEkVFRSkhIaHeU+kvKSws1NixY5WSkqKdO3cqKSlJSUlJ2rNnj6Pm5ptv1pIlS7R7925t2bJFYWFhGjFihE6dOiXp4qNLTpw44fR58MEH1a1bN8eLmOvq6tS6dWtNmzat3kue/9Pp06c1fvx4DR8+vMHlffr0cdrWli1bnJbHxMRo5cqV2rt3rzZu3Ci73a4RI0aorq6uyT/HaxW3dAP4SdzS3Xy4pfuXFRsbq4EDB2rJkiWSLj71PjQ0VKmpqUpPT69Xn5ycrOrqauXm5jrGBg8erOjoaGVnZze4jUu/T95///0Gg8f58+fVpUsXpaamav78+fWW/+EPf9Dp06eVk5PT4PxjxoxReHi4vL29lZOTo9LSUseyhQsX1hv7Kbt27VJUVJT279+vHj16NHm9loJbugEAHqe2tlbFxcVOR0JMJpMsFouKiooaXKeoqKjekZOEhIRG62tra7Vs2TIFBAQoKiqqwZr169fr22+/dTxp3xUrV67UN998o4yMjEZrvvrqK4WEhKh79+4aN26cDh8+3GhtdXW1Vq5cqW7duik0NNTlfq41hBoAgCGUl5errq7O8YT7S4KCgmS1Whtcx2q1Nqk+NzdX119/vcxms55//nnl5+erY8eODc65YsUKJSQk6MYbb3Sp/6+++krp6el644035OPT8AP/Y2Nj9eqrryovL08vvfSSDhw4oF//+tc6c+aMU93//u//6vrrr9f111+v//f//p/y8/Pl6+vrUj/XIkINAAA/YdiwYSotLVVhYaFGjhype++9t8HrdI4ePaqNGzcqJSXFpfnr6up033336bHHHtPNN9/caN0dd9yhe+65R/369VNCQoI2bNig06dP66233nKqGzdunHbu3KkPP/xQN998s+69916dO3fOpZ6uRYQaAIAhdOzYUd7e3iorK3MaLysrU3BwcIPrBAcHN6m+TZs26tmzpwYPHqwVK1bIx8dHK1asqDffypUrFRgYqNGjR7vU+5kzZ7Rjxw5NnTpVPj4+8vHx0eOPP65PP/1UPj4+2rRpU4PrtWvXTjfffLP279/vNB4QEKDw8HDddtttevvtt7Vv3z79/e9/d6mnaxGhBgBgCL6+voqJiVFBQYFjzGazqaCgQHFxcQ2uExcX51QvSfn5+Y3W//u8NTU1TmN2u10rV67U+PHj1apVK5d69/f31+7du1VaWur4PPTQQ7rllltUWlqq2NjYBtc7e/asvv76a3Xu3LnRue12u+x2e71+jcjlt3QDANBSpaWlacKECRowYIAGDRqkrKwsVVdXOy7aHT9+vLp06aLMzExJ0vTp0zV06FAtXrxYiYmJWr16tXbs2KFly5ZJunih7ZNPPqnRo0erc+fOKi8v19KlS3Xs2DHdc889TtvetGmTDhw4oAcffLDB3j7//HPV1taqoqJCZ86ccdzBFB0dLZPJpL59+zrVd+rUSWaz2Wn8j3/8o37729+qa9euOn78uDIyMuTt7a2xY8dKkr755hutWbNGI0aM0A033KCjR49q0aJFat26tUaNGvXzf8AtHKEGAGAYycnJOnXqlBYsWCCr1aro6Gjl5eU5LgY+fPiwTKYfT1LEx8dr1apVmjdvnubOnavw8HDl5OQ4goS3t7f27dun1157TeXl5QoMDNTAgQO1efNm9enTx2nbK1asUHx8fL2H4V0yatQoHTp0yPHn/v37S7p4JKWpjh49qrFjx+rbb7/VDTfcoCFDhuiTTz7RDTfcIEkym83avHmzsrKy9N133ykoKEi33XabCgsL1alTpyZv51rFc2oA/CSeU9N8eE4N4BqeUwMAADwOp58AANccjh42HyMdPeRIDQAAMARCDQAAMARCDQAAMARCDQAAMARCDQAAMARCDQAAMARCDQAAMARCDQAAMARCDQAAMARCDQAAMARCDQAAMARCDQAAMARCDQAAMIQrCjVLly5VWFiYzGazYmNjtW3btsvWr127VhERETKbzYqMjNSGDRucli9cuFARERFq06aN2rdvL4vFoq1btzrVhIWFycvLy+mzaNGiK2kfAAAYkMuhZs2aNUpLS1NGRoZKSkoUFRWlhIQEnTx5ssH6wsJCjR07VikpKdq5c6eSkpKUlJSkPXv2OGpuvvlmLVmyRLt379aWLVsUFhamESNG6NSpU05zPf744zpx4oTjk5qa6mr7AADAoFwONc8995wmTZqkiRMnqnfv3srOztZ1112nV155pcH6F154QSNHjtSsWbPUq1cvPfHEE7r11lu1ZMkSR819990ni8Wi7t27q0+fPnruuedUVVWlXbt2Oc3Vtm1bBQcHOz5t2rRxtX0AAGBQLoWa2tpaFRcXy2Kx/DiBySSLxaKioqIG1ykqKnKql6SEhIRG62tra7Vs2TIFBAQoKirKadmiRYsUGBio/v3769lnn9WFCxca7bWmpkZVVVVOHwAAYFw+rhSXl5errq5OQUFBTuNBQUHat29fg+tYrdYG661Wq9NYbm6uxowZo++//16dO3dWfn6+Onbs6Fg+bdo03XrrrerQoYMKCws1Z84cnThxQs8991yD283MzNRjjz3mytcDAADXMJdCzS9p2LBhKi0tVXl5uZYvX657771XW7duVadOnSRJaWlpjtp+/frJ19dXU6ZMUWZmpvz8/OrNN2fOHKd1qqqqFBoa+st/EQAA4BYunX7q2LGjvL29VVZW5jReVlam4ODgBtcJDg5uUn2bNm3Us2dPDR48WCtWrJCPj49WrFjRaC+xsbG6cOGCDh482OByPz8/+fv7O30AAIBxuRRqfH19FRMTo4KCAseYzWZTQUGB4uLiGlwnLi7OqV6S8vPzG63/93lramoaXV5aWiqTyeQ4kgMAADyby6ef0tLSNGHCBA0YMECDBg1SVlaWqqurNXHiREnS+PHj1aVLF2VmZkqSpk+frqFDh2rx4sVKTEzU6tWrtWPHDi1btkySVF1drSeffFKjR49W586dVV5erqVLl+rYsWO65557JF282Hjr1q0aNmyY2rZtq6KiIs2cOVP333+/2rdv31w/CwAAcA1zOdQkJyfr1KlTWrBggaxWq6Kjo5WXl+e4GPjw4cMymX48ABQfH69Vq1Zp3rx5mjt3rsLDw5WTk6O+fftKkry9vbVv3z699tprKi8vV2BgoAYOHKjNmzerT58+ki6eSlq9erUWLlyompoadevWTTNnznS6ZgYAAHg2L7vdbnd3E1dDVVWVAgICVFlZyfU1gIvC0t9zdwuGcXBRortbMAT2yebT0vdJV35/8+4nAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCFcUapYuXaqwsDCZzWbFxsZq27Ztl61fu3atIiIiZDabFRkZqQ0bNjgtX7hwoSIiItSmTRu1b99eFotFW7dudaqpqKjQuHHj5O/vr3bt2iklJUVnz569kvYBAIABuRxq1qxZo7S0NGVkZKikpERRUVFKSEjQyZMnG6wvLCzU2LFjlZKSop07dyopKUlJSUnas2ePo+bmm2/WkiVLtHv3bm3ZskVhYWEaMWKETp065agZN26cPvvsM+Xn5ys3N1cfffSRJk+efAVfGQAAGJGX3W63u7JCbGysBg4cqCVLlkiSbDabQkNDlZqaqvT09Hr1ycnJqq6uVm5urmNs8ODBio6OVnZ2doPbqKqqUkBAgN5//30NHz5ce/fuVe/evbV9+3YNGDBAkpSXl6dRo0bp6NGjCgkJqTdHTU2NampqnOYMDQ1VZWWl/P39XfnKgMcLS3/P3S0YxsFFie5uwRDYJ5tPS98nL2WCpvz+dulITW1trYqLi2WxWH6cwGSSxWJRUVFRg+sUFRU51UtSQkJCo/W1tbVatmyZAgICFBUV5ZijXbt2jkAjSRaLRSaTqd5pqksyMzMVEBDg+ISGhrryVQEAwDXGpVBTXl6uuro6BQUFOY0HBQXJarU2uI7Vam1SfW5urq6//nqZzWY9//zzys/PV8eOHR1zdOrUyanex8dHHTp0aHS7c+bMUWVlpeNz5MgRV74qAAC4xvi4u4FLhg0bptLSUpWXl2v58uW69957tXXr1nphpqn8/Pzk5+fXzF0CAICWyqUjNR07dpS3t7fKysqcxsvKyhQcHNzgOsHBwU2qb9OmjXr27KnBgwdrxYoV8vHx0YoVKxxz/OeFyBcuXFBFRUWj2wUAAJ7FpVDj6+urmJgYFRQUOMZsNpsKCgoUFxfX4DpxcXFO9ZKUn5/faP2/z3vpQt+4uDidPn1axcXFjuWbNm2SzWZTbGysK18BAAAYlMunn9LS0jRhwgQNGDBAgwYNUlZWlqqrqzVx4kRJ0vjx49WlSxdlZmZKkqZPn66hQ4dq8eLFSkxM1OrVq7Vjxw4tW7ZMklRdXa0nn3xSo0ePVufOnVVeXq6lS5fq2LFjuueeeyRJvXr10siRIzVp0iRlZ2fr/Pnzmjp1qsaMGdPgnU8AAMDzuBxqkpOTderUKS1YsEBWq1XR0dHKy8tzXAx8+PBhmUw/HgCKj4/XqlWrNG/ePM2dO1fh4eHKyclR3759JUne3t7at2+fXnvtNZWXlyswMFADBw7U5s2b1adPH8c8b775pqZOnarhw4fLZDLprrvu0osvvvhzvz8AADAIl59Tc61y5T53AM54JkjzaenPBLlWsE82n5a+T/5iz6kBAABoqQg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEK4o1CxdulRhYWEym82KjY3Vtm3bLlu/du1aRUREyGw2KzIyUhs2bHAsO3/+vGbPnq3IyEi1adNGISEhGj9+vI4fP+40R1hYmLy8vJw+ixYtupL2AQCAAbkcatasWaO0tDRlZGSopKREUVFRSkhI0MmTJxusLyws1NixY5WSkqKdO3cqKSlJSUlJ2rNnjyTp+++/V0lJiebPn6+SkhKtW7dOX3zxhUaPHl1vrscff1wnTpxwfFJTU11tHwAAGJSX3W63u7JCbGysBg4cqCVLlkiSbDabQkNDlZqaqvT09Hr1ycnJqq6uVm5urmNs8ODBio6OVnZ2doPb2L59uwYNGqRDhw7ppptuknTxSM2MGTM0Y8YMV9p1qKqqUkBAgCorK+Xv739FcwCeKiz9PXe3YBgHFyW6uwVDYJ9sPi19n3Tl97dLR2pqa2tVXFwsi8Xy4wQmkywWi4qKihpcp6ioyKlekhISEhqtl6TKykp5eXmpXbt2TuOLFi1SYGCg+vfvr2effVYXLlxodI6amhpVVVU5fQAAgHH5uFJcXl6uuro6BQUFOY0HBQVp3759Da5jtVobrLdarQ3Wnzt3TrNnz9bYsWOdEtm0adN06623qkOHDiosLNScOXN04sQJPffccw3Ok5mZqccee8yVrwcAAK5hLoWaX9r58+d17733ym6366WXXnJalpaW5vj3fv36ydfXV1OmTFFmZqb8/PzqzTVnzhyndaqqqhQaGvrLNQ8AANzKpVDTsWNHeXt7q6yszGm8rKxMwcHBDa4THBzcpPpLgebQoUPatGnTT543i42N1YULF3Tw4EHdcsst9Zb7+fk1GHYAAIAxuXRNja+vr2JiYlRQUOAYs9lsKigoUFxcXIPrxMXFOdVLUn5+vlP9pUDz1Vdf6f3331dgYOBP9lJaWiqTyaROnTq58hUAAIBBuXz6KS0tTRMmTNCAAQM0aNAgZWVlqbq6WhMnTpQkjR8/Xl26dFFmZqYkafr06Ro6dKgWL16sxMRErV69Wjt27NCyZcskXQw0d999t0pKSpSbm6u6ujrH9TYdOnSQr6+vioqKtHXrVg0bNkxt27ZVUVGRZs6cqfvvv1/t27dvrp8FAAC4hrkcapKTk3Xq1CktWLBAVqtV0dHRysvLc1wMfPjwYZlMPx4Aio+P16pVqzRv3jzNnTtX4eHhysnJUd++fSVJx44d0/r16yVJ0dHRTtv64IMPdPvtt8vPz0+rV6/WwoULVVNTo27dumnmzJlO18wAAADP5vJzaq5VPKcGuHI8E6T5tPRnglwr2CebT0vfJ3+x59QAAAC0VIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCIQaAABgCFcUapYuXaqwsDCZzWbFxsZq27Ztl61fu3atIiIiZDabFRkZqQ0bNjiWnT9/XrNnz1ZkZKTatGmjkJAQjR8/XsePH3eao6KiQuPGjZO/v7/atWunlJQUnT179kraBwAABuRyqFmzZo3S0tKUkZGhkpISRUVFKSEhQSdPnmywvrCwUGPHjlVKSop27typpKQkJSUlac+ePZKk77//XiUlJZo/f75KSkq0bt06ffHFFxo9erTTPOPGjdNnn32m/Px85ebm6qOPPtLkyZOv4CsDAAAj8rLb7XZXVoiNjdXAgQO1ZMkSSZLNZlNoaKhSU1OVnp5erz45OVnV1dXKzc11jA0ePFjR0dHKzs5ucBvbt2/XoEGDdOjQId10003au3evevfure3bt2vAgAGSpLy8PI0aNUpHjx5VSEjIT/ZdVVWlgIAAVVZWyt/f35WvDHi8sPT33N2CYRxclOjuFgyBfbL5tPR90pXf3y4dqamtrVVxcbEsFsuPE5hMslgsKioqanCdoqIip3pJSkhIaLRekiorK+Xl5aV27do55mjXrp0j0EiSxWKRyWTS1q1bG5yjpqZGVVVVTh8AAGBcLoWa8vJy1dXVKSgoyGk8KChIVqu1wXWsVqtL9efOndPs2bM1duxYRyKzWq3q1KmTU52Pj486dOjQ6DyZmZkKCAhwfEJDQ5v0HQEAwLWpRd39dP78ed17772y2+166aWXftZcc+bMUWVlpeNz5MiRZuoSAAC0RD6uFHfs2FHe3t4qKytzGi8rK1NwcHCD6wQHBzep/lKgOXTokDZt2uR03iw4OLjehcgXLlxQRUVFo9v18/OTn59fk78bAAC4trl0pMbX11cxMTEqKChwjNlsNhUUFCguLq7BdeLi4pzqJSk/P9+p/lKg+eqrr/T+++8rMDCw3hynT59WcXGxY2zTpk2y2WyKjY115SsAAACDculIjSSlpaVpwoQJGjBggAYNGqSsrCxVV1dr4sSJkqTx48erS5cuyszMlCRNnz5dQ4cO1eLFi5WYmKjVq1drx44dWrZsmaSLgebuu+9WSUmJcnNzVVdX57hOpkOHDvL19VWvXr00cuRITZo0SdnZ2Tp//rymTp2qMWPGNOnOJwAAYHwuh5rk5GSdOnVKCxYskNVqVXR0tPLy8hwXAx8+fFgm048HgOLj47Vq1SrNmzdPc+fOVXh4uHJyctS3b19J0rFjx7R+/XpJUnR0tNO2PvjgA91+++2SpDfffFNTp07V8OHDZTKZdNddd+nFF1+8ku8MAAAMyOXn1FyreE4NcOV4JkjzaenPBLlWsE82n5a+T/5iz6kBAABoqQg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEAg1AADAEK4o1CxdulRhYWEym82KjY3Vtm3bLlu/du1aRUREyGw2KzIyUhs2bHBavm7dOo0YMUKBgYHy8vJSaWlpvTluv/12eXl5OX0eeuihK2kfAAAYkMuhZs2aNUpLS1NGRoZKSkoUFRWlhIQEnTx5ssH6wsJCjR07VikpKdq5c6eSkpKUlJSkPXv2OGqqq6s1ZMgQPf3005fd9qRJk3TixAnH55lnnnG1fQAAYFA+rq7w3HPPadKkSZo4caIkKTs7W++9955eeeUVpaen16t/4YUXNHLkSM2aNUuS9MQTTyg/P19LlixRdna2JOmBBx6QJB08ePCy277uuusUHBzcpD5rampUU1Pj+HNVVVWT1gMAANcml47U1NbWqri4WBaL5ccJTCZZLBYVFRU1uE5RUZFTvSQlJCQ0Wn85b775pjp27Ki+fftqzpw5+v777xutzczMVEBAgOMTGhrq8vYAAMC1w6UjNeXl5aqrq1NQUJDTeFBQkPbt29fgOlartcF6q9XqUqP33XefunbtqpCQEO3atUuzZ8/WF198oXXr1jVYP2fOHKWlpTn+XFVVRbABAMDAXD795C6TJ092/HtkZKQ6d+6s4cOH6+uvv1aPHj3q1fv5+cnPz+9qtggAANzIpdNPHTt2lLe3t8rKypzGy8rKGr3WJTg42KX6poqNjZUk7d+//2fNAwAAjMGlUOPr66uYmBgVFBQ4xmw2mwoKChQXF9fgOnFxcU71kpSfn99ofVNduu27c+fOP2seAABgDC6ffkpLS9OECRM0YMAADRo0SFlZWaqurnbcDTV+/Hh16dJFmZmZkqTp06dr6NChWrx4sRITE7V69Wrt2LFDy5Ytc8xZUVGhw4cP6/jx45KkL774QtLFozzBwcH6+uuvtWrVKo0aNUqBgYHatWuXZs6cqdtuu039+vX72T8EAABw7XM51CQnJ+vUqVNasGCBrFaroqOjlZeX57gY+PDhwzKZfjwAFB8fr1WrVmnevHmaO3euwsPDlZOTo759+zpq1q9f7whFkjRmzBhJUkZGhhYuXChfX1+9//77jgAVGhqqu+66S/PmzbviLw4AAIzFy263293dxNVQVVWlgIAAVVZWyt/f393tANeUsPT33N2CYRxclOjuFgyBfbL5tPR90pXf37z7CQAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGAKhBgAAGMIVhZqlS5cqLCxMZrNZsbGx2rZt22Xr165dq4iICJnNZkVGRmrDhg1Oy9etW6cRI0YoMDBQXl5eKi0trTfHuXPn9PDDDyswMFDXX3+97rrrLpWVlV1J+wAAwIBcDjVr1qxRWlqaMjIyVFJSoqioKCUkJOjkyZMN1hcWFmrs2LFKSUnRzp07lZSUpKSkJO3Zs8dRU11drSFDhujpp59udLszZ87UP/7xD61du1Yffvihjh8/rt///veutg8AAAzKy263211ZITY2VgMHDtSSJUskSTabTaGhoUpNTVV6enq9+uTkZFVXVys3N9cxNnjwYEVHRys7O9up9uDBg+rWrZt27typ6Ohox3hlZaVuuOEGrVq1Snfffbckad++ferVq5eKioo0ePDgn+y7qqpKAQEBqqyslL+/vytfGfB4YenvubsFwzi4KNHdLRgC+2Tzaen7pCu/v106UlNbW6vi4mJZLJYfJzCZZLFYVFRU1OA6RUVFTvWSlJCQ0Gh9Q4qLi3X+/HmneSIiInTTTTc1Ok9NTY2qqqqcPgAAwLhcCjXl5eWqq6tTUFCQ03hQUJCsVmuD61itVpfqG5vD19dX7dq1a/I8mZmZCggIcHxCQ0ObvD0AAHDtMezdT3PmzFFlZaXjc+TIEXe3BAAAfkE+rhR37NhR3t7e9e46KisrU3BwcIPrBAcHu1Tf2By1tbU6ffq009Gay83j5+cnPz+/Jm8DAABc21w6UuPr66uYmBgVFBQ4xmw2mwoKChQXF9fgOnFxcU71kpSfn99ofUNiYmLUqlUrp3m++OILHT582KV5AACAcbl0pEaS0tLSNGHCBA0YMECDBg1SVlaWqqurNXHiREnS+PHj1aVLF2VmZkqSpk+frqFDh2rx4sVKTEzU6tWrtWPHDi1btswxZ0VFhQ4fPqzjx49LuhhYpItHaIKDgxUQEKCUlBSlpaWpQ4cO8vf3V2pqquLi4pp05xMAADA+l0NNcnKyTp06pQULFshqtSo6Olp5eXmOi4EPHz4sk+nHA0Dx8fFatWqV5s2bp7lz5yo8PFw5OTnq27evo2b9+vWOUCRJY8aMkSRlZGRo4cKFkqTnn39eJpNJd911l2pqapSQkKD//d//vaIvDQAAjMfl59Rcq3hODXDleCZI82npzwS5VrBPNp+Wvk/+Ys+pAQAAaKkINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINQAAwBAINS3Y0qVLFRYWJrPZrNjYWG3btu2y9WvXrlVERITMZrMiIyO1YcMGp+V2u10LFixQ586d1bp1a1ksFn311VdONWFhYfLy8nL6LFq0yLH8iy++0LBhwxQUFCSz2azu3btr3rx5On/+vKPm1VdfrTeH2Wx22s7Zs2c1depU3XjjjWrdurV69+6t7Ozset+pqKhI//Vf/6U2bdrI399ft912m3744QdJ0r/+9a9627n02b59e9N+yAAAwyDUtFBr1qxRWlqaMjIyVFJSoqioKCUkJOjkyZMN1hcWFmrs2LFKSUnRzp07lZSUpKSkJO3Zs8dR88wzz+jFF19Udna2tm7dqjZt2ighIUHnzp1zmuvxxx/XiRMnHJ/U1FTHslatWmn8+PH65z//qS+++EJZWVlavny5MjIynObw9/d3muPQoUNOy9PS0pSXl6c33nhDe/fu1YwZMzR16lStX7/eUVNUVKSRI0dqxIgR2rZtm7Zv366pU6fKZLq428bHxztt48SJE3rwwQfVrVs3DRgw4Mp+8ACAa5aPuxtAw5577jlNmjRJEydOlCRlZ2frvffe0yuvvKL09PR69S+88IJGjhypWbNmSZKeeOIJ5efna8mSJcrOzpbdbldWVpbmzZunO++8U5L0+uuvKygoSDk5ORozZoxjrrZt2yo4OLjBvrp3767u3bs7/ty1a1f961//0ubNm53qvLy8Gp1DuhjCJkyYoNtvv12SNHnyZL388svatm2bRo8eLUmaOXOmpk2b5vR9b7nlFse/+/r6Om3j/Pnzevfdd5WamiovL69Gtw0AMCaO1LRAtbW1Ki4ulsVicYyZTCZZLBYVFRU1uE5RUZFTvSQlJCQ46g8cOCCr1epUExAQoNjY2HpzLlq0SIGBgerfv7+effZZXbhwodFe9+/fr7y8PA0dOtRp/OzZs+ratatCQ0N155136rPPPnNaHh8fr/Xr1+vYsWOy2+364IMP9OWXX2rEiBGSpJMnT2rr1q3q1KmT4uPjFRQUpKFDh2rLli2N9rJ+/Xp9++23jiAIAPAshJoWqLy8XHV1dQoKCnIaDwoKktVqbXAdq9V62fpL//ypOadNm6bVq1frgw8+0JQpU/TUU0/p0Ucfrbe9+Ph4mc1mhYeH69e//rUef/xxx7JbbrlFr7zyit5991298cYbstlsio+P19GjRx01f/nLX9S7d2/deOON8vX11ciRI7V06VLddtttkqRvvvlGkrRw4UJNmjRJeXl5uvXWWzV8+PB61wFdsmLFCiUkJOjGG29scDkAwNg4/QQnaWlpjn/v16+ffH19NWXKFGVmZsrPz8+xbM2aNTpz5ow+/fRTzZo1S3/+858d4ScuLk5xcXGO2vj4ePXq1Usvv/yynnjiCUkXQ80nn3yi9evXq2vXrvroo4/08MMPKyQkRBaLRTabTZI0ZcoUx5GX/v37q6CgQK+88ooyMzOd+j569Kg2btyot95665f5wQAAWjxCTQvUsWNHeXt7q6yszGm8rKys0etUgoODL1t/6Z9lZWXq3LmzU010dHSjvcTGxurChQs6ePCg0/UsoaGhkqTevXurrq5OkydP1iOPPCJvb+96c7Rq1Ur9+/fX/v37JUk//PCD5s6dq7///e9KTEyUdDFAlZaW6s9//rMsFoujx969ezvN1atXLx0+fLjeNlauXKnAwEDH9TgAAM/D6acWyNfXVzExMSooKHCM2Ww2FRQUOB0B+XdxcXFO9ZKUn5/vqO/WrZuCg4OdaqqqqrR169ZG55Sk0tJSmUwmderUqdEam82m8+fPO46u/Ke6ujrt3r3bEVTOnz+v8+fPO+5iusTb29sxR1hYmEJCQvTFF1841Xz55Zfq2rWr05jdbtfKlSs1fvx4tWrVqtE+AQDGxpGaFiotLU0TJkzQgAEDNGjQIGVlZam6utpxKmb8+PHq0qWL4zTM9OnTNXToUC1evFiJiYlavXq1duzYoWXLlkm6eDfSjBkz9Kc//Unh4eHq1q2b5s+fr5CQECUlJUm6eLHx1q1bNWzYMLVt21ZFRUWaOXOm7r//frVv316S9Oabb6pVq1aKjIyUn5+fduzYoTlz5ig5OdkRKB5//HENHjxYPXv21OnTp/Xss8/q0KFDevDBByVdvN176NChmjVrllq3bq2uXbvqww8/1Ouvv67nnnvO0e+sWbOUkZGhqKgoRUdH67XXXtO+ffv09ttvO/2sNm3apAMHDjjmBwB4JkJNC5WcnKxTp05pwYIFslqtio6OVl5enuNC38OHDzsd6YiPj9eqVas0b948zZ07V+Hh4crJyVHfvn0dNY8++qiqq6s1efJknT59WkOGDFFeXp7jwXh+fn5avXq1Fi5cqJqaGnXr1k0zZ850us7Gx8dHTz/9tL788kvZ7XZ17dpVU6dO1cyZMx013333nSZNmiSr1ar27dsrJiZGhYWFTqeSVq9erTlz5mjcuHGqqKhQ165d9eSTT+qhhx5y1MyYMUPnzp3TzJkzVVFRoaioKOXn56tHjx5OP6sVK1YoPj5eERERzfTTBwBci7zsdrvd3U1cDVVVVQoICFBlZaX8/f3d3Q5wTQlLf8/dLRjGwUWJ7m7BENgnm09L3ydd+f3NNTUAAMAQruj009KlS/Xss8/KarUqKipKf/nLXzRo0KBG69euXav58+fr4MGDCg8P19NPP61Ro0Y5ltvtdmVkZGj58uU6ffq0fvWrX+mll15SeHi4oyYsLKzeo/YzMzMbfLrutY6/gTSPlv63DwBA83L5SE1LfScRAADwbC6Hmn9/J9GlNytfd911euWVVxqs//d3EvXq1UtPPPGEbr31Vi1ZskSS6r2TqF+/fnr99dd1/Phx5eTkOM116Z1Elz5t2rRptM+amhpVVVU5fQAAgHG5FGqupXcSZWZmKiAgwPG59LA4AABgTC5dU3O5dxLt27evwXWa851Et956qzp06KDCwkLNmTNHJ06ccDzX5D/NmTPH6Vbkqqoqgg0AAAZ2zTynpqnvJLrEz8+vwXEAAGBMLp1++qXfSdTUOSXndxIBAAC4FGqutXcSAQAAz+Hy6aeW+k4iAADg2VwONS31nUQAAMCz8e6nFognCjcPnijcfNgnmw/7ZfNgn2w+LX2f5N1PAADA4xBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIRBqAACAIVxRqFm6dKnCwsJkNpsVGxurbdu2XbZ+7dq1ioiIkNlsVmRkpDZs2OC03G63a8GCBercubNat24ti8Wir776yqmmoqJC48aNk7+/v9q1a6eUlBSdPXv2StoHAAAG5HKoWbNmjdLS0pSRkaGSkhJFRUUpISFBJ0+ebLC+sLBQY8eOVUpKinbu3KmkpCQlJSVpz549jppnnnlGL774orKzs7V161a1adNGCQkJOnfunKNm3Lhx+uyzz5Sfn6/c3Fx99NFHmjx58hV8ZQAAYERedrvd7soKsbGxGjhwoJYsWSJJstlsCg0NVWpqqtLT0+vVJycnq7q6Wrm5uY6xwYMHKzo6WtnZ2bLb7QoJCdEjjzyiP/7xj5KkyspKBQUF6dVXX9WYMWO0d+9e9e7dW9u3b9eAAQMkSXl5eRo1apSOHj2qkJCQn+y7qqpKAQEBqqyslL+/vytf+aoLS3/P3S0YwsFFie5uwTDYJ5sP+2XzYJ9sPi19n3Tl97ePKxPX1taquLhYc+bMcYyZTCZZLBYVFRU1uE5RUZHS0tKcxhISEpSTkyNJOnDggKxWqywWi2N5QECAYmNjVVRUpDFjxqioqEjt2rVzBBpJslgsMplM2rp1q373u9/V225NTY1qamocf66srJR08YfT0tlqvnd3C4ZwLfxvfa1gn2w+7JfNg32y+bT0ffJSf005BuNSqCkvL1ddXZ2CgoKcxoOCgrRv374G17FarQ3WW61Wx/JLY5er6dSpk3PjPj7q0KGDo+Y/ZWZm6rHHHqs3Hhoa2tjXg8EEZLm7A6A+9ku0NNfKPnnmzBkFBARctsalUHMtmTNnjtMRIpvNpoqKCgUGBsrLy8uNnV37qqqqFBoaqiNHjrT4U3nwDOyTaInYL5uH3W7XmTNnmnSpiUuhpmPHjvL29lZZWZnTeFlZmYKDgxtcJzg4+LL1l/5ZVlamzp07O9VER0c7av7zQuQLFy6ooqKi0e36+fnJz8/Paaxdu3aX/4Jwib+/P/9HRYvCPomWiP3y5/upIzSXuHT3k6+vr2JiYlRQUOAYs9lsKigoUFxcXIPrxMXFOdVLUn5+vqO+W7duCg4OdqqpqqrS1q1bHTVxcXE6ffq0iouLHTWbNm2SzWZTbGysK18BAAAYlMunn9LS0jRhwgQNGDBAgwYNUlZWlqqrqzVx4kRJ0vjx49WlSxdlZmZKkqZPn66hQ4dq8eLFSkxM1OrVq7Vjxw4tW7ZMkuTl5aUZM2boT3/6k8LDw9WtWzfNnz9fISEhSkpKkiT16tVLI0eO1KRJk5Sdna3z589r6tSpGjNmTJMORwEAAONzOdQkJyfr1KlTWrBggaxWq6Kjo5WXl+e40Pfw4cMymX48ABQfH69Vq1Zp3rx5mjt3rsLDw5WTk6O+ffs6ah599FFVV1dr8uTJOn36tIYMGaK8vDyZzWZHzZtvvqmpU6dq+PDhMplMuuuuu/Tiiy/+nO+OK+Tn56eMjIx6p/cAd2GfREvEfnn1ufycGgAAgJaIdz8BAABDINQAAABDINQAAABDINQAAABDINQAAABDMOxrEgAAuNq++uorffDBBzp58qRsNpvTsgULFripK8/BLd0AADSD5cuX67//+7/VsWNHBQcHO71n0MvLSyUlJW7szjMQatBkp0+f1ttvv62vv/5as2bNUocOHVRSUqKgoCB16dLF3e3BA9ntdr399tuN/s143bp1buoMnqhr1676n//5H82ePdvdrXgsTj+hSXbt2iWLxaKAgAAdPHhQkyZNUocOHbRu3TodPnxYr7/+urtbhAeaMWOGXn75ZQ0bNkxBQUFOfzMGrrbvvvtO99xzj7vb8GgcqUGTWCwW3XrrrXrmmWfUtm1bffrpp+revbsKCwt133336eDBg+5uER6oQ4cOeuONNzRq1Ch3twIoJSVFAwcO1EMPPeTuVjwWR2rQJNu3b9fLL79cb7xLly6yWq1u6AiQAgIC1L17d3e3AUiSevbsqfnz5+uTTz5RZGSkWrVq5bR82rRpburMc3CkBk3SqVMnbdy4Uf3793c6UpOfn6//83/+j44cOeLuFuGBXnvtNeXl5emVV15R69at3d0OPFy3bt0aXebl5aVvvvnmKnbjmQg1aJIHH3xQ3377rd566y116NBBu3btkre3t5KSknTbbbcpKyvL3S3CA/3www/63e9+p48//lhhYWH1/mbM3SaAZyHUoEkqKyt19913a8eOHTpz5oxCQkJktVoVFxenDRs2qE2bNu5uER7o3nvv1QcffKC77767wQuFMzIy3NQZPFltba0OHDigHj16yMeHqzyuJkINXLJlyxbt2rVLZ8+e1a233iqLxeLuluDB2rRpo40bN2rIkCHubgXQ999/r9TUVL322muSpC+//FLdu3dXamqqunTpovT0dDd3aHxESLhkyJAh/AJBixEaGip/f393twFIkubMmaNPP/1U//rXvzRy5EjHuMVi0cKFCwk1VwHvfkKTffjhh/rtb3+rnj17qmfPnho9erQ2b97s7rbgwRYvXqxHH32URwqgRcjJydGSJUs0ZMgQp1Ohffr00ddff+3GzjwHoQZN8sYbb8hisei6667TtGnTNG3aNJnNZg0fPlyrVq1yd3vwUPfff78++OAD9ejRQ23btlWHDh2cPsDVdOrUKXXq1KneeHV1NQ+GvEq4pgZN0qtXL02ePFkzZ850Gn/uuee0fPly7d27102dwZNdunahMRMmTLhKnQDSbbfdpnvuuUepqalq27atdu3apW7duik1NVVfffWV8vLy3N2i4RFq0CR+fn767LPP1LNnT6fx/fv3q2/fvjp37pybOgOAlmHLli264447dP/99+vVV1/VlClT9Pnnn6uwsFAffvihYmJi3N2i4XH6CU0SGhqqgoKCeuPvv/++QkND3dAR4OzcuXOqqqpy+gBX05AhQ1RaWqoLFy4oMjJS//znP9WpUycVFRURaK4S7n5CkzzyyCOaNm2aSktLFR8fL0n6+OOP9eqrr+qFF15wc3fwVNXV1Zo9e7beeustffvtt/WW19XVuaEreLIePXpo+fLl7m7DY3GkBk3y3//931q9erV2796tGTNmaMaMGdqzZ4/WrFmjKVOmuLs9eKhHH31UmzZt0ksvvSQ/Pz/99a9/1WOPPaaQkBDeHI+rrqSkRLt373b8+d1331VSUpLmzp2r2tpaN3bmObimBsA166abbtLrr7+u22+/Xf7+/iopKVHPnj31t7/9Tf/3//5fbdiwwd0twoMMHDhQ6enpuuuuu/TNN9+od+/e+v3vf6/t27crMTGR18lcBRypQZNs375dW7durTe+detW7dixww0dAVJFRYXjLd3+/v6qqKiQdPHaho8++sidrcEDffnll4qOjpYkrV27VkOHDtWqVav06quv6p133nFvcx6CUIMmefjhhxt8E/exY8f08MMPu6EjQOrevbsOHDggSYqIiNBbb70lSfrHP/6hdu3aubEzeCK73S6bzSbp4k0Uo0aNknTxRovy8nJ3tuYxCDVoks8//1y33nprvfH+/fvr888/d0NHgDRx4kR9+umnkqT09HQtXbpUZrNZM2fO1KxZs9zcHTzNgAED9Kc//Ul/+9vf9OGHHyoxMVGSdODAAQUFBbm5O8/A3U9oEj8/P5WVlTkO9V9y4sQJ3kILt/n3h0FaLBbt27dPxcXF6tmzp/r16+fGzuCJsrKyNG7cOOXk5Oj/+//+P8dzvd5++23HXaP4ZXGhMJpk7NixOnHihN59910FBARIkk6fPq2kpCR16tTJcdgfcJdz587JbDa7uw14qLq6On388ceKjIxU+/btnZadO3dO3t7eatWqlZu68xyEGjTJsWPHdNttt+nbb79V//79JUmlpaUKCgpSfn4+D+CDW9TV1empp55Sdna2ysrK9OWXX6p79+6aP3++wsLClJKS4u4W4UHMZrP27t2rbt26ubsVj8U1NWiSLl26aNeuXXrmmWfUu3dvxcTE6IUXXtDu3bsJNHCbJ598Uq+++qqeeeYZ+fr6Osb79u2rv/71r27sDJ6ob9+++uabb9zdhkfjSA2Aa1bPnj318ssva/jw4Wrbtq0+/fRTde/eXfv27VNcXJy+++47d7cID5KXl6c5c+boiSeeUExMjNq0aeO03N/f302deQ6O1KBJXnvtNb333nuOPz/66KNq166d4uPjdejQITd2Bk927Nixei9ZlSSbzabz58+7oSN4slGjRunTTz/V6NGjdeONN6p9+/Zq37692rVrV+86G/wyuG0FTfLUU0/ppZdekiQVFRVpyZIlysrKUm5urmbOnKl169a5uUN4ot69e2vz5s3q2rWr0/jbb7/tuPYLuFo++OADd7fg8Qg1aJIjR444/kack5Oju+++W5MnT9avfvUr3X777e5tDh5rwYIFmjBhgo4dOyabzaZ169bpiy++0Ouvv67c3Fx3twcPM3ToUHe34PE4/YQmuf766x1vQf7nP/+p3/zmN5IuXu3/ww8/uLM1eLA777xT//jHP/T++++rTZs2WrBggfbu3at//OMfjn0UuJo2b96s+++/X/Hx8Tp27Jgk6W9/+5u2bNni5s48A6EGTfKb3/xGDz74oB588EF9+eWXjsd/f/bZZwoLC3Nvc/BIFy5c0OOPP65u3bopPz9fJ0+e1Pfff68tW7ZoxIgR7m4PHuidd95RQkKCWrdurZKSEtXU1EiSKisr9dRTT7m5O89AqEGTLF26VHFxcTp16pTeeecdBQYGSpKKi4s1duxYN3cHT+Tj46NnnnlGFy5ccHcrgCTpT3/6k7Kzs7V8+XKnB+396le/UklJiRs78xzc0g3gmnXnnXfq97//vSZMmODuVgBdd911+vzzzxUWFub0iIFvvvlGvXv31rlz59zdouFxoTAatWvXLvXt21cmk0m7du26bC3v2YE73HHHHUpPT9fu3bsbfC7I6NGj3dQZPFFwcLD2799f75T8li1b6r03D78MjtSgUSaTSVarVZ06dZLJZJKXl5ca2l28vLxUV1fnhg7h6Uymxs+gs1/iasvMzNQbb7yhV155Rb/5zW+0YcMGHTp0SDNnztT8+fOVmprq7hYNjyM1aNSBAwd0ww03OP4daGlsNpu7WwAc0tPTZbPZNHz4cH3//fe67bbb5Ofnpz/+8Y8EmquEIzVokm+//dZxcfCRI0e0fPly/fDDDxo9erR+/etfu7k7eCKbzaZXX31V69at08GDB+Xl5aXu3bvrrrvu0gMPPCAvLy93twgPVVtbq/379+vs2bPq3bu3rr/+ene35DEINbis3bt367e//a2OHDmi8PBwrV69WiNHjlR1dbVMJpOqq6v19ttvKykpyd2twoPY7Xb99re/1YYNGxQVFaWIiAjZ7Xbt3btXu3fv1ujRo5WTk+PuNgFcZYQaXNYdd9whHx8fpaen629/+5tyc3OVkJCg5cuXS5JSU1NVXFysTz75xM2dwpOsXLlS06dP17vvvqthw4Y5Ldu0aZOSkpK0ZMkSjR8/3k0dwhP97ne/a/AIoZeXl8xms3r27Kn77rtPt9xyixu68wyEGlxWx44dtWnTJvXr109nz56Vv7+/tm/frpiYGEnSvn37NHjwYJ0+fdq9jcKjjBgxQv/1X/+l9PT0Bpc/9dRT+vDDD7Vx48ar3Bk82R/+8Afl5OSoXbt2jv9GlpSU6PTp0xoxYoQ+/fRTHTx4UAUFBfrVr37l5m6NiYfv4bIqKioUHBws6eKrEtq0aeP0ttn27dvrzJkz7moPHmrXrl0aOXJko8vvuOMOffrpp1exI+DiLd333XefvvnmG73zzjt655139PXXX+v+++9Xjx49tHfvXk2YMEGzZ892d6uGRajBT/rPw6lcgAl3q6ioUFBQUKPLg4KC9N13313FjgBpxYoVmjFjhtOjBkwmk1JTU7Vs2TJ5eXlp6tSp2rNnjxu7NDZu6cZP+sMf/iA/Pz9J0rlz5/TQQw85HnJ26d0mwNVUV1cnH5/G//Pl7e3N6xNw1V24cEH79u3TzTff7DS+b98+xzOTzGYzfzH8BRFqcFn/+fj5+++/v14NF2PiarPb7U5h+z8RtuEODzzwgFJSUjR37lwNHDhQkrR9+3Y99dRTjv9Ofvjhh+rTp4872zQ0LhQGcM2ZOHFik+pWrlz5C3cC/Kiurk6LFi3SkiVLVFZWJuniqdDU1FTNnj1b3t7eOnz4sEwmk2688UY3d2tMhBoAAH6mCxcuaNWqVUpISFBQUJCqqqokSf7+/m7uzLMQagAAaAbXXXed9u7dq65du7q7FY/F3U8AADSDQYMGaefOne5uw6NxoTAAAM3gf/7nf/TII4/o6NGjiomJcdwlekm/fv3c1Jnn4PQTAADN4N+fT3OJl5eX7Ha7vLy8HLd145fDkRoAAJrBgQMH3N2Cx+NIDQAAMASO1AAA0Ey+/vprZWVlae/evZKk3r17a/r06erRo4ebO/MM3P0EAEAz2Lhxo3r37q1t27apX79+6tevn7Zu3ao+ffooPz/f3e15BE4/AQDQDPr376+EhAQtWrTIaTw9PV3//Oc/VVJS4qbOPAehBgCAZmA2m7V7926Fh4c7jX/55Zfq16+fzp0756bOPAennwAAaAY33HCDSktL642XlpaqU6dOV78hD8SFwgAA/AyPP/64/vjHP2rSpEmaPHmyvvnmG8XHx0uSPv74Yz399NNKS0tzc5eegdNPAAD8DN7e3jpx4oRuuOEGZWVlafHixTp+/LgkKSQkRLNmzdK0adPk5eXl5k6Nj1ADAMDPYDKZZLVanU4xnTlzRpLUtm1bd7XlkTj9BADAz/SfR2EIM+7BkRoAAH4Gk8mkgICAnzy9VFFRcZU68lwcqQEA4Gd67LHHFBAQ4O42PB5HagAA+BkauqYG7sFzagAA+Bm4q6nlINQAAPAzcMKj5eD0EwAAMASO1AAAAEMg1AAAAEMg1AAAAEMg1AAAAEMg1AAAAEMg1AAAAEMg1AAAAEP4/wFPDqP9Hns85AAAAABJRU5ErkJggg==",
            "text/plain": [
              "\u003cFigure size 640x480 with 1 Axes\u003e"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "ax = stats['abs_change_pct'].plot.bar()\n",
        "ax.bar_label(ax.containers[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "metadata": {
        "tags": [],
        "id": "HVYgjvVvxLVM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The max change in percentage(L-Infinity Distance) is 0.04250395810498114\n"
          ]
        }
      ],
      "source": [
        "print(f\"The max change in percentage(L-Infinity Distance) is {max(stats['abs_change_pct'])}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ehOoYfEMxLVM"
      },
      "source": [
        "### Jensen-Shannon Divergence"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 63,
      "metadata": {
        "tags": [],
        "id": "4bC9aOm7xLVM"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003etrain_n\u003c/th\u003e\n",
              "      \u003cth\u003eserve_n\u003c/th\u003e\n",
              "      \u003cth\u003etrain_pct\u003c/th\u003e\n",
              "      \u003cth\u003eserve_pct\u003c/th\u003e\n",
              "      \u003cth\u003eabs_change_pct\u003c/th\u003e\n",
              "      \u003cth\u003emix\u003c/th\u003e\n",
              "      \u003cth\u003etrain_kl\u003c/th\u003e\n",
              "      \u003cth\u003eserve_kl\u003c/th\u003e\n",
              "      \u003cth\u003eJSD\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003eBiscoe\u003c/th\u003e\n",
              "      \u003ctd\u003e116\u003c/td\u003e\n",
              "      \u003ctd\u003e1054\u003c/td\u003e\n",
              "      \u003ctd\u003e0.487395\u003c/td\u003e\n",
              "      \u003ctd\u003e0.492754\u003c/td\u003e\n",
              "      \u003ctd\u003e0.005359\u003c/td\u003e\n",
              "      \u003ctd\u003e0.490074\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.003855\u003c/td\u003e\n",
              "      \u003ctd\u003e0.003876\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000011\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003eDream\u003c/th\u003e\n",
              "      \u003ctd\u003e86\u003c/td\u003e\n",
              "      \u003ctd\u003e682\u003c/td\u003e\n",
              "      \u003ctd\u003e0.361345\u003c/td\u003e\n",
              "      \u003ctd\u003e0.318841\u003c/td\u003e\n",
              "      \u003ctd\u003e0.042504\u003c/td\u003e\n",
              "      \u003ctd\u003e0.340093\u003c/td\u003e\n",
              "      \u003ctd\u003e0.031599\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.029682\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000959\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003eTorgersen\u003c/th\u003e\n",
              "      \u003ctd\u003e36\u003c/td\u003e\n",
              "      \u003ctd\u003e403\u003c/td\u003e\n",
              "      \u003ctd\u003e0.151261\u003c/td\u003e\n",
              "      \u003ctd\u003e0.188406\u003c/td\u003e\n",
              "      \u003ctd\u003e0.037145\u003c/td\u003e\n",
              "      \u003ctd\u003e0.169833\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.025273\u003c/td\u003e\n",
              "      \u003ctd\u003e0.028209\u003c/td\u003e\n",
              "      \u003ctd\u003e0.001468\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "           train_n  serve_n  train_pct  serve_pct  abs_change_pct       mix  \\\n",
              "Biscoe         116     1054   0.487395   0.492754        0.005359  0.490074   \n",
              "Dream           86      682   0.361345   0.318841        0.042504  0.340093   \n",
              "Torgersen       36      403   0.151261   0.188406        0.037145  0.169833   \n",
              "\n",
              "           train_kl  serve_kl       JSD  \n",
              "Biscoe    -0.003855  0.003876  0.000011  \n",
              "Dream      0.031599 -0.029682  0.000959  \n",
              "Torgersen -0.025273  0.028209  0.001468  "
            ]
          },
          "execution_count": 63,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "stats['mix'] = (stats['train_pct'] + stats['serve_pct']) / 2\n",
        "stats['train_kl'] = stats['train_pct'] * np.log2(stats['train_pct'] / stats['mix'])\n",
        "stats['serve_kl'] = stats['serve_pct'] * np.log2(stats['serve_pct'] / stats['mix'])\n",
        "stats['JSD'] = (stats['train_kl'] + stats['serve_kl']) / 2\n",
        "stats"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 64,
      "metadata": {
        "tags": [],
        "id": "LblWEfuRxLVM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The total for the Jansen-Shannon Divergence is 0.0024371908205822954\n"
          ]
        }
      ],
      "source": [
        "print(f\"The total for the Jansen-Shannon Divergence is {sum(stats['JSD'])}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XvFEwy3PxLVM"
      },
      "source": [
        "### Verify Calculations with `ML.VALIDATE_DATA_DRIFT`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 65,
      "metadata": {
        "tags": [],
        "id": "ESj7jrcdxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c67a28f47e5e4fa78ff5bead7e1299b6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7ec068d8da9c4ba59c0b6b5fe1453c61",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003einput\u003c/th\u003e\n",
              "      \u003cth\u003emetric\u003c/th\u003e\n",
              "      \u003cth\u003ethreshold\u003c/th\u003e\n",
              "      \u003cth\u003evalue\u003c/th\u003e\n",
              "      \u003cth\u003eis_anomaly\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.042504\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "    input   metric  threshold     value  is_anomaly\n",
              "0  island  L_INFTY        0.3  0.042504       False"
            ]
          },
          "execution_count": 65,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.VALIDATE_DATA_DRIFT(\n",
        "    (\n",
        "        SELECT island\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "        WHERE splits = 'TRAIN'\n",
        "    ),\n",
        "    (\n",
        "        SELECT island\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "    ),\n",
        "    STRUCT('L_INFTY' AS categorical_metric_type)\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 66,
      "metadata": {
        "tags": [],
        "id": "_tpHjAaxxLVM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The max change in percentage(L-Infinity Distance) is 0.04250395810498114\n"
          ]
        }
      ],
      "source": [
        "print(f\"The max change in percentage(L-Infinity Distance) is {max(stats['abs_change_pct'])}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 67,
      "metadata": {
        "tags": [],
        "id": "vl9yM5KaxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "35d6f95e8ea04d32b0b843a9c787d0c9",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "4e50263f3cac4c38b002d282ddda5c56",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003einput\u003c/th\u003e\n",
              "      \u003cth\u003emetric\u003c/th\u003e\n",
              "      \u003cth\u003ethreshold\u003c/th\u003e\n",
              "      \u003cth\u003evalue\u003c/th\u003e\n",
              "      \u003cth\u003eis_anomaly\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.002437\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "    input                     metric  threshold     value  is_anomaly\n",
              "0  island  JENSEN_SHANNON_DIVERGENCE        0.3  0.002437       False"
            ]
          },
          "execution_count": 67,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.VALIDATE_DATA_DRIFT(\n",
        "    (\n",
        "        SELECT island\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "        WHERE splits = 'TRAIN'\n",
        "    ),\n",
        "    (\n",
        "        SELECT island\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "    ),\n",
        "    STRUCT('JENSEN_SHANNON_DIVERGENCE' AS categorical_metric_type)\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 68,
      "metadata": {
        "tags": [],
        "id": "bdhhX8L4xLVM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The total for the Jansen-Shannon Divergence is 0.0024371908205822954\n"
          ]
        }
      ],
      "source": [
        "print(f\"The total for the Jansen-Shannon Divergence is {sum(stats['JSD'])}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1FJ2SZ0XxLVM"
      },
      "source": [
        "---\n",
        "## More Applications For Model Monitoring"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AsFGk1nUxLVM"
      },
      "source": [
        "---\n",
        "### Monitoring Transformed Features\n",
        "\n",
        "The [`CREATE MODEL`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create) statement included a [`TRANSFORM`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create#transform).  This is available to be called directly for input data with the [`ML.TRANSFORM`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-transform) function.  This can be useful for considering monitoring the transformed features."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s0dsoxxIxLVM"
      },
      "source": [
        "#### Transformations With `ML.TRANSFORM`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 69,
      "metadata": {
        "tags": [],
        "id": "eAhIwFzNxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "4eaea820a9d04dea923a2a76da66ef54",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "01d4242dd4f3485f898ff584f749175b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003ebody_mass_g\u003c/th\u003e\n",
              "      \u003cth\u003eculmen_length_mm\u003c/th\u003e\n",
              "      \u003cth\u003eculmen_depth_mm\u003c/th\u003e\n",
              "      \u003cth\u003eflipper_length_mm\u003c/th\u003e\n",
              "      \u003cth\u003esex\u003c/th\u003e\n",
              "      \u003cth\u003eisland\u003c/th\u003e\n",
              "      \u003cth\u003especies\u003c/th\u003e\n",
              "      \u003cth\u003einstance_timestamp\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003e0.891275\u003c/td\u003e\n",
              "      \u003ctd\u003e0.132769\u003c/td\u003e\n",
              "      \u003ctd\u003e-1.432310\u003c/td\u003e\n",
              "      \u003ctd\u003ebin_3\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-18 00:00:00+00:00\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNaN\u003c/td\u003e\n",
              "      \u003ctd\u003eNone\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-18 00:00:00+00:00\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003e0.668453\u003c/td\u003e\n",
              "      \u003ctd\u003e-1.829147\u003c/td\u003e\n",
              "      \u003ctd\u003e0.487270\u003c/td\u003e\n",
              "      \u003ctd\u003ebin_3\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "      \u003ctd\u003eTorgersen\u003c/td\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-18 00:00:00+00:00\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003e1.031657\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.338846\u003c/td\u003e\n",
              "      \u003ctd\u003e1.548091\u003c/td\u003e\n",
              "      \u003ctd\u003ebin_3\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "      \u003ctd\u003eTorgersen\u003c/td\u003e\n",
              "      \u003ctd\u003eAdelie Penguin (Pygoscelis adeliae)\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-18 00:00:00+00:00\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003e1.552640\u003c/td\u003e\n",
              "      \u003ctd\u003e1.189185\u003c/td\u003e\n",
              "      \u003ctd\u003e-1.078703\u003c/td\u003e\n",
              "      \u003ctd\u003ebin_3\u003c/td\u003e\n",
              "      \u003ctd\u003eMALE\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-18 00:00:00+00:00\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "   body_mass_g  culmen_length_mm  culmen_depth_mm flipper_length_mm     sex  \\\n",
              "0     0.891275          0.132769        -1.432310             bin_3  FEMALE   \n",
              "1          NaN               NaN              NaN              None  FEMALE   \n",
              "2     0.668453         -1.829147         0.487270             bin_3  FEMALE   \n",
              "3     1.031657         -0.338846         1.548091             bin_3  FEMALE   \n",
              "4     1.552640          1.189185        -1.078703             bin_3    MALE   \n",
              "\n",
              "      island                              species        instance_timestamp  \n",
              "0     Biscoe    Gentoo penguin (Pygoscelis papua) 2024-04-18 00:00:00+00:00  \n",
              "1     Biscoe    Gentoo penguin (Pygoscelis papua) 2024-04-18 00:00:00+00:00  \n",
              "2  Torgersen  Adelie Penguin (Pygoscelis adeliae) 2024-04-18 00:00:00+00:00  \n",
              "3  Torgersen  Adelie Penguin (Pygoscelis adeliae) 2024-04-18 00:00:00+00:00  \n",
              "4     Biscoe    Gentoo penguin (Pygoscelis papua) 2024-04-18 00:00:00+00:00  "
            ]
          },
          "execution_count": 69,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.TRANSFORM(\n",
        "    MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "        LIMIT 5\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "btM5KOpDxLVM"
      },
      "source": [
        "#### Monitoring Skew (Manually) - As Drift from Training Data With `ML.VALIDATE_DATA_DRIFT`\n",
        "\n",
        "Compare today, or any data, to the training data by directly comparing to the the model created in BigQuery."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 70,
      "metadata": {
        "tags": [],
        "id": "ULW1asARxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d0aa67047e624524b0258eb8291248b6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "bbe82f6836784e99a4c71fbb65733555",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003einput\u003c/th\u003e\n",
              "      \u003cth\u003emetric\u003c/th\u003e\n",
              "      \u003cth\u003ethreshold\u003c/th\u003e\n",
              "      \u003cth\u003evalue\u003c/th\u003e\n",
              "      \u003cth\u003eis_anomaly\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003ebody_mass_g\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.740493\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_depth_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.087689\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.269247\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eflipper_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.641350\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.042504\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e5\u003c/th\u003e\n",
              "      \u003ctd\u003esex\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.088540\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e6\u003c/th\u003e\n",
              "      \u003ctd\u003especies\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.066070\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "               input                     metric  threshold     value  \\\n",
              "0        body_mass_g  JENSEN_SHANNON_DIVERGENCE        0.3  0.740493   \n",
              "1    culmen_depth_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.087689   \n",
              "2   culmen_length_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.269247   \n",
              "3  flipper_length_mm                    L_INFTY        0.3  0.641350   \n",
              "4             island                    L_INFTY        0.3  0.042504   \n",
              "5                sex                    L_INFTY        0.3  0.088540   \n",
              "6            species                    L_INFTY        0.3  0.066070   \n",
              "\n",
              "   is_anomaly  \n",
              "0        True  \n",
              "1       False  \n",
              "2       False  \n",
              "3        True  \n",
              "4       False  \n",
              "5       False  \n",
              "6       False  "
            ]
          },
          "execution_count": 70,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.VALIDATE_DATA_DRIFT(\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM ML.TRANSFORM(\n",
        "            MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "            (\n",
        "                SELECT *\n",
        "                FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "                WHERE splits = 'TRAIN'\n",
        "            )\n",
        "        )\n",
        "    ),\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM ML.TRANSFORM(\n",
        "            MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "            (\n",
        "                SELECT *\n",
        "                FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "                WHERE DATE(instance_timestamp) \u003e= CURRENT_DATE()\n",
        "            )\n",
        "        )\n",
        "    )\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UEx4YZUtxLVM"
      },
      "source": [
        "#### Monitoring Drift (Change Over Time) With `ML.VALIDATE_DATA_DRIFT`\n",
        "\n",
        "Compare today to 5 days ago and set the a lower, more sensitive, threshold of `0.03` (default is `0.3`) for all features."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 71,
      "metadata": {
        "tags": [],
        "id": "MrnUQbh4xLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3347d2e1f3f0412db64483f91b096216",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "45497cfa7f77469b9fe761726b1fff9f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003einput\u003c/th\u003e\n",
              "      \u003cth\u003emetric\u003c/th\u003e\n",
              "      \u003cth\u003ethreshold\u003c/th\u003e\n",
              "      \u003cth\u003evalue\u003c/th\u003e\n",
              "      \u003cth\u003eis_anomaly\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003ebody_mass_g\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.059447\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_depth_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000000\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000000\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eflipper_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000000\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003einstance_timestamp\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e1.000000\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e5\u003c/th\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000000\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e6\u003c/th\u003e\n",
              "      \u003ctd\u003esex\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000000\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e7\u003c/th\u003e\n",
              "      \u003ctd\u003especies\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000000\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                input                     metric  threshold     value  \\\n",
              "0         body_mass_g  JENSEN_SHANNON_DIVERGENCE       0.03  0.059447   \n",
              "1     culmen_depth_mm  JENSEN_SHANNON_DIVERGENCE       0.03  0.000000   \n",
              "2    culmen_length_mm  JENSEN_SHANNON_DIVERGENCE       0.03  0.000000   \n",
              "3   flipper_length_mm                    L_INFTY       0.03  0.000000   \n",
              "4  instance_timestamp                    L_INFTY       0.03  1.000000   \n",
              "5              island                    L_INFTY       0.03  0.000000   \n",
              "6                 sex                    L_INFTY       0.03  0.000000   \n",
              "7             species                    L_INFTY       0.03  0.000000   \n",
              "\n",
              "   is_anomaly  \n",
              "0        True  \n",
              "1       False  \n",
              "2       False  \n",
              "3       False  \n",
              "4        True  \n",
              "5       False  \n",
              "6       False  \n",
              "7       False  "
            ]
          },
          "execution_count": 71,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.VALIDATE_DATA_DRIFT(\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM ML.TRANSFORM(\n",
        "            MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "            (\n",
        "                SELECT *\n",
        "                FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "                WHERE DATE(instance_timestamp) = DATE_SUB(CURRENT_DATE(), INTERVAL 5 DAY)\n",
        "            )\n",
        "        )\n",
        "    ),\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM ML.TRANSFORM(\n",
        "            MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "            (\n",
        "                SELECT *\n",
        "                FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "                WHERE DATE(instance_timestamp) \u003e= CURRENT_DATE()\n",
        "            )\n",
        "        )\n",
        "    ),\n",
        "    STRUCT(\n",
        "        0.03 AS categorical_default_threshold,\n",
        "        0.03 AS numerical_default_threshold\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "6JipjqrWxLVM"
      },
      "source": [
        "---\n",
        "### Monitoring Feature Attributions\n",
        "\n",
        "BigQuery ML models, like the deep neural network classifier (`model_type = 'DNN_CLASSIFIER'`) used above, can also serve explanations using the BigQuery function [`ML.EXPLAIN_PREDICT`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-explain-predict).  This function returns feature attributions for each feature used in the model to explain how much contribution was made to the final prediction for a given instance.\n",
        "\n",
        "**Feature Attributions**\n",
        "\n",
        "The feature attributions from `ML.EXPLAIN_PREDICT` are [local explanations](https://cloud.google.com/bigquery/docs/xai-overview#local_versus_global_explainability), specific to the instance being predicted.  The attribution values are given for each feature and they add up to the `prediction_value`. In this case the model is a classifier and the `prediction_value` is the logit value (log-odds) of the predicted class.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MYd9Oj4QxLVM"
      },
      "source": [
        "#### Serving Feature Attributions\n",
        "\n",
        "As an example, the following cell gets the feature attributions for a single penguin with for the latest values of the features:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 72,
      "metadata": {
        "tags": [],
        "id": "hkvRzx2bxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "59abb1aab2874061a6315c5a36d1dc24",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5ee203344a4c48c0b681ebef90cf8c6a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003epredicted_species\u003c/th\u003e\n",
              "      \u003cth\u003eprobability\u003c/th\u003e\n",
              "      \u003cth\u003ebaseline_prediction_value\u003c/th\u003e\n",
              "      \u003cth\u003eprediction_value\u003c/th\u003e\n",
              "      \u003cth\u003eapproximation_error\u003c/th\u003e\n",
              "      \u003cth\u003efeature\u003c/th\u003e\n",
              "      \u003cth\u003eattribution\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e0.814165\u003c/td\u003e\n",
              "      \u003ctd\u003e0.546813\u003c/td\u003e\n",
              "      \u003ctd\u003e1.94978\u003c/td\u003e\n",
              "      \u003ctd\u003e0.0\u003c/td\u003e\n",
              "      \u003ctd\u003eflipper_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003e0.709371\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e0.814165\u003c/td\u003e\n",
              "      \u003ctd\u003e0.546813\u003c/td\u003e\n",
              "      \u003ctd\u003e1.94978\u003c/td\u003e\n",
              "      \u003ctd\u003e0.0\u003c/td\u003e\n",
              "      \u003ctd\u003eculmen_depth_mm\u003c/td\u003e\n",
              "      \u003ctd\u003e0.495780\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e0.814165\u003c/td\u003e\n",
              "      \u003ctd\u003e0.546813\u003c/td\u003e\n",
              "      \u003ctd\u003e1.94978\u003c/td\u003e\n",
              "      \u003ctd\u003e0.0\u003c/td\u003e\n",
              "      \u003ctd\u003ebody_mass_g\u003c/td\u003e\n",
              "      \u003ctd\u003e0.174635\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e0.814165\u003c/td\u003e\n",
              "      \u003ctd\u003e0.546813\u003c/td\u003e\n",
              "      \u003ctd\u003e1.94978\u003c/td\u003e\n",
              "      \u003ctd\u003e0.0\u003c/td\u003e\n",
              "      \u003ctd\u003eculmen_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003e0.013566\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e0.814165\u003c/td\u003e\n",
              "      \u003ctd\u003e0.546813\u003c/td\u003e\n",
              "      \u003ctd\u003e1.94978\u003c/td\u003e\n",
              "      \u003ctd\u003e0.0\u003c/td\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003e0.009056\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e5\u003c/th\u003e\n",
              "      \u003ctd\u003eGentoo penguin (Pygoscelis papua)\u003c/td\u003e\n",
              "      \u003ctd\u003e0.814165\u003c/td\u003e\n",
              "      \u003ctd\u003e0.546813\u003c/td\u003e\n",
              "      \u003ctd\u003e1.94978\u003c/td\u003e\n",
              "      \u003ctd\u003e0.0\u003c/td\u003e\n",
              "      \u003ctd\u003esex\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000558\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                   predicted_species  probability  baseline_prediction_value  \\\n",
              "0  Gentoo penguin (Pygoscelis papua)     0.814165                   0.546813   \n",
              "1  Gentoo penguin (Pygoscelis papua)     0.814165                   0.546813   \n",
              "2  Gentoo penguin (Pygoscelis papua)     0.814165                   0.546813   \n",
              "3  Gentoo penguin (Pygoscelis papua)     0.814165                   0.546813   \n",
              "4  Gentoo penguin (Pygoscelis papua)     0.814165                   0.546813   \n",
              "5  Gentoo penguin (Pygoscelis papua)     0.814165                   0.546813   \n",
              "\n",
              "   prediction_value  approximation_error            feature  attribution  \n",
              "0           1.94978                  0.0  flipper_length_mm     0.709371  \n",
              "1           1.94978                  0.0    culmen_depth_mm     0.495780  \n",
              "2           1.94978                  0.0        body_mass_g     0.174635  \n",
              "3           1.94978                  0.0   culmen_length_mm     0.013566  \n",
              "4           1.94978                  0.0             island     0.009056  \n",
              "5           1.94978                  0.0                sex     0.000558  "
            ]
          },
          "execution_count": 72,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "WITH\n",
        "    EXPLAIN AS (\n",
        "        SELECT * EXCEPT(body_mass_g, sex, island, flipper_length_mm, culmen_depth_mm, culmen_length_mm)\n",
        "        FROM ML.EXPLAIN_PREDICT(\n",
        "            MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "            (\n",
        "                SELECT * EXCEPT(instance_timestamp, species)\n",
        "                FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "                WHERE DATE(instance_timestamp) = CURRENT_DATE()\n",
        "                LIMIT 1\n",
        "            ), \n",
        "            STRUCT(10 AS top_k_features)\n",
        "        )\n",
        "    ),\n",
        "    ATTRIBUTIONS AS (\n",
        "        SELECT * EXCEPT(top_feature_attributions),\n",
        "        FROM EXPLAIN\n",
        "        JOIN UNNEST(top_feature_attributions)\n",
        "    )\n",
        "SELECT * \n",
        "FROM ATTRIBUTIONS"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dBbiLejrxLVM"
      },
      "source": [
        "#### Reviewing Feature Attributions\n",
        "\n",
        "Get attributions for all penguins:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 73,
      "metadata": {
        "tags": [],
        "id": "snFO5mowxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d1cd977dc3e745c0a28b33b17c769c90",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e83b6a923f4b43cdaa33daf9866347db",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "%%bigquery explain\n",
        "WITH\n",
        "    EXPLAIN AS (\n",
        "        SELECT * EXCEPT(body_mass_g, sex, island, flipper_length_mm, culmen_depth_mm, culmen_length_mm)\n",
        "        FROM ML.EXPLAIN_PREDICT(\n",
        "            MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "            (\n",
        "                SELECT * EXCEPT(instance_timestamp, species)\n",
        "                FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "                WHERE DATE(instance_timestamp) = CURRENT_DATE()\n",
        "            ), \n",
        "            STRUCT(10 AS top_k_features)\n",
        "        )\n",
        "    ),\n",
        "    ATTRIBUTIONS AS (\n",
        "        SELECT * EXCEPT(top_feature_attributions),\n",
        "        FROM EXPLAIN\n",
        "        JOIN UNNEST(top_feature_attributions)\n",
        "    ),\n",
        "    PIVOT_ATTRIBUTION AS (\n",
        "        SELECT body_mass_g, sex, island, flipper_length_mm, culmen_depth_mm, culmen_length_mm\n",
        "        FROM ATTRIBUTIONS PIVOT(MAX(attribution) FOR feature IN ('body_mass_g', 'sex', 'island', 'flipper_length_mm', 'culmen_depth_mm', 'culmen_length_mm'))\n",
        "    )\n",
        "SELECT *\n",
        "FROM PIVOT_ATTRIBUTION"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 74,
      "metadata": {
        "tags": [],
        "id": "_wHnt0xGxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003ebody_mass_g\u003c/th\u003e\n",
              "      \u003cth\u003esex\u003c/th\u003e\n",
              "      \u003cth\u003eisland\u003c/th\u003e\n",
              "      \u003cth\u003eflipper_length_mm\u003c/th\u003e\n",
              "      \u003cth\u003eculmen_depth_mm\u003c/th\u003e\n",
              "      \u003cth\u003eculmen_length_mm\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003e-0.000922\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.001108\u003c/td\u003e\n",
              "      \u003ctd\u003e0.171124\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.464971\u003c/td\u003e\n",
              "      \u003ctd\u003e0.176994\u003c/td\u003e\n",
              "      \u003ctd\u003e0.565229\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003e0.226313\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.000169\u003c/td\u003e\n",
              "      \u003ctd\u003e0.011402\u003c/td\u003e\n",
              "      \u003ctd\u003e0.710311\u003c/td\u003e\n",
              "      \u003ctd\u003e0.442352\u003c/td\u003e\n",
              "      \u003ctd\u003e0.013318\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003e0.386535\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.000169\u003c/td\u003e\n",
              "      \u003ctd\u003e0.026905\u003c/td\u003e\n",
              "      \u003ctd\u003e0.687082\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.471522\u003c/td\u003e\n",
              "      \u003ctd\u003e0.014066\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003e0.247754\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.000169\u003c/td\u003e\n",
              "      \u003ctd\u003e0.011402\u003c/td\u003e\n",
              "      \u003ctd\u003e0.710158\u003c/td\u003e\n",
              "      \u003ctd\u003e0.420504\u003c/td\u003e\n",
              "      \u003ctd\u003e0.013318\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003e0.247754\u003c/td\u003e\n",
              "      \u003ctd\u003e-0.000169\u003c/td\u003e\n",
              "      \u003ctd\u003e0.011402\u003c/td\u003e\n",
              "      \u003ctd\u003e0.710158\u003c/td\u003e\n",
              "      \u003ctd\u003e0.420504\u003c/td\u003e\n",
              "      \u003ctd\u003e0.013318\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "   body_mass_g       sex    island  flipper_length_mm  culmen_depth_mm  \\\n",
              "0    -0.000922 -0.001108  0.171124          -0.464971         0.176994   \n",
              "1     0.226313 -0.000169  0.011402           0.710311         0.442352   \n",
              "2     0.386535 -0.000169  0.026905           0.687082        -0.471522   \n",
              "3     0.247754 -0.000169  0.011402           0.710158         0.420504   \n",
              "4     0.247754 -0.000169  0.011402           0.710158         0.420504   \n",
              "\n",
              "   culmen_length_mm  \n",
              "0          0.565229  \n",
              "1          0.013318  \n",
              "2          0.014066  \n",
              "3          0.013318  \n",
              "4          0.013318  "
            ]
          },
          "execution_count": 74,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "explain.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wAl6WTLkxLVM"
      },
      "source": [
        "#### Monitor Feature Attributions For Drift with `ML.VALIDATE_DATA_DRIFT`\n",
        "\n",
        "Compare yesterday to 25 days ago:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 75,
      "metadata": {
        "tags": [],
        "id": "MOJ2-yDWxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "edc215b9e1f64abb8d8943af93a4d6c0",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "577a961ca0f144f3a8acc81d1fa0c603",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003einput\u003c/th\u003e\n",
              "      \u003cth\u003emetric\u003c/th\u003e\n",
              "      \u003cth\u003ethreshold\u003c/th\u003e\n",
              "      \u003cth\u003evalue\u003c/th\u003e\n",
              "      \u003cth\u003eis_anomaly\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003ebody_mass_g\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.029422\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_depth_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.051733\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.094459\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eflipper_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.138318\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.076407\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e5\u003c/th\u003e\n",
              "      \u003ctd\u003esex\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.605144\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "               input                     metric  threshold     value  \\\n",
              "0        body_mass_g  JENSEN_SHANNON_DIVERGENCE        0.3  0.029422   \n",
              "1    culmen_depth_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.051733   \n",
              "2   culmen_length_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.094459   \n",
              "3  flipper_length_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.138318   \n",
              "4             island  JENSEN_SHANNON_DIVERGENCE        0.3  0.076407   \n",
              "5                sex  JENSEN_SHANNON_DIVERGENCE        0.3  0.605144   \n",
              "\n",
              "   is_anomaly  \n",
              "0       False  \n",
              "1       False  \n",
              "2       False  \n",
              "3       False  \n",
              "4       False  \n",
              "5        True  "
            ]
          },
          "execution_count": 75,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "WITH\n",
        "    EXPLAIN_BASE AS (\n",
        "        SELECT * EXCEPT(body_mass_g, sex, island, flipper_length_mm, culmen_depth_mm, culmen_length_mm)\n",
        "        FROM ML.EXPLAIN_PREDICT(\n",
        "            MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "            (\n",
        "                SELECT * EXCEPT(instance_timestamp, species)\n",
        "                FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "                WHERE DATE(instance_timestamp) = DATE_SUB(CURRENT_DATE(), INTERVAL 25 DAY)\n",
        "            ), \n",
        "            STRUCT(10 AS top_k_features)\n",
        "        )\n",
        "    ),\n",
        "    ATTRIBUTIONS_BASE AS (\n",
        "        SELECT * EXCEPT(top_feature_attributions),\n",
        "        FROM EXPLAIN_BASE\n",
        "        JOIN UNNEST(top_feature_attributions)\n",
        "    ),\n",
        "    PIVOT_ATTRIBUTION_BASE AS (\n",
        "        SELECT body_mass_g, sex, island, flipper_length_mm, culmen_depth_mm, culmen_length_mm\n",
        "        FROM ATTRIBUTIONS_BASE PIVOT(MAX(attribution) FOR feature IN ('body_mass_g', 'sex', 'island', 'flipper_length_mm', 'culmen_depth_mm', 'culmen_length_mm'))\n",
        "    ),\n",
        "    EXPLAIN_COMP AS (\n",
        "        SELECT * EXCEPT(body_mass_g, sex, island, flipper_length_mm, culmen_depth_mm, culmen_length_mm)\n",
        "        FROM ML.EXPLAIN_PREDICT(\n",
        "            MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "            (\n",
        "                SELECT * EXCEPT(instance_timestamp, species)\n",
        "                FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "                WHERE DATE(instance_timestamp) = DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY)\n",
        "            ), \n",
        "            STRUCT(10 AS top_k_features)\n",
        "        )\n",
        "    ),\n",
        "    ATTRIBUTIONS_COMP AS (\n",
        "        SELECT * EXCEPT(top_feature_attributions),\n",
        "        FROM EXPLAIN_COMP\n",
        "        JOIN UNNEST(top_feature_attributions)\n",
        "    ),\n",
        "    PIVOT_ATTRIBUTION_COMP AS (\n",
        "        SELECT body_mass_g, sex, island, flipper_length_mm, culmen_depth_mm, culmen_length_mm\n",
        "        FROM ATTRIBUTIONS_COMP PIVOT(MAX(attribution) FOR feature IN ('body_mass_g', 'sex', 'island', 'flipper_length_mm', 'culmen_depth_mm', 'culmen_length_mm'))\n",
        "    )\n",
        "SELECT *\n",
        "FROM ML.VALIDATE_DATA_DRIFT(\n",
        "    TABLE PIVOT_ATTRIBUTION_BASE,\n",
        "    TABLE PIVOT_ATTRIBUTION_COMP\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vpXk4BkAxLVM"
      },
      "source": [
        "---\n",
        "## Model Monitoring For Online Inference With Vertex AI\n",
        "\n",
        "The model trained above was trained using BigQuery ML's [`CREATE MODEL` statement](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create) along with the training option `MODEL_REGISTRY = \"vertex_ai\"` which automatically registers the model in the [Vertex AI Model Registry](https://cloud.google.com/vertex-ai/docs/model-registry/introduction).  This means the BigQuery ML models can be managed directly in Vertex AI - [more information](https://cloud.google.com/bigquery/docs/managing-models-vertex).\n",
        "\n",
        "Models in the Vertex AI Model Registry can be used to [serve batch or online predictions](https://cloud.google.com/vertex-ai/docs/predictions/get-predictions).  In this section the [Vertex AI Python SDK](https://cloud.google.com/vertex-ai/docs/start/use-vertex-ai-python-sdk) is used to [deploy the model](https://cloud.google.com/vertex-ai/docs/general/deployment) created in this workflow and [get online predictions](https://cloud.google.com/vertex-ai/docs/predictions/get-online-predictions).\n",
        "\n",
        "The deployment of the model will include an extra setting to [setup logging of prediction request and responses to BigQuery](https://cloud.google.com/vertex-ai/docs/predictions/online-prediction-logging#enabling-and-disabling).  By logging the request and responses to BigQuery we can easily use the model monitoring functions covered in this tutorial."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OnAG772_xLVM"
      },
      "source": [
        "### Setup Client"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 76,
      "metadata": {
        "tags": [],
        "id": "zjbDJ28mxLVM"
      },
      "outputs": [],
      "source": [
        "from google.cloud import aiplatform\n",
        "aiplatform.init(project = 'statmike-mlops-349915', location = 'us-central1')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YtlAzuQsxLVM"
      },
      "source": [
        "### Get Model From Vertex AI Model Registry"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 77,
      "metadata": {
        "tags": [],
        "id": "56pABTbfxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'projects/1026793852137/locations/us-central1/models/classify_species_rf@53'"
            ]
          },
          "execution_count": 77,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model = aiplatform.Model(model_name = 'classify_species_rf')\n",
        "model.versioned_resource_name"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "grXdxo86xLVM"
      },
      "source": [
        "### Deploy Model To Endpoint"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 78,
      "metadata": {
        "tags": [],
        "id": "iubcT9AbxLVM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Creating Endpoint\n",
            "Create Endpoint backing LRO: projects/1026793852137/locations/us-central1/endpoints/8376174138397556736/operations/1114944957099016192\n",
            "Endpoint created. Resource name: projects/1026793852137/locations/us-central1/endpoints/8376174138397556736\n",
            "To use this Endpoint in another session:\n",
            "endpoint = aiplatform.Endpoint('projects/1026793852137/locations/us-central1/endpoints/8376174138397556736')\n",
            "Deploying model to Endpoint : projects/1026793852137/locations/us-central1/endpoints/8376174138397556736\n",
            "Using default machine_type: n1-standard-2\n",
            "Deploy Endpoint model backing LRO: projects/1026793852137/locations/us-central1/endpoints/8376174138397556736/operations/6925591231011487744\n",
            "Endpoint model deployed. Resource name: projects/1026793852137/locations/us-central1/endpoints/8376174138397556736\n"
          ]
        }
      ],
      "source": [
        "endpoint = model.deploy()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 79,
      "metadata": {
        "tags": [],
        "id": "akZYq1M0xLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'projects/1026793852137/locations/us-central1/endpoints/8376174138397556736'"
            ]
          },
          "execution_count": 79,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "endpoint.resource_name"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SQrLNu-GxLVM"
      },
      "source": [
        "### Get Data For Predictions\n",
        "\n",
        "Get a sample of data from the serving table to use as online prediction requests."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 80,
      "metadata": {
        "tags": [],
        "id": "AHITZBEHxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ce1be3b69b0948c594009429ee0ad9b7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c4810be7049a4d64b2c3ba5921f773aa",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "%%bigquery online\n",
        "SELECT * EXCEPT(species, instance_timestamp)\n",
        "FROM statmike-mlops-349915.bqml_model_monitoring.serving\n",
        "WHERE RAND() \u003c 0.1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 81,
      "metadata": {
        "tags": [],
        "id": "mcXAq3BbxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'island': 'Biscoe',\n",
              " 'culmen_length_mm': 47.8,\n",
              " 'culmen_depth_mm': 15.0,\n",
              " 'sex': 'MALE',\n",
              " 'flipper_length_mm': 264.9457340434056,\n",
              " 'body_mass_g': 6746.1164813378955}"
            ]
          },
          "execution_count": 81,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import numpy as np\n",
        "online = online.dropna()\n",
        "online['sex'].replace(np.nan, '', inplace=True)\n",
        "online = online.to_dict(orient = 'records')\n",
        "online[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "tags": [],
        "id": "wyovuazoxLVM"
      },
      "source": [
        "### Get Predictions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 82,
      "metadata": {
        "tags": [],
        "id": "tl23FLI6xLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Prediction(predictions=[{'predicted_species': 'Gentoo penguin (Pygoscelis papua)', 'species_probs': [0.8151684403419495, 0.09227550774812698, 0.09255607426166534], 'species_values': ['Gentoo penguin (Pygoscelis papua)', 'Chinstrap penguin (Pygoscelis antarctica)', 'Adelie Penguin (Pygoscelis adeliae)']}], deployed_model_id='679865422849245184', metadata=None, model_version_id='53', model_resource_name='projects/1026793852137/locations/us-central1/models/classify_species_rf', explanations=None)"
            ]
          },
          "execution_count": 82,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "endpoint.predict(instances = online[0:1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fDjVFYD8xLVM"
      },
      "source": [
        "### Enable Prediction request-response logging To BigQuery\n",
        "\n",
        "The endpoint can be enabled for logging both the request and response for predictions.  These logs are directed to a BigQuery table.  This is enable throught the `gcloud ai endpoints update` command - [reference](https://cloud.google.com/sdk/gcloud/reference/ai/endpoints/update)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 83,
      "metadata": {
        "tags": [],
        "id": "NSivrnxexLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'8376174138397556736'"
            ]
          },
          "execution_count": 83,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "endpoint.name"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 84,
      "metadata": {
        "tags": [],
        "id": "sbG1ga0SxLVM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Using endpoint [https://us-central1-aiplatform.googleapis.com/]\n",
            "Updated Vertex AI endpoint [projects/1026793852137/locations/us-central1/endpoints/8376174138397556736].\n"
          ]
        }
      ],
      "source": [
        "!gcloud ai endpoints update $endpoint.name \\\n",
        "--region=us-central1 \\\n",
        "--request-response-logging-table=bq://statmike-mlops-349915.bqml_model_monitoring.serving_online"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9SfVeN7JxLVM"
      },
      "source": [
        "### Send Prediction Requests To Endpoint\n",
        "\n",
        "These will get logged to the BigQuery table specified above."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 85,
      "metadata": {
        "tags": [],
        "id": "fy8lcGYgxLVM"
      },
      "outputs": [],
      "source": [
        "for instance in online:\n",
        "    prediction = endpoint.predict(instances = [instance])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 86,
      "metadata": {
        "tags": [],
        "id": "uBMVLtYNxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a6782cc7ee5e45bca64f0435258be86b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "34cdbc67415446adb19fdea496cfc0f5",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003eendpoint\u003c/th\u003e\n",
              "      \u003cth\u003edeployed_model_id\u003c/th\u003e\n",
              "      \u003cth\u003elogging_time\u003c/th\u003e\n",
              "      \u003cth\u003erequest_id\u003c/th\u003e\n",
              "      \u003cth\u003erequest_payload\u003c/th\u003e\n",
              "      \u003cth\u003eresponse_payload\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003eprojects/1026793852137/locations/us-central1/endpoints/8376174138397556736\u003c/td\u003e\n",
              "      \u003ctd\u003e679865422849245184\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-18 12:18:11.197298+00:00\u003c/td\u003e\n",
              "      \u003ctd\u003e3672078733502561545.000000000\u003c/td\u003e\n",
              "      \u003ctd\u003e[{\"sex\":\"FEMALE\",\"culmen_length_mm\":35,\"culmen_depth_mm\":17.9,\"flipper_length_mm\":194.51871831435113,\"body_mass_g\":3800.4606022451417,\"island\":\"Biscoe\"}]\u003c/td\u003e\n",
              "      \u003ctd\u003e[{\"species_probs\":[0.096641786396503448,0.096392765641212463,0.80696547031402588],\"species_values\":[\"Gentoo penguin (Pygoscelis papua)\",\"Chinstrap penguin (Pygoscelis antarctica)\",\"Adelie Penguin (Pygoscelis adeliae)\"],\"predicted_species\":\"Adelie Penguin (Pygoscelis adeliae)\"}]\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eprojects/1026793852137/locations/us-central1/endpoints/8376174138397556736\u003c/td\u003e\n",
              "      \u003ctd\u003e679865422849245184\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-18 12:18:11.748816+00:00\u003c/td\u003e\n",
              "      \u003ctd\u003e4897722793025007221.000000000\u003c/td\u003e\n",
              "      \u003ctd\u003e[{\"flipper_length_mm\":228.06072187777977,\"island\":\"Biscoe\",\"body_mass_g\":4996.1526538602338,\"culmen_length_mm\":41.3,\"culmen_depth_mm\":21.1,\"sex\":\"MALE\"}]\u003c/td\u003e\n",
              "      \u003ctd\u003e[{\"species_values\":[\"Gentoo penguin (Pygoscelis papua)\",\"Chinstrap penguin (Pygoscelis antarctica)\",\"Adelie Penguin (Pygoscelis adeliae)\"],\"predicted_species\":\"Adelie Penguin (Pygoscelis adeliae)\",\"species_probs\":[0.36388087272644037,0.12630975246429441,0.50980937480926514]}]\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eprojects/1026793852137/locations/us-central1/endpoints/8376174138397556736\u003c/td\u003e\n",
              "      \u003ctd\u003e679865422849245184\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-18 12:18:11.793376+00:00\u003c/td\u003e\n",
              "      \u003ctd\u003e2532566483074682501.000000000\u003c/td\u003e\n",
              "      \u003ctd\u003e[{\"body_mass_g\":5226.7781687599345,\"sex\":\"FEMALE\",\"culmen_length_mm\":40.9,\"flipper_length_mm\":246.77387673794192,\"island\":\"Biscoe\",\"culmen_depth_mm\":13.7}]\u003c/td\u003e\n",
              "      \u003ctd\u003e[{\"species_values\":[\"Gentoo penguin (Pygoscelis papua)\",\"Chinstrap penguin (Pygoscelis antarctica)\",\"Adelie Penguin (Pygoscelis adeliae)\"],\"predicted_species\":\"Gentoo penguin (Pygoscelis papua)\",\"species_probs\":[0.74543070793151855,0.0999060869216919,0.15466322004795069]}]\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eprojects/1026793852137/locations/us-central1/endpoints/8376174138397556736\u003c/td\u003e\n",
              "      \u003ctd\u003e679865422849245184\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-18 12:18:11.929502+00:00\u003c/td\u003e\n",
              "      \u003ctd\u003e3023598592391758369.000000000\u003c/td\u003e\n",
              "      \u003ctd\u003e[{\"body_mass_g\":3536.15893149333,\"sex\":\"FEMALE\",\"island\":\"Dream\",\"culmen_length_mm\":46.1,\"culmen_depth_mm\":18.2,\"flipper_length_mm\":195.83599345931358}]\u003c/td\u003e\n",
              "      \u003ctd\u003e[{\"species_probs\":[0.10982932150363919,0.76431870460510254,0.12585197389125821],\"species_values\":[\"Gentoo penguin (Pygoscelis papua)\",\"Chinstrap penguin (Pygoscelis antarctica)\",\"Adelie Penguin (Pygoscelis adeliae)\"],\"predicted_species\":\"Chinstrap penguin (Pygoscelis antarctica)\"}]\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eprojects/1026793852137/locations/us-central1/endpoints/8376174138397556736\u003c/td\u003e\n",
              "      \u003ctd\u003e679865422849245184\u003c/td\u003e\n",
              "      \u003ctd\u003e2024-04-18 12:18:11.991745+00:00\u003c/td\u003e\n",
              "      \u003ctd\u003e8767896355494798527.000000000\u003c/td\u003e\n",
              "      \u003ctd\u003e[{\"sex\":\"MALE\",\"culmen_depth_mm\":18.1,\"body_mass_g\":3947.9486959646842,\"culmen_length_mm\":37.8,\"flipper_length_mm\":200.89548253625452,\"island\":\"Dream\"}]\u003c/td\u003e\n",
              "      \u003ctd\u003e[{\"species_values\":[\"Gentoo penguin (Pygoscelis papua)\",\"Chinstrap penguin (Pygoscelis antarctica)\",\"Adelie Penguin (Pygoscelis adeliae)\"],\"predicted_species\":\"Adelie Penguin (Pygoscelis adeliae)\",\"species_probs\":[0.10563097149133679,0.1376055181026459,0.7567635178565979]}]\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                                                                     endpoint  \\\n",
              "0  projects/1026793852137/locations/us-central1/endpoints/8376174138397556736   \n",
              "1  projects/1026793852137/locations/us-central1/endpoints/8376174138397556736   \n",
              "2  projects/1026793852137/locations/us-central1/endpoints/8376174138397556736   \n",
              "3  projects/1026793852137/locations/us-central1/endpoints/8376174138397556736   \n",
              "4  projects/1026793852137/locations/us-central1/endpoints/8376174138397556736   \n",
              "\n",
              "    deployed_model_id                     logging_time  \\\n",
              "0  679865422849245184 2024-04-18 12:18:11.197298+00:00   \n",
              "1  679865422849245184 2024-04-18 12:18:11.748816+00:00   \n",
              "2  679865422849245184 2024-04-18 12:18:11.793376+00:00   \n",
              "3  679865422849245184 2024-04-18 12:18:11.929502+00:00   \n",
              "4  679865422849245184 2024-04-18 12:18:11.991745+00:00   \n",
              "\n",
              "                      request_id  \\\n",
              "0  3672078733502561545.000000000   \n",
              "1  4897722793025007221.000000000   \n",
              "2  2532566483074682501.000000000   \n",
              "3  3023598592391758369.000000000   \n",
              "4  8767896355494798527.000000000   \n",
              "\n",
              "                                                                                                                                               request_payload  \\\n",
              "0    [{\"sex\":\"FEMALE\",\"culmen_length_mm\":35,\"culmen_depth_mm\":17.9,\"flipper_length_mm\":194.51871831435113,\"body_mass_g\":3800.4606022451417,\"island\":\"Biscoe\"}]   \n",
              "1    [{\"flipper_length_mm\":228.06072187777977,\"island\":\"Biscoe\",\"body_mass_g\":4996.1526538602338,\"culmen_length_mm\":41.3,\"culmen_depth_mm\":21.1,\"sex\":\"MALE\"}]   \n",
              "2  [{\"body_mass_g\":5226.7781687599345,\"sex\":\"FEMALE\",\"culmen_length_mm\":40.9,\"flipper_length_mm\":246.77387673794192,\"island\":\"Biscoe\",\"culmen_depth_mm\":13.7}]   \n",
              "3     [{\"body_mass_g\":3536.15893149333,\"sex\":\"FEMALE\",\"island\":\"Dream\",\"culmen_length_mm\":46.1,\"culmen_depth_mm\":18.2,\"flipper_length_mm\":195.83599345931358}]   \n",
              "4     [{\"sex\":\"MALE\",\"culmen_depth_mm\":18.1,\"body_mass_g\":3947.9486959646842,\"culmen_length_mm\":37.8,\"flipper_length_mm\":200.89548253625452,\"island\":\"Dream\"}]   \n",
              "\n",
              "                                                                                                                                                                                                                                                                             response_payload  \n",
              "0      [{\"species_probs\":[0.096641786396503448,0.096392765641212463,0.80696547031402588],\"species_values\":[\"Gentoo penguin (Pygoscelis papua)\",\"Chinstrap penguin (Pygoscelis antarctica)\",\"Adelie Penguin (Pygoscelis adeliae)\"],\"predicted_species\":\"Adelie Penguin (Pygoscelis adeliae)\"}]  \n",
              "1        [{\"species_values\":[\"Gentoo penguin (Pygoscelis papua)\",\"Chinstrap penguin (Pygoscelis antarctica)\",\"Adelie Penguin (Pygoscelis adeliae)\"],\"predicted_species\":\"Adelie Penguin (Pygoscelis adeliae)\",\"species_probs\":[0.36388087272644037,0.12630975246429441,0.50980937480926514]}]  \n",
              "2           [{\"species_values\":[\"Gentoo penguin (Pygoscelis papua)\",\"Chinstrap penguin (Pygoscelis antarctica)\",\"Adelie Penguin (Pygoscelis adeliae)\"],\"predicted_species\":\"Gentoo penguin (Pygoscelis papua)\",\"species_probs\":[0.74543070793151855,0.0999060869216919,0.15466322004795069]}]  \n",
              "3  [{\"species_probs\":[0.10982932150363919,0.76431870460510254,0.12585197389125821],\"species_values\":[\"Gentoo penguin (Pygoscelis papua)\",\"Chinstrap penguin (Pygoscelis antarctica)\",\"Adelie Penguin (Pygoscelis adeliae)\"],\"predicted_species\":\"Chinstrap penguin (Pygoscelis antarctica)\"}]  \n",
              "4          [{\"species_values\":[\"Gentoo penguin (Pygoscelis papua)\",\"Chinstrap penguin (Pygoscelis antarctica)\",\"Adelie Penguin (Pygoscelis adeliae)\"],\"predicted_species\":\"Adelie Penguin (Pygoscelis adeliae)\",\"species_probs\":[0.10563097149133679,0.1376055181026459,0.7567635178565979]}]  "
            ]
          },
          "execution_count": 86,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM `statmike-mlops-349915.bqml_model_monitoring.serving_online`\n",
        "LIMIT 5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "75lsUH6txLVM"
      },
      "source": [
        "### Create A View Of All Serving Data\n",
        "\n",
        "Combine the batch serving data with the online serving data in a view."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 87,
      "metadata": {
        "tags": [],
        "id": "I5P_qiq7xLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "612f6e9a0b6a4d98aeea71035896a51f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5a84b93ea76f463184adebca2cc3a5fe",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003einstance_timestamp\u003c/th\u003e\n",
              "      \u003cth\u003eisland\u003c/th\u003e\n",
              "      \u003cth\u003esex\u003c/th\u003e\n",
              "      \u003cth\u003eflipper_length_mm\u003c/th\u003e\n",
              "      \u003cth\u003ebody_mass_g\u003c/th\u003e\n",
              "      \u003cth\u003eculmen_length_mm\u003c/th\u003e\n",
              "      \u003cth\u003eculmen_depth_mm\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003e2024-04-18 12:18:11.438940+00:00\u003c/td\u003e\n",
              "      \u003ctd\u003eDream\u003c/td\u003e\n",
              "      \u003ctd\u003eMALE\u003c/td\u003e\n",
              "      \u003ctd\u003e256.845386\u003c/td\u003e\n",
              "      \u003ctd\u003e5518.321747\u003c/td\u003e\n",
              "      \u003ctd\u003e53.5\u003c/td\u003e\n",
              "      \u003ctd\u003e19.9\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003e2024-04-18 12:18:11.613571+00:00\u003c/td\u003e\n",
              "      \u003ctd\u003eDream\u003c/td\u003e\n",
              "      \u003ctd\u003eMALE\u003c/td\u003e\n",
              "      \u003ctd\u003e263.914438\u003c/td\u003e\n",
              "      \u003ctd\u003e5013.543963\u003c/td\u003e\n",
              "      \u003ctd\u003e49.0\u003c/td\u003e\n",
              "      \u003ctd\u003e19.6\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003e2024-04-18 12:18:11.707137+00:00\u003c/td\u003e\n",
              "      \u003ctd\u003eTorgersen\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "      \u003ctd\u003e228.776258\u003c/td\u003e\n",
              "      \u003ctd\u003e4766.420181\u003c/td\u003e\n",
              "      \u003ctd\u003e36.7\u003c/td\u003e\n",
              "      \u003ctd\u003e18.8\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003e2024-04-18 12:18:11.908256+00:00\u003c/td\u003e\n",
              "      \u003ctd\u003eDream\u003c/td\u003e\n",
              "      \u003ctd\u003eFEMALE\u003c/td\u003e\n",
              "      \u003ctd\u003e212.195724\u003c/td\u003e\n",
              "      \u003ctd\u003e3795.663191\u003c/td\u003e\n",
              "      \u003ctd\u003e36.8\u003c/td\u003e\n",
              "      \u003ctd\u003e18.5\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003e2024-04-18 12:18:12.143982+00:00\u003c/td\u003e\n",
              "      \u003ctd\u003eBiscoe\u003c/td\u003e\n",
              "      \u003ctd\u003eMALE\u003c/td\u003e\n",
              "      \u003ctd\u003e232.357711\u003c/td\u003e\n",
              "      \u003ctd\u003e5583.769020\u003c/td\u003e\n",
              "      \u003ctd\u003e43.2\u003c/td\u003e\n",
              "      \u003ctd\u003e19.0\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "                instance_timestamp     island     sex  flipper_length_mm  \\\n",
              "0 2024-04-18 12:18:11.438940+00:00      Dream    MALE         256.845386   \n",
              "1 2024-04-18 12:18:11.613571+00:00      Dream    MALE         263.914438   \n",
              "2 2024-04-18 12:18:11.707137+00:00  Torgersen  FEMALE         228.776258   \n",
              "3 2024-04-18 12:18:11.908256+00:00      Dream  FEMALE         212.195724   \n",
              "4 2024-04-18 12:18:12.143982+00:00     Biscoe    MALE         232.357711   \n",
              "\n",
              "   body_mass_g  culmen_length_mm  culmen_depth_mm  \n",
              "0  5518.321747              53.5             19.9  \n",
              "1  5013.543963              49.0             19.6  \n",
              "2  4766.420181              36.7             18.8  \n",
              "3  3795.663191              36.8             18.5  \n",
              "4  5583.769020              43.2             19.0  "
            ]
          },
          "execution_count": 87,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT\n",
        "    logging_time as instance_timestamp,\n",
        "    JSON_VALUE(request, \"$.island\") as island,\n",
        "    JSON_VALUE(request, \"$.sex\") as sex,\n",
        "    CAST(JSON_VALUE(request, \"$.flipper_length_mm\") AS FLOAT64) as flipper_length_mm,\n",
        "    CAST(JSON_VALUE(request, \"$.body_mass_g\") AS FLOAT64) as body_mass_g,\n",
        "    CAST(JSON_VALUE(request, \"$.culmen_length_mm\") AS FLOAT64) as culmen_length_mm,\n",
        "    CAST(JSON_VALUE(request, \"$.culmen_depth_mm\") AS FLOAT64) as culmen_depth_mm\n",
        "FROM `statmike-mlops-349915.bqml_model_monitoring.serving_online`,\n",
        "UNNEST(request_payload) as request\n",
        "LIMIT 5"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 88,
      "metadata": {
        "tags": [],
        "id": "p7WUp_cjxLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ab26f2c638a742748ebf3b7f2ca9b758",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: []\n",
              "Index: []"
            ]
          },
          "execution_count": 88,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "CREATE OR REPLACE VIEW `statmike-mlops-349915.bqml_model_monitoring.serving_all` AS\n",
        "WITH\n",
        "    BATCH AS (\n",
        "        SELECT instance_timestamp, island, sex, flipper_length_mm, body_mass_g, culmen_length_mm, culmen_depth_mm\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "    ),\n",
        "    ONLINE AS (\n",
        "        SELECT\n",
        "            logging_time as instance_timestamp,\n",
        "            JSON_VALUE(request, \"$.island\") as island,\n",
        "            JSON_VALUE(request, \"$.sex\") as sex,\n",
        "            CAST(JSON_VALUE(request, \"$.flipper_length_mm\") AS FLOAT64) as flipper_length_mm,\n",
        "            CAST(JSON_VALUE(request, \"$.body_mass_g\") AS FLOAT64) as body_mass_g,\n",
        "            CAST(JSON_VALUE(request, \"$.culmen_length_mm\") AS FLOAT64) as culmen_length_mm,\n",
        "            CAST(JSON_VALUE(request, \"$.culmen_depth_mm\") AS FLOAT64) as culmen_depth_mm\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving_online`,\n",
        "        UNNEST(request_payload) as request\n",
        "    )\n",
        "SELECT * FROM BATCH\n",
        "UNION ALL\n",
        "SELECT * FROM ONLINE\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oA7ISP2hxLVM"
      },
      "source": [
        "### Monitoring Skew (Change from training) With `ML.VALIDATE_DATA_SKEW`\n",
        "\n",
        "Compare today, or any data, to the training data by directly comparing to the the model created in BigQuery."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 89,
      "metadata": {
        "tags": [],
        "id": "F2CVwZ48xLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7f4038ad195548f681284d66b420e17c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3010faf8745942bab174ef0404a6d7f2",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003einput\u003c/th\u003e\n",
              "      \u003cth\u003emetric\u003c/th\u003e\n",
              "      \u003cth\u003ethreshold\u003c/th\u003e\n",
              "      \u003cth\u003evalue\u003c/th\u003e\n",
              "      \u003cth\u003eis_anomaly\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003ebody_mass_g\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.137045\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_depth_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.029471\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.058987\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eflipper_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.494604\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.063670\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e5\u003c/th\u003e\n",
              "      \u003ctd\u003esex\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.3\u003c/td\u003e\n",
              "      \u003ctd\u003e0.077660\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "               input                     metric  threshold     value  \\\n",
              "0        body_mass_g  JENSEN_SHANNON_DIVERGENCE        0.3  0.137045   \n",
              "1    culmen_depth_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.029471   \n",
              "2   culmen_length_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.058987   \n",
              "3  flipper_length_mm  JENSEN_SHANNON_DIVERGENCE        0.3  0.494604   \n",
              "4             island                    L_INFTY        0.3  0.063670   \n",
              "5                sex                    L_INFTY        0.3  0.077660   \n",
              "\n",
              "   is_anomaly  \n",
              "0       False  \n",
              "1       False  \n",
              "2       False  \n",
              "3        True  \n",
              "4       False  \n",
              "5       False  "
            ]
          },
          "execution_count": 89,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.VALIDATE_DATA_SKEW(\n",
        "    MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "    (\n",
        "        SELECT *\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving_all`\n",
        "        WHERE DATE(instance_timestamp) \u003e= CURRENT_DATE()\n",
        "    )\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5H0aYsJFxLVM"
      },
      "source": [
        "### Monitoring Drift (Change Over Time) With `ML.VALIDATE_DATA_DRIFT`\n",
        "\n",
        "Compare today to 5 days ago and set the a lower, more sensitive, threshold of `0.03` (default is `0.3`) for all features."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 90,
      "metadata": {
        "tags": [],
        "id": "h079kOX5xLVM"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3cf8273087054bd287b22eb8e642a4bc",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Query is running:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "773617c07a724f7eada756c867fed7cc",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          |"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\u003cdiv\u003e\n",
              "\u003cstyle scoped\u003e\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "\u003c/style\u003e\n",
              "\u003ctable border=\"1\" class=\"dataframe\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr style=\"text-align: right;\"\u003e\n",
              "      \u003cth\u003e\u003c/th\u003e\n",
              "      \u003cth\u003einput\u003c/th\u003e\n",
              "      \u003cth\u003emetric\u003c/th\u003e\n",
              "      \u003cth\u003ethreshold\u003c/th\u003e\n",
              "      \u003cth\u003evalue\u003c/th\u003e\n",
              "      \u003cth\u003eis_anomaly\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e0\u003c/th\u003e\n",
              "      \u003ctd\u003ebody_mass_g\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.072752\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e1\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_depth_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.002961\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e2\u003c/th\u003e\n",
              "      \u003ctd\u003eculmen_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.000824\u003c/td\u003e\n",
              "      \u003ctd\u003eFalse\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e3\u003c/th\u003e\n",
              "      \u003ctd\u003eflipper_length_mm\u003c/td\u003e\n",
              "      \u003ctd\u003eJENSEN_SHANNON_DIVERGENCE\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.158921\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e4\u003c/th\u003e\n",
              "      \u003ctd\u003eisland\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.042130\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth\u003e5\u003c/th\u003e\n",
              "      \u003ctd\u003esex\u003c/td\u003e\n",
              "      \u003ctd\u003eL_INFTY\u003c/td\u003e\n",
              "      \u003ctd\u003e0.03\u003c/td\u003e\n",
              "      \u003ctd\u003e0.034852\u003c/td\u003e\n",
              "      \u003ctd\u003eTrue\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n",
              "\u003c/div\u003e"
            ],
            "text/plain": [
              "               input                     metric  threshold     value  \\\n",
              "0        body_mass_g  JENSEN_SHANNON_DIVERGENCE       0.03  0.072752   \n",
              "1    culmen_depth_mm  JENSEN_SHANNON_DIVERGENCE       0.03  0.002961   \n",
              "2   culmen_length_mm  JENSEN_SHANNON_DIVERGENCE       0.03  0.000824   \n",
              "3  flipper_length_mm  JENSEN_SHANNON_DIVERGENCE       0.03  0.158921   \n",
              "4             island                    L_INFTY       0.03  0.042130   \n",
              "5                sex                    L_INFTY       0.03  0.034852   \n",
              "\n",
              "   is_anomaly  \n",
              "0        True  \n",
              "1       False  \n",
              "2       False  \n",
              "3        True  \n",
              "4        True  \n",
              "5        True  "
            ]
          },
          "execution_count": 90,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%bigquery\n",
        "SELECT *\n",
        "FROM ML.VALIDATE_DATA_DRIFT(\n",
        "    (\n",
        "        SELECT * EXCEPT(instance_timestamp)\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving_all`\n",
        "        WHERE DATE(instance_timestamp) = DATE_SUB(CURRENT_DATE(), INTERVAL 5 DAY)\n",
        "    ),\n",
        "    (\n",
        "        SELECT * EXCEPT(instance_timestamp)\n",
        "        FROM `statmike-mlops-349915.bqml_model_monitoring.serving_all`\n",
        "        WHERE DATE(instance_timestamp) = CURRENT_DATE() \n",
        "    ),\n",
        "    STRUCT(\n",
        "        0.03 AS categorical_default_threshold,\n",
        "        0.03 AS numerical_default_threshold\n",
        "    )\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VYXdRUF9xLVM"
      },
      "source": [
        "### Remove Endpoint \n",
        "To prevent ongoing cost for this demonstration, remove the endpoint.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 91,
      "metadata": {
        "tags": [],
        "id": "YGdDJia9xLVM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Undeploying Endpoint model: projects/1026793852137/locations/us-central1/endpoints/8376174138397556736\n",
            "Undeploy Endpoint model backing LRO: projects/1026793852137/locations/us-central1/endpoints/8376174138397556736/operations/3796627428965613568\n",
            "Endpoint model undeployed. Resource name: projects/1026793852137/locations/us-central1/endpoints/8376174138397556736\n",
            "Deleting Endpoint : projects/1026793852137/locations/us-central1/endpoints/8376174138397556736\n",
            "Delete Endpoint  backing LRO: projects/1026793852137/locations/us-central1/operations/2956072779763417088\n",
            "Endpoint deleted. . Resource name: projects/1026793852137/locations/us-central1/endpoints/8376174138397556736\n"
          ]
        }
      ],
      "source": [
        "endpoint.delete(force = True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KkJVLLfJxLVM"
      },
      "source": [
        "---\n",
        "## Continous Monitoring\n",
        "\n",
        "The methods above can be combined into a job that runs both skew and drift detection.  The jobs can be scheduled or triggered to make the process of model monitoring continous - continous monitoring.  If the model quality can be evaluated with confidence then the automation can even be extended to retrain and update the model for continous training.   \n",
        "\n",
        "This section shows one of many ways to create a model monitoring job. For models created in BigQuery with BigQuery ML it also shows a retraining action as part of the job when alerts are detected.\n",
        "\n",
        "These types of jobs could be automated for continous monitoring with one of these methods:\n",
        "- [BigQuery Scheduled Queries](https://cloud.google.com/bigquery/docs/scheduling-queries)\n",
        "    - Featured below\n",
        "- [Dataform](https://cloud.google.com/dataform/docs/quickstart-create-workflow)\n",
        "- [Workflows](https://cloud.google.com/workflows/docs/tutorials/bigquery-parallel-jobs)\n",
        "- [Cloud Composer](https://cloud.google.com/composer/docs/data-analytics-googlecloud)\n",
        "- [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/gcpc-list#bqml_components)\n",
        "\n",
        "This section will show how to:\n",
        "- Develop model monitoring jobs:\n",
        "    - with alterting and retraining\n",
        "- Build tables to capture continous monitoring results\n",
        "- Extend monitoring jobs to BigQuery Scheduled Queries\n",
        "- Backfill monitoring jobs for a past time period"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gfxVoF1fxLVM"
      },
      "source": [
        "### Model Monitoring Job: With Alerting And Retraining\n",
        "\n",
        "A BigQuery SQL job that uses the [procedural language](https://cloud.google.com/bigquery/docs/reference/standard-sql/procedural-language) to execute multiple statements (steps) as a single query.  In this case, monitoring skew and drift, detecting any feature level anomalies in either and reporting back the anomalies via a log note with a forced error.\n",
        "\n",
        "This job also includes retraining of the BigQuery ML model:\n",
        "In this case the example is simplified to retrain the model and directly implement it as a replacement with `CREATE OR REPLACE MODEL`.  In a production enviornment additional pre/post steps might take place as part of this workflow:\n",
        "- on skew/drift detection:\n",
        "    - Prior to retraining:\n",
        "        - ensure detection is not due to data errors, schema changes, or system changes\n",
        "    - After retrainning:\n",
        "        - evaluate the model and compare to current production version\n",
        "        - conditionally deploy the new version if it is better in any/all key metrics\n",
        "        - consider rollout strategies for the retrained model\n",
        "        - trigger human review before deployment of the replacement"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 92,
      "metadata": {
        "tags": [],
        "id": "M5b6femExLVM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Executing query with job ID: 4aed911b-fc79-430f-8a59-e70f0f55f464\n",
            "Query executing: 0.26s"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "ERROR:\n",
            " 400 Query error: \n",
            "\n",
            "Monitoring Report:\n",
            "\tNo drift detected.\n",
            "\tNo skew detected.\n",
            "\n",
            " at [142:9]\n",
            "\n",
            "Location: US\n",
            "Job ID: 4aed911b-fc79-430f-8a59-e70f0f55f464\n",
            "\n"
          ]
        }
      ],
      "source": [
        "%%bigquery\n",
        "DECLARE drift_anomalies ARRAY\u003cSTRING\u003e;\n",
        "DECLARE drift STRING;\n",
        "DECLARE skew_anomalies ARRAY\u003cSTRING\u003e;\n",
        "DECLARE skew STRING;\n",
        "\n",
        "# Monitor Drift:\n",
        "SET drift_anomalies = (\n",
        "    SELECT ARRAY_AGG(input)\n",
        "    FROM ML.VALIDATE_DATA_DRIFT(\n",
        "        (\n",
        "            SELECT * EXCEPT(species, instance_timestamp)\n",
        "            FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "            WHERE DATE(instance_timestamp) = DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)\n",
        "        ),\n",
        "        (\n",
        "            SELECT * EXCEPT(species, instance_timestamp)\n",
        "            FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "            WHERE DATE(instance_timestamp) = DATE_SUB(CURRENT_DATE(), INTERVAL 29 DAY)\n",
        "        ),\n",
        "        STRUCT(\n",
        "            0.4 AS categorical_default_threshold,\n",
        "            0.4 AS numerical_default_threshold\n",
        "        )\n",
        "    )\n",
        "    WHERE is_anomaly = True\n",
        ");\n",
        "IF(ARRAY_LENGTH(drift_anomalies) \u003e 0) THEN\n",
        "    SET drift = CONCAT(\n",
        "        \"Found data drift (\", ARRAY_TO_STRING(drift_anomalies, \", \"), \")\"\n",
        "    );\n",
        "    ELSE SET drift = 'No drift detected.';\n",
        "END IF;\n",
        "\n",
        "# Monitor Skew\n",
        "SET skew_anomalies = (\n",
        "    SELECT ARRAY_AGG(input)\n",
        "    FROM ML.VALIDATE_DATA_SKEW(\n",
        "        MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "        (\n",
        "            SELECT * EXCEPT(species, instance_timestamp)\n",
        "            FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "            WHERE DATE(instance_timestamp) = DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)\n",
        "        )\n",
        "    )\n",
        "    WHERE is_anomaly = True\n",
        ");\n",
        "IF(ARRAY_LENGTH(skew_anomalies) \u003e 0) THEN\n",
        "    SET skew = CONCAT(\n",
        "        \"Found data skew (\", ARRAY_TO_STRING(skew_anomalies, \", \"), \")\"\n",
        "    );\n",
        "    ELSE SET skew = 'No skew detected.';\n",
        "END IF;\n",
        "\n",
        "# Prepare Alert\n",
        "IF(ARRAY_LENGTH(drift_anomalies) \u003e 0 OR ARRAY_LENGTH(skew_anomalies) \u003e 0) THEN\n",
        "    BEGIN\n",
        "        DECLARE train_accuracy FLOAT64;\n",
        "        DECLARE recent_accuracy FLOAT64;\n",
        "        DECLARE retrain_accuracy FLOAT64;\n",
        "\n",
        "        # get current models evaluation metrics: accuracy\n",
        "        SET train_accuracy = (\n",
        "            SELECT accuracy\n",
        "            FROM ML.EVALUATE (\n",
        "                MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`\n",
        "            )\n",
        "        );\n",
        "\n",
        "        # get current models recent evaluation metrics: accuracy\n",
        "        SET recent_accuracy = (\n",
        "            SELECT accuracy\n",
        "            FROM ML.EVALUATE (\n",
        "                MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "                (\n",
        "                    SELECT *\n",
        "                    FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "                    WHERE DATE(instance_timestamp) = DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) \n",
        "                )\n",
        "            )\n",
        "        );\n",
        "\n",
        "        # retrain the model\n",
        "        CREATE OR REPLACE MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`\n",
        "            TRANSFORM(\n",
        "                ML.ROBUST_SCALER(body_mass_g) OVER() AS body_mass_g,\n",
        "                ML.STANDARD_SCALER(culmen_length_mm) OVER() AS culmen_length_mm,\n",
        "                ML.STANDARD_SCALER(culmen_depth_mm) OVER() AS culmen_depth_mm,\n",
        "                ML.QUANTILE_BUCKETIZE(flipper_length_mm, 3) OVER() AS flipper_length_mm,\n",
        "                ML.IMPUTER(sex, 'most_frequent') OVER() AS sex,\n",
        "                ML.IMPUTER(island, 'most_frequent') OVER() AS island,\n",
        "                species\n",
        "            )\n",
        "            OPTIONS(\n",
        "                MODEL_TYPE = 'RANDOM_FOREST_CLASSIFIER',\n",
        "                INPUT_LABEL_COLS = ['species'],\n",
        "\n",
        "                # data specifics\n",
        "                DATA_SPLIT_METHOD = 'AUTO_SPLIT',\n",
        "\n",
        "                # model specifics\n",
        "                AUTO_CLASS_WEIGHTS = FALSE,\n",
        "                NUM_PARALLEL_TREE= 150,\n",
        "                TREE_METHOD = 'HIST',\n",
        "                SUBSAMPLE = 0.85,\n",
        "                COLSAMPLE_BYTREE = 0.9,\n",
        "                ENABLE_GLOBAL_EXPLAIN = TRUE,\n",
        "\n",
        "                # register model in Vertex AI For Online Serving\n",
        "                MODEL_REGISTRY = 'VERTEX_AI'\n",
        "            )\n",
        "        AS\n",
        "            SELECT species, island, culmen_length_mm, culmen_depth_mm, sex, flipper_length_mm, body_mass_g\n",
        "            FROM `statmike-mlops-349915.bqml_model_monitoring.training`\n",
        "            UNION ALL\n",
        "            SELECT species, island, culmen_length_mm, culmen_depth_mm, sex, flipper_length_mm, body_mass_g\n",
        "            FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "            WHERE DATE(instance_timestamp) = DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)\n",
        "        ;\n",
        "\n",
        "        # get retrained models evaluation metrics: accuracy\n",
        "        SET retrain_accuracy = (\n",
        "            SELECT accuracy\n",
        "            FROM ML.EVALUATE (\n",
        "                MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`\n",
        "            )\n",
        "        );\n",
        "\n",
        "        SELECT ERROR(\n",
        "            CONCAT(\n",
        "                \"\\n\\nMonitoring Report:\",\n",
        "                \"\\n\\t\", drift,\n",
        "                \"\\n\\t\", skew,\n",
        "                \"\\nThe Model was retrained:\",\n",
        "                \"\\n\\taccuracy of prior model: \", train_accuracy,\n",
        "                \"\\n\\trecent accuracy of prior model: \", recent_accuracy,\n",
        "                \"\\n\\taccuracy after retraining: \", retrain_accuracy,\n",
        "                \"\\n\\n\"\n",
        "            )\n",
        "        );\n",
        "    END;\n",
        "    ELSE\n",
        "        SELECT ERROR(\n",
        "            CONCAT(\n",
        "                \"\\n\\nMonitoring Report:\",\n",
        "                \"\\n\\t\", drift,\n",
        "                \"\\n\\t\", skew,\n",
        "                \"\\n\\n\"\n",
        "            )\n",
        "        );\n",
        "END IF;"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r47M55B_xLVM"
      },
      "source": [
        "### From Job To Scheduled Query\n",
        "\n",
        "The model monitoring job created above can be modified and scheduled using [scheduled queries](https://cloud.google.com/bigquery/docs/scheduling-queries).  A key feature of the modification is making use of the parameter `@run_date` to parameterize the queries reference to dates (see [configuration options](https://cloud.google.com/bigquery/docs/scheduling-queries#configuration_options)).  This allows the query to run as though it is on different dates which will be helpful as we backfill the model monitoring runs after scheduling for today and forward.\n",
        "\n",
        "Note, the next cell is writing the content of the query to a file named `model_monitoring_job.sql`.\n",
        "Other modification to the job above:\n",
        "- Drift is set to compare the last weeks serving data to the prior week\n",
        "- Skew is set to compare the last weeks serving data to the baseline training data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 93,
      "metadata": {
        "tags": [],
        "id": "k6V3uWxRxLVM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Overwriting model_monitoring_job.sql\n"
          ]
        }
      ],
      "source": [
        "%%writefile model_monitoring_job.sql\n",
        "DECLARE drift_anomalies ARRAY\u003cSTRING\u003e;\n",
        "DECLARE drift STRING;\n",
        "DECLARE skew_anomalies ARRAY\u003cSTRING\u003e;\n",
        "DECLARE skew STRING;\n",
        "\n",
        "# Monitor Drift\n",
        "SET drift_anomalies = (\n",
        "    SELECT ARRAY_AGG(input)\n",
        "    FROM ML.VALIDATE_DATA_DRIFT(\n",
        "        (\n",
        "            SELECT * EXCEPT(instance_timestamp)\n",
        "            FROM `statmike-mlops-349915.bqml_model_monitoring.serving_all`\n",
        "            WHERE\n",
        "                DATE(instance_timestamp) \u003e= DATE_SUB(@run_date, INTERVAL 2 WEEK)\n",
        "                AND\n",
        "                DATE(instance_timestamp) \u003c DATE_SUB(@run_date, INTERVAL 1 WEEK)\n",
        "        ),\n",
        "        (\n",
        "            SELECT * EXCEPT(instance_timestamp)\n",
        "            FROM `statmike-mlops-349915.bqml_model_monitoring.serving_all`\n",
        "            WHERE DATE(instance_timestamp) \u003e= DATE_SUB(@run_date, INTERVAL 1 WEEK)\n",
        "        ),\n",
        "        STRUCT(\n",
        "            0.4 AS categorical_default_threshold,\n",
        "            0.4 AS numerical_default_threshold\n",
        "        )\n",
        "    )\n",
        "    WHERE is_anomaly = True\n",
        ");\n",
        "IF(ARRAY_LENGTH(drift_anomalies) \u003e 0) THEN\n",
        "    SET drift = CONCAT(\n",
        "        \"Found data drift (\", ARRAY_TO_STRING(drift_anomalies, \", \"), \")\"\n",
        "    );\n",
        "    ELSE SET drift = 'No drift detected.';\n",
        "END IF;\n",
        "\n",
        "# Monitor Skew\n",
        "SET skew_anomalies = (\n",
        "    SELECT ARRAY_AGG(input)\n",
        "    FROM ML.VALIDATE_DATA_SKEW(\n",
        "        MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "        (\n",
        "            SELECT * EXCEPT(instance_timestamp)\n",
        "            FROM `statmike-mlops-349915.bqml_model_monitoring.serving_all`\n",
        "            WHERE DATE(instance_timestamp) \u003e= DATE_SUB(@run_date, INTERVAL 1 WEEK)\n",
        "        )\n",
        "    )\n",
        "    WHERE is_anomaly = True\n",
        ");\n",
        "IF(ARRAY_LENGTH(skew_anomalies) \u003e 0) THEN\n",
        "    SET skew = CONCAT(\n",
        "        \"Found data skew (\", ARRAY_TO_STRING(skew_anomalies, \", \"), \")\"\n",
        "    );\n",
        "    ELSE SET skew = 'No skew detected.';\n",
        "END IF;\n",
        "\n",
        "# Prepare Alert\n",
        "IF(ARRAY_LENGTH(drift_anomalies) \u003e 0 OR ARRAY_LENGTH(skew_anomalies) \u003e 0) THEN\n",
        "    BEGIN\n",
        "        DECLARE train_accuracy FLOAT64;\n",
        "        DECLARE recent_accuracy FLOAT64;\n",
        "        DECLARE retrain_accuracy FLOAT64;\n",
        "\n",
        "        # get current models evaluation metrics: accuracy\n",
        "        SET train_accuracy = (\n",
        "            SELECT accuracy\n",
        "            FROM ML.EVALUATE (\n",
        "                MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`\n",
        "            )\n",
        "        );\n",
        "\n",
        "        # get current models recent evaluation metrics: accuracy\n",
        "        SET recent_accuracy = (\n",
        "            SELECT accuracy\n",
        "            FROM ML.EVALUATE (\n",
        "                MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`,\n",
        "                (\n",
        "                    SELECT *\n",
        "                    FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "                    WHERE DATE(instance_timestamp) \u003e= DATE_SUB(@run_date, INTERVAL 1 WEEK) \n",
        "                )\n",
        "            )\n",
        "        );\n",
        "\n",
        "        # retrain the model\n",
        "        CREATE OR REPLACE MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`\n",
        "            TRANSFORM(\n",
        "                ML.ROBUST_SCALER(body_mass_g) OVER() AS body_mass_g,\n",
        "                ML.STANDARD_SCALER(culmen_length_mm) OVER() AS culmen_length_mm,\n",
        "                ML.STANDARD_SCALER(culmen_depth_mm) OVER() AS culmen_depth_mm,\n",
        "                ML.QUANTILE_BUCKETIZE(flipper_length_mm, 3) OVER() AS flipper_length_mm,\n",
        "                ML.IMPUTER(sex, 'most_frequent') OVER() AS sex,\n",
        "                ML.IMPUTER(island, 'most_frequent') OVER() AS island,\n",
        "                species\n",
        "            )\n",
        "            OPTIONS(\n",
        "                MODEL_TYPE = 'RANDOM_FOREST_CLASSIFIER',\n",
        "                INPUT_LABEL_COLS = ['species'],\n",
        "\n",
        "                # data specifics\n",
        "                DATA_SPLIT_METHOD = 'AUTO_SPLIT',\n",
        "\n",
        "                # model specifics\n",
        "                AUTO_CLASS_WEIGHTS = FALSE,\n",
        "                NUM_PARALLEL_TREE= 150,\n",
        "                TREE_METHOD = 'HIST',\n",
        "                SUBSAMPLE = 0.85,\n",
        "                COLSAMPLE_BYTREE = 0.9,\n",
        "                ENABLE_GLOBAL_EXPLAIN = TRUE,\n",
        "\n",
        "                # register model in Vertex AI For Online Serving\n",
        "                MODEL_REGISTRY = 'VERTEX_AI'\n",
        "            )\n",
        "        AS\n",
        "            SELECT species, island, culmen_length_mm, culmen_depth_mm, sex, flipper_length_mm, body_mass_g\n",
        "            FROM `statmike-mlops-349915.bqml_model_monitoring.training_split`\n",
        "            WHERE splits = 'TRAIN'\n",
        "            UNION ALL\n",
        "            SELECT species, island, culmen_length_mm, culmen_depth_mm, sex, flipper_length_mm, body_mass_g\n",
        "            FROM `statmike-mlops-349915.bqml_model_monitoring.serving`\n",
        "            WHERE DATE(instance_timestamp) \u003c @run_date\n",
        "        ;\n",
        "\n",
        "        # get retrained models evaluation metrics: accuracy\n",
        "        SET retrain_accuracy = (\n",
        "            SELECT accuracy\n",
        "            FROM ML.EVALUATE (\n",
        "                MODEL `statmike-mlops-349915.bqml_model_monitoring.classify_species_rf`\n",
        "            )\n",
        "        );\n",
        "\n",
        "        SELECT ERROR(\n",
        "            CONCAT(\n",
        "                \"\\n\\nMonitoring Report for \", @run_date, \":\",\n",
        "                \"\\n\\t\", drift,\n",
        "                \"\\n\\t\", skew,\n",
        "                \"\\nThe Model was retrained:\",\n",
        "                \"\\n\\taccuracy of prior model: \", train_accuracy,\n",
        "                \"\\n\\trecent accuracy of prior model: \", recent_accuracy,\n",
        "                \"\\n\\taccuracy after retraining: \", retrain_accuracy,\n",
        "                \"\\n\\n\"\n",
        "            )\n",
        "        );\n",
        "    END;\n",
        "    # ELSE ... supressed alerts on runs without anomalies detected\n",
        "END IF;"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RIEeYgDmxLVM"
      },
      "source": [
        "### Schedule The Query\n",
        "\n",
        "The query can be scheduled in [a number of ways](https://cloud.google.com/bigquery/docs/scheduling-queries#set_up_scheduled_queries) including the console, bq cli, and Python. Here, the Python client will be used.\n",
        "- **References**\n",
        "    - [Python Client For BigQuery Data Transfer](https://cloud.google.com/python/docs/reference/bigquerydatatransfer/latest)\n",
        "    - [Set up schedule queries with Python client](https://cloud.google.com/bigquery/docs/scheduling-queries#python)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 94,
      "metadata": {
        "tags": [],
        "id": "d41gcuvrxLVM"
      },
      "outputs": [],
      "source": [
        "from google.cloud import bigquery_datatransfer\n",
        "from datetime import datetime, timezone, timedelta\n",
        "import time\n",
        "transfer_client = bigquery_datatransfer.DataTransferServiceClient()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 95,
      "metadata": {
        "tags": [],
        "id": "D_B7BJv_xLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "datetime.datetime(2024, 4, 19, 12, 26, 56, 372673, tzinfo=datetime.timezone.utc)"
            ]
          },
          "execution_count": 95,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "start_time = datetime.now(timezone.utc) + timedelta(days=1) # start tomorrow\n",
        "start_time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 96,
      "metadata": {
        "tags": [],
        "id": "Fif3zUIdxLVM"
      },
      "outputs": [],
      "source": [
        "with open('model_monitoring_job.sql', 'r') as file:\n",
        "    query = file.read()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 97,
      "metadata": {
        "tags": [],
        "id": "mXDuvzE8xLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'projects/statmike-mlops-349915/locations/us'"
            ]
          },
          "execution_count": 97,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "transfer_client.common_location_path('statmike-mlops-349915', 'us')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 98,
      "metadata": {
        "tags": [],
        "id": "dB5Z20a8xLVM"
      },
      "outputs": [],
      "source": [
        "scheduled_query = transfer_client.create_transfer_config(\n",
        "    #parent = transfer_client.common_project_path('statmike-mlops-349915'),\n",
        "    parent = transfer_client.common_location_path('statmike-mlops-349915', 'us'),\n",
        "    transfer_config = bigquery_datatransfer.TransferConfig(\n",
        "        display_name = 'Continous Model Monitoring',\n",
        "        data_source_id = 'scheduled_query',\n",
        "        params = {'query': query},\n",
        "        schedule = 'every 24 hours',\n",
        "        schedule_options = bigquery_datatransfer.ScheduleOptions(\n",
        "            start_time=start_time.strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n",
        "        ),\n",
        "        email_preferences = bigquery_datatransfer.EmailPreferences(\n",
        "            enable_failure_email = True\n",
        "        )\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 99,
      "metadata": {
        "tags": [],
        "id": "JPf_vTk4xLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "('Continous Model Monitoring',\n",
              " 'projects/1026793852137/locations/us/transferConfigs/664bfe7b-0000-25b2-b432-582429bf87dc')"
            ]
          },
          "execution_count": 99,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "scheduled_query.display_name, scheduled_query.name"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 100,
      "metadata": {
        "tags": [],
        "id": "STVEOI4xxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'2024-04-20 12:26:00.000000'"
            ]
          },
          "execution_count": 100,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "scheduled_query.next_run_time.strftime(\"%Y-%m-%d %H:%M:%S.%f\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SZvOLhu9xLVM"
      },
      "source": [
        "Take a moment to see the query in the console:\n",
        "- Console \u003e BigQuery \u003e Scheduled Queries"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vjz-ic-hxLVM"
      },
      "source": [
        "### Backfill History With Scheduled Query\n",
        "\n",
        "While the client can use a start and end time to run a range of backfill dates, it will runs these backwards from most recent to the earliest time.  Due to the nature of the model monitoring job it is more desirable to run these in the order from earliest to most recent.  This is accomplished by launching each days job separately in the desired order."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 101,
      "metadata": {
        "tags": [],
        "id": "9H0H0CLdxLVM"
      },
      "outputs": [],
      "source": [
        "# -30 days\n",
        "start_time = datetime.now(timezone.utc) + timedelta(days = -30)\n",
        "backfill_job = transfer_client.start_manual_transfer_runs(\n",
        "    request = bigquery_datatransfer.StartManualTransferRunsRequest(\n",
        "        parent = scheduled_query.name,\n",
        "        requested_run_time = start_time \n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 102,
      "metadata": {
        "tags": [],
        "id": "HFaOXRcvxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'projects/1026793852137/locations/us/transferConfigs/664bfe7b-0000-25b2-b432-582429bf87dc/runs/66497d6d-0000-27c4-b08b-582429aa5714'"
            ]
          },
          "execution_count": 102,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "backfill_job.runs[0].name"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 103,
      "metadata": {
        "tags": [],
        "id": "fzrxtaQMxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'RUNNING'"
            ]
          },
          "execution_count": 103,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "transfer_client.get_transfer_run(\n",
        "    name = backfill_job.runs[0].name\n",
        ").state.name"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2N7BPMXOxLVM"
      },
      "source": [
        "**NOTE:** A State of 'FAILED' is actually the correct end state as the method of using SQL ERRORS to pass the results and trigger notification emails was used. A State of 'SUCCEEDED' will represent runs without an anomaly detected.  [Read more about TrasferState](https://cloud.google.com/python/docs/reference/bigquerydatatransfer/latest/google.cloud.bigquery_datatransfer_v1.types.TransferState)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 109,
      "metadata": {
        "tags": [],
        "id": "vFa_FuJNxLVM"
      },
      "outputs": [],
      "source": [
        "while True:\n",
        "    state = transfer_client.get_transfer_run(\n",
        "        name = backfill_job.runs[0].name\n",
        "    ).state.name\n",
        "    if state in ['FAILED', 'SUCCEEDED', 'CANCELLED']:\n",
        "        break\n",
        "    else:\n",
        "        time.sleep(30)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 110,
      "metadata": {
        "tags": [],
        "id": "DFcEVvhIxLVM"
      },
      "outputs": [],
      "source": [
        "backfill_jobs =[backfill_job]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 111,
      "metadata": {
        "tags": [],
        "id": "IlrBaNbpxLVM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2024-03-20 12:42:33.086478+00:00\n",
            "2024-03-21 12:47:35.168343+00:00\n",
            "2024-03-22 12:51:26.721053+00:00\n",
            "2024-03-23 12:55:18.180583+00:00\n",
            "2024-03-24 13:00:19.834230+00:00\n",
            "2024-03-25 13:06:42.190407+00:00\n",
            "2024-03-26 13:08:02.842911+00:00\n",
            "2024-03-27 13:09:23.412273+00:00\n",
            "2024-03-28 13:15:45.706222+00:00\n",
            "2024-03-29 13:17:17.508638+00:00\n",
            "2024-03-30 13:18:38.115473+00:00\n",
            "2024-03-31 13:22:29.587652+00:00\n",
            "2024-04-01 13:23:50.225386+00:00\n",
            "2024-04-02 13:25:10.789751+00:00\n",
            "2024-04-03 13:26:31.344224+00:00\n",
            "2024-04-04 13:27:51.908045+00:00\n",
            "2024-04-05 13:31:43.342797+00:00\n",
            "2024-04-06 13:33:03.922409+00:00\n",
            "2024-04-07 13:34:24.907581+00:00\n",
            "2024-04-08 13:35:45.492920+00:00\n",
            "2024-04-09 13:37:06.131821+00:00\n",
            "2024-04-10 13:38:26.779414+00:00\n",
            "2024-04-11 13:39:47.413350+00:00\n",
            "2024-04-12 13:41:07.977307+00:00\n",
            "2024-04-13 13:42:28.600386+00:00\n",
            "2024-04-14 13:43:49.166801+00:00\n",
            "2024-04-15 13:45:09.751688+00:00\n",
            "2024-04-16 13:46:30.412630+00:00\n",
            "2024-04-17 13:47:51.086130+00:00\n",
            "2024-04-18 13:49:11.652702+00:00\n"
          ]
        }
      ],
      "source": [
        "# Loop From -29 to -0 days\n",
        "for days in range(30)[::-1]:\n",
        "    start_time = datetime.now(timezone.utc) + timedelta(days = -1*days)\n",
        "    print(start_time)\n",
        "    # start job\n",
        "    backfill_jobs.append(\n",
        "        transfer_client.start_manual_transfer_runs(\n",
        "            request = bigquery_datatransfer.StartManualTransferRunsRequest(\n",
        "                parent = scheduled_query.name,\n",
        "                requested_run_time = start_time \n",
        "            )\n",
        "        )\n",
        "    )\n",
        "    # monitor job before continuing:\n",
        "    while True:\n",
        "        state = transfer_client.get_transfer_run(\n",
        "            name = backfill_jobs[-1].runs[0].name\n",
        "        ).state.name\n",
        "        if state in ['FAILED', 'SUCCEEDED', 'CANCELLED']:\n",
        "            break\n",
        "        else:\n",
        "            time.sleep(10)    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 112,
      "metadata": {
        "tags": [],
        "id": "XkEjjBLAxLVM"
      },
      "outputs": [],
      "source": [
        "backfill_results = [transfer_client.get_transfer_run(name = j.runs[0].name) for j in backfill_jobs]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 113,
      "metadata": {
        "tags": [],
        "id": "bf4097IVxLVM"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\u003cstyle type=\"text/css\"\u003e\n",
              "#T_52ea5_row0_col0, #T_52ea5_row0_col1, #T_52ea5_row1_col0, #T_52ea5_row1_col1, #T_52ea5_row2_col0, #T_52ea5_row2_col1, #T_52ea5_row3_col0, #T_52ea5_row3_col1, #T_52ea5_row4_col0, #T_52ea5_row4_col1, #T_52ea5_row5_col0, #T_52ea5_row5_col1, #T_52ea5_row6_col0, #T_52ea5_row6_col1, #T_52ea5_row7_col0, #T_52ea5_row7_col1, #T_52ea5_row8_col0, #T_52ea5_row8_col1, #T_52ea5_row9_col0, #T_52ea5_row9_col1, #T_52ea5_row10_col0, #T_52ea5_row10_col1, #T_52ea5_row11_col0, #T_52ea5_row11_col1, #T_52ea5_row12_col0, #T_52ea5_row12_col1, #T_52ea5_row13_col0, #T_52ea5_row13_col1, #T_52ea5_row14_col0, #T_52ea5_row14_col1, #T_52ea5_row15_col0, #T_52ea5_row15_col1, #T_52ea5_row16_col0, #T_52ea5_row16_col1, #T_52ea5_row17_col0, #T_52ea5_row17_col1, #T_52ea5_row18_col0, #T_52ea5_row18_col1, #T_52ea5_row19_col0, #T_52ea5_row19_col1, #T_52ea5_row20_col0, #T_52ea5_row20_col1, #T_52ea5_row21_col0, #T_52ea5_row21_col1, #T_52ea5_row22_col0, #T_52ea5_row22_col1, #T_52ea5_row23_col0, #T_52ea5_row23_col1, #T_52ea5_row24_col0, #T_52ea5_row24_col1, #T_52ea5_row25_col0, #T_52ea5_row25_col1, #T_52ea5_row26_col0, #T_52ea5_row26_col1, #T_52ea5_row27_col0, #T_52ea5_row27_col1, #T_52ea5_row28_col0, #T_52ea5_row28_col1, #T_52ea5_row29_col0, #T_52ea5_row29_col1, #T_52ea5_row30_col0, #T_52ea5_row30_col1 {\n",
              "  text-align: left;\n",
              "  white-space: pre-wrap;\n",
              "}\n",
              "\u003c/style\u003e\n",
              "\u003ctable id=\"T_52ea5\"\u003e\n",
              "  \u003cthead\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth class=\"blank level0\" \u003e\u0026nbsp;\u003c/th\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_col0\" class=\"col_heading level0 col0\" \u003eDate\u003c/th\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_col1\" class=\"col_heading level0 col1\" \u003eResult\u003c/th\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/thead\u003e\n",
              "  \u003ctbody\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row0\" class=\"row_heading level0 row0\" \u003e0\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row0_col0\" class=\"data row0 col0\" \u003e2024-03-19 12:27:40.075971+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row0_col1\" class=\"data row0 col1\" \u003eQuery error: \n",
              "\n",
              "Monitoring Report for 2024-03-19:\n",
              "\tNo drift detected.\n",
              "\tFound data skew (flipper_length_mm)\n",
              "The Model was retrained:\n",
              "\taccuracy of prior model: 1\n",
              "\trecent accuracy of prior model: 0.9270687237026648\n",
              "\taccuracy after retraining: 0.99159663865546221\n",
              "\n",
              " at [132:9]; JobID: 1026793852137:scheduled_query_66497d6d-0000-27c4-b08b-582429aa5714\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row1\" class=\"row_heading level0 row1\" \u003e1\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row1_col0\" class=\"data row1 col0\" \u003e2024-03-20 12:42:33.086478+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row1_col1\" class=\"data row1 col1\" \u003eQuery error: \n",
              "\n",
              "Monitoring Report for 2024-03-20:\n",
              "\tNo drift detected.\n",
              "\tFound data skew (flipper_length_mm)\n",
              "The Model was retrained:\n",
              "\taccuracy of prior model: 1\n",
              "\trecent accuracy of prior model: 0.91070593735390371\n",
              "\taccuracy after retraining: 0.990228013029316\n",
              "\n",
              " at [132:9]; JobID: 1026793852137:scheduled_query_66231953-0000-2387-8e1c-14c14ef35818\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row2\" class=\"row_heading level0 row2\" \u003e2\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row2_col0\" class=\"data row2 col0\" \u003e2024-03-21 12:47:35.168343+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row2_col1\" class=\"data row2 col1\" \u003eQuery error: \n",
              "\n",
              "Monitoring Report for 2024-03-21:\n",
              "\tNo drift detected.\n",
              "\tFound data skew (flipper_length_mm)\n",
              "The Model was retrained:\n",
              "\taccuracy of prior model: 1\n",
              "\trecent accuracy of prior model: 0.91771856007480124\n",
              "\taccuracy after retraining: 0.99734042553191493\n",
              "\n",
              " at [132:9]; JobID: 1026793852137:scheduled_query_662320bc-0000-2387-8e1c-14c14ef35818\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row3\" class=\"row_heading level0 row3\" \u003e3\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row3_col0\" class=\"data row3 col0\" \u003e2024-03-22 12:51:26.721053+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row3_col1\" class=\"data row3 col1\" \u003eQuery error: \n",
              "\n",
              "Monitoring Report for 2024-03-22:\n",
              "\tNo drift detected.\n",
              "\tFound data skew (flipper_length_mm)\n",
              "The Model was retrained:\n",
              "\taccuracy of prior model: 1\n",
              "\trecent accuracy of prior model: 0.76110331930808783\n",
              "\taccuracy after retraining: 0.99775280898876417\n",
              "\n",
              " at [132:9]; JobID: 1026793852137:scheduled_query_66287403-0000-2082-92b0-c82add7a0624\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row4\" class=\"row_heading level0 row4\" \u003e4\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row4_col0\" class=\"data row4 col0\" \u003e2024-03-23 12:55:18.180583+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row4_col1\" class=\"data row4 col1\" \u003eQuery error: \n",
              "\n",
              "Monitoring Report for 2024-03-23:\n",
              "\tNo drift detected.\n",
              "\tFound data skew (flipper_length_mm)\n",
              "The Model was retrained:\n",
              "\taccuracy of prior model: 1\n",
              "\trecent accuracy of prior model: 0.90416082281439913\n",
              "\taccuracy after retraining: 1\n",
              "\n",
              " at [132:9]; JobID: 1026793852137:scheduled_query_665c3b7e-0000-226e-a4f9-001a1143dbe2\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row5\" class=\"row_heading level0 row5\" \u003e5\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row5_col0\" class=\"data row5 col0\" \u003e2024-03-24 13:00:19.834230+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row5_col1\" class=\"data row5 col1\" \u003eQuery error: \n",
              "\n",
              "Monitoring Report for 2024-03-24:\n",
              "\tNo drift detected.\n",
              "\tFound data skew (flipper_length_mm)\n",
              "The Model was retrained:\n",
              "\taccuracy of prior model: 1\n",
              "\trecent accuracy of prior model: 0.903225806451613\n",
              "\taccuracy after retraining: 0.98130841121495327\n",
              "\n",
              " at [132:9]; JobID: 1026793852137:scheduled_query_664f72c7-0000-2d79-84fa-14c14eea023c\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row6\" class=\"row_heading level0 row6\" \u003e6\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row6_col0\" class=\"data row6 col0\" \u003e2024-03-25 13:06:42.190407+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row6_col1\" class=\"data row6 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row7\" class=\"row_heading level0 row7\" \u003e7\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row7_col0\" class=\"data row7 col0\" \u003e2024-03-26 13:08:02.842911+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row7_col1\" class=\"data row7 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row8\" class=\"row_heading level0 row8\" \u003e8\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row8_col0\" class=\"data row8 col0\" \u003e2024-03-27 13:09:23.412273+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row8_col1\" class=\"data row8 col1\" \u003eQuery error: \n",
              "\n",
              "Monitoring Report for 2024-03-27:\n",
              "\tNo drift detected.\n",
              "\tFound data skew (flipper_length_mm)\n",
              "The Model was retrained:\n",
              "\taccuracy of prior model: 1\n",
              "\trecent accuracy of prior model: 0.95265700483091775\n",
              "\taccuracy after retraining: 1\n",
              "\n",
              " at [132:9]; JobID: 1026793852137:scheduled_query_6660683f-0000-20f1-9fd9-001a1143e9ec\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row9\" class=\"row_heading level0 row9\" \u003e9\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row9_col0\" class=\"data row9 col0\" \u003e2024-03-28 13:15:45.706222+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row9_col1\" class=\"data row9 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row10\" class=\"row_heading level0 row10\" \u003e10\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row10_col0\" class=\"data row10 col0\" \u003e2024-03-29 13:17:17.508638+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row10_col1\" class=\"data row10 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row11\" class=\"row_heading level0 row11\" \u003e11\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row11_col0\" class=\"data row11 col0\" \u003e2024-03-30 13:18:38.115473+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row11_col1\" class=\"data row11 col1\" \u003eQuery error: \n",
              "\n",
              "Monitoring Report for 2024-03-30:\n",
              "\tNo drift detected.\n",
              "\tFound data skew (flipper_length_mm)\n",
              "The Model was retrained:\n",
              "\taccuracy of prior model: 1\n",
              "\trecent accuracy of prior model: 0.99409554482018248\n",
              "\taccuracy after retraining: 0.98969072164948457\n",
              "\n",
              " at [132:9]; JobID: 1026793852137:scheduled_query_66513042-0000-24d3-80bc-d4f547f2bda4\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row12\" class=\"row_heading level0 row12\" \u003e12\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row12_col0\" class=\"data row12 col0\" \u003e2024-03-31 13:22:29.587652+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row12_col1\" class=\"data row12 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row13\" class=\"row_heading level0 row13\" \u003e13\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row13_col0\" class=\"data row13 col0\" \u003e2024-04-01 13:23:50.225386+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row13_col1\" class=\"data row13 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row14\" class=\"row_heading level0 row14\" \u003e14\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row14_col0\" class=\"data row14 col0\" \u003e2024-04-02 13:25:10.789751+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row14_col1\" class=\"data row14 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row15\" class=\"row_heading level0 row15\" \u003e15\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row15_col0\" class=\"data row15 col0\" \u003e2024-04-03 13:26:31.344224+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row15_col1\" class=\"data row15 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row16\" class=\"row_heading level0 row16\" \u003e16\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row16_col0\" class=\"data row16 col0\" \u003e2024-04-04 13:27:51.908045+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row16_col1\" class=\"data row16 col1\" \u003eQuery error: \n",
              "\n",
              "Monitoring Report for 2024-04-04:\n",
              "\tNo drift detected.\n",
              "\tFound data skew (flipper_length_mm)\n",
              "The Model was retrained:\n",
              "\taccuracy of prior model: 1\n",
              "\trecent accuracy of prior model: 0.98550724637681164\n",
              "\taccuracy after retraining: 0.99264705882352944\n",
              "\n",
              " at [132:9]; JobID: 1026793852137:scheduled_query_66219591-0000-2690-8ea2-14c14ef21f40\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row17\" class=\"row_heading level0 row17\" \u003e17\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row17_col0\" class=\"data row17 col0\" \u003e2024-04-05 13:31:43.342797+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row17_col1\" class=\"data row17 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row18\" class=\"row_heading level0 row18\" \u003e18\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row18_col0\" class=\"data row18 col0\" \u003e2024-04-06 13:33:03.922409+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row18_col1\" class=\"data row18 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row19\" class=\"row_heading level0 row19\" \u003e19\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row19_col0\" class=\"data row19 col0\" \u003e2024-04-07 13:34:24.907581+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row19_col1\" class=\"data row19 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row20\" class=\"row_heading level0 row20\" \u003e20\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row20_col0\" class=\"data row20 col0\" \u003e2024-04-08 13:35:45.492920+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row20_col1\" class=\"data row20 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row21\" class=\"row_heading level0 row21\" \u003e21\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row21_col0\" class=\"data row21 col0\" \u003e2024-04-09 13:37:06.131821+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row21_col1\" class=\"data row21 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row22\" class=\"row_heading level0 row22\" \u003e22\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row22_col0\" class=\"data row22 col0\" \u003e2024-04-10 13:38:26.779414+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row22_col1\" class=\"data row22 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row23\" class=\"row_heading level0 row23\" \u003e23\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row23_col0\" class=\"data row23 col0\" \u003e2024-04-11 13:39:47.413350+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row23_col1\" class=\"data row23 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row24\" class=\"row_heading level0 row24\" \u003e24\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row24_col0\" class=\"data row24 col0\" \u003e2024-04-12 13:41:07.977307+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row24_col1\" class=\"data row24 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row25\" class=\"row_heading level0 row25\" \u003e25\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row25_col0\" class=\"data row25 col0\" \u003e2024-04-13 13:42:28.600386+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row25_col1\" class=\"data row25 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row26\" class=\"row_heading level0 row26\" \u003e26\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row26_col0\" class=\"data row26 col0\" \u003e2024-04-14 13:43:49.166801+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row26_col1\" class=\"data row26 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row27\" class=\"row_heading level0 row27\" \u003e27\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row27_col0\" class=\"data row27 col0\" \u003e2024-04-15 13:45:09.751688+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row27_col1\" class=\"data row27 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row28\" class=\"row_heading level0 row28\" \u003e28\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row28_col0\" class=\"data row28 col0\" \u003e2024-04-16 13:46:30.412630+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row28_col1\" class=\"data row28 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row29\" class=\"row_heading level0 row29\" \u003e29\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row29_col0\" class=\"data row29 col0\" \u003e2024-04-17 13:47:51.086130+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row29_col1\" class=\"data row29 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "    \u003ctr\u003e\n",
              "      \u003cth id=\"T_52ea5_level0_row30\" class=\"row_heading level0 row30\" \u003e30\u003c/th\u003e\n",
              "      \u003ctd id=\"T_52ea5_row30_col0\" class=\"data row30 col0\" \u003e2024-04-18 13:49:11.652702+00:00\u003c/td\u003e\n",
              "      \u003ctd id=\"T_52ea5_row30_col1\" class=\"data row30 col1\" \u003e\u003c/td\u003e\n",
              "    \u003c/tr\u003e\n",
              "  \u003c/tbody\u003e\n",
              "\u003c/table\u003e\n"
            ],
            "text/plain": [
              "\u003cpandas.io.formats.style.Styler at 0x7faa0ba83b20\u003e"
            ]
          },
          "execution_count": 113,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pd.DataFrame(\n",
        "    {\n",
        "        'Date':[r.run_time for r in backfill_results],\n",
        "        'Result':[r.error_status.message for r in backfill_results]\n",
        "    }\n",
        ").style.set_properties(**{'text-align': 'left', 'white-space': 'pre-wrap'})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fFxz1FN7xLVM"
      },
      "source": [
        "---\n",
        "## More Resources\n",
        "\n",
        "Did you also check out the ['BigQuery ML (BQML) - Model Monitoring Introduction'](./bqml-model-monitoring-introduction.ipynb) for quick overview of model monitoring with BigQuery?  If not, try it out also. \n",
        "\n",
        "Read more about model monitoring in the documentation for the model monitoring functions:\n",
        "* [BigQuery ML Documentation](https://cloud.google.com/bigquery-ml/docs/)\n",
        "* [BigQuery ML Model Monitoring](https://cloud.google.com/bigquery/docs/model-monitoring-overview)\n",
        "    - [`ML.DESCRIBE_DATA`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-describe-data)\n",
        "    - [`ML.VALIDATE_DATA_SKEW`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-validate-data-skew)\n",
        "    - [`ML.VALIDATE_DATA_DRIFT`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-validate-data-drift)\n",
        "    - [`ML.TFDV_DESCRIBE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-tfdv-describe)\n",
        "    - [`ML.TFDV_VALIDATE`](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-tfdv-validate)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "81bi8YKTxLVM"
      },
      "source": [
        "---\n",
        "## Remove Resources Created In This Notebook\n",
        "\n",
        "The content is contained within the BigQuery dataset named `bqml_model_monitoring` created above and Vertex AI (endpoint created above).\n",
        "\n",
        "The following will delete the Vertex AI and BigQuery resources created above and the BigQuery dataset if it is empty."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 145,
      "metadata": {
        "tags": [],
        "id": "Rh4Iv31oxLVM"
      },
      "outputs": [],
      "source": [
        "# to remove resources change this flag to True:\n",
        "remove = False"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 148,
      "metadata": {
        "id": "s8mkK2nMxLVM"
      },
      "outputs": [],
      "source": [
        "if remove:\n",
        "    # client for BigQuery\n",
        "    from google.cloud import bigquery\n",
        "    bq = bigquery.Client(project = 'statmike-mlops-349915')\n",
        "    \n",
        "    # delete BigQuery table(s) and model(s) created above\n",
        "    bq.delete_table('statmike-mlops-349915.bqml_model_monitoring.training_split')\n",
        "    bq.delete_table('statmike-mlops-349915.bqml_model_monitoring.serving')\n",
        "    bq.delete_table('statmike-mlops-349915.bqml_model_monitoring.serving_online')\n",
        "    bq.delete_table('statmike-mlops-349915.bqml_model_monitoring.serving_all')\n",
        "    bq.delete_model('statmike-mlops-349915.bqml_model_monitoring.classify_species_rf')\n",
        "    \n",
        "    # delete BigQuery dataset - fails if other content was in the dataset than what this notebook created\n",
        "    bq.delete_dataset('statmike-mlops-349915.bqml_model_monitoring', delete_contents = False)\n",
        "    \n",
        "    # delete the scheduled query\n",
        "    transfer_client.delete_transfer_config(\n",
        "        name = scheduled_query.name\n",
        "    )"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "authorship_tag": "ABX9TyMy6GbowZC6hovxyAVQ0UWd",
      "provenance": []
    },
    "environment": {
      "kernel": "conda-root-py",
      "name": "workbench-notebooks.m119",
      "type": "gcloud",
      "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m119"
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "conda-root-py"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.14"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "1109199e5b67477788dcf20b48558d41": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_93f32edfd72e4a29ac69d2e202d6e14c",
              "IPY_MODEL_3da40e23ffbd47db8d1a250292255c32",
              "IPY_MODEL_bbd5c7d0941a458dbaee866da7e5366f"
            ],
            "layout": "IPY_MODEL_e2e3c77141c04fa7885bf4b4c0e314f4"
          }
        },
        "16d1acd4a3e94cf9bc3d683ff7eb1125": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1b76a7321f1e482e92b6725f5d120700": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2d65e5a9073f4b65b95c2c629255cac1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "39a8f6a0c95342299729b62b7a818084": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3acd5dce7d1b4a84b1b5540ec8d37ed3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "3da40e23ffbd47db8d1a250292255c32": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1b76a7321f1e482e92b6725f5d120700",
            "max": 5,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6827ef49075d48a1bb2582fb7fd52389",
            "value": 5
          }
        },
        "441b987af9044d1f9326f906d0f1d218": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "48543e1564664f73b553a0ae5bfb0843": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4e5a11ea456d49a8bd7563e9b2c35511": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "543a939bbf684e49ad83771418dfdd09": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "549c9135cfbe467ebc3a60d6ad5750e6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e0194e91019f4661a01322bbda67ac18",
            "placeholder": "​",
            "style": "IPY_MODEL_6b0ff87e15fd4ff684355a521e0f7a1f",
            "value": ""
          }
        },
        "5805c14443ec4d3ebf24ea8dda735a99": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6827ef49075d48a1bb2582fb7fd52389": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6b0ff87e15fd4ff684355a521e0f7a1f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8930cc60223a4b56a70a17b1c5d03a93": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_48543e1564664f73b553a0ae5bfb0843",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3acd5dce7d1b4a84b1b5540ec8d37ed3",
            "value": 1
          }
        },
        "93f32edfd72e4a29ac69d2e202d6e14c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_543a939bbf684e49ad83771418dfdd09",
            "placeholder": "​",
            "style": "IPY_MODEL_4e5a11ea456d49a8bd7563e9b2c35511",
            "value": "Downloading: 100%"
          }
        },
        "bbd5c7d0941a458dbaee866da7e5366f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5805c14443ec4d3ebf24ea8dda735a99",
            "placeholder": "​",
            "style": "IPY_MODEL_441b987af9044d1f9326f906d0f1d218",
            "value": ""
          }
        },
        "c656f444dbc4405a915eaecde42707d5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_16d1acd4a3e94cf9bc3d683ff7eb1125",
            "placeholder": "​",
            "style": "IPY_MODEL_2d65e5a9073f4b65b95c2c629255cac1",
            "value": "Job ID a2f9486b-cfda-4aad-959b-b5485f8f5dd8 successfully executed: 100%"
          }
        },
        "ddb7ba0f48b24355b23492fcb1db5939": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_c656f444dbc4405a915eaecde42707d5",
              "IPY_MODEL_8930cc60223a4b56a70a17b1c5d03a93",
              "IPY_MODEL_549c9135cfbe467ebc3a60d6ad5750e6"
            ],
            "layout": "IPY_MODEL_39a8f6a0c95342299729b62b7a818084"
          }
        },
        "e0194e91019f4661a01322bbda67ac18": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e2e3c77141c04fa7885bf4b4c0e314f4": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}