{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Tce3stUlHN0L"
   },
   "source": [
    "##### Copyright 2020 The TensorFlow Authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:36.683809Z",
     "iopub.status.busy": "2024-04-20T11:24:36.683242Z",
     "iopub.status.idle": "2024-04-20T11:24:36.687044Z",
     "shell.execute_reply": "2024-04-20T11:24:36.686480Z"
    },
    "id": "tuOe1ymfHZPu"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8yo62ffS5TF5"
   },
   "source": [
    "# Inspect and debug decision forest models\n",
    "\n",
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://www.tensorflow.org/decision_forests/tutorials/advanced_colab\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/decision-forests/blob/main/documentation/tutorials/advanced_colab.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/tensorflow/decision-forests/blob/main/documentation/tutorials/advanced_colab.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View on GitHub</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://storage.googleapis.com/tensorflow_docs/decision-forests/documentation/tutorials/advanced_colab.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
    "  </td>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "84wIz8LPiLDF"
   },
   "source": [
    "In this colab, you will learn how to inspect and create the structure of a model directly. We assume you are familiar with the concepts introduced in the\n",
    "[beginner](beginner_colab.ipynb) and [intermediate](intermediate_colab.ipynb)\n",
    "colabs.\n",
    "\n",
    "In this colab, you will:\n",
    "\n",
    "1.  Train a Random Forest model and access its structure programmatically.\n",
    "\n",
    "1.  Create a Random Forest model by hand and use it as a classical model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Rzskapxq7gdo"
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:36.690640Z",
     "iopub.status.busy": "2024-04-20T11:24:36.690055Z",
     "iopub.status.idle": "2024-04-20T11:24:41.452072Z",
     "shell.execute_reply": "2024-04-20T11:24:41.451100Z"
    },
    "id": "mZiInVYfffAb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tensorflow_decision_forests\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Using cached tensorflow_decision_forests-1.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.0 kB)\r\n",
      "Requirement already satisfied: numpy in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (1.26.4)\r\n",
      "Requirement already satisfied: pandas in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (2.2.2)\r\n",
      "Requirement already satisfied: tensorflow~=2.16.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (2.16.1)\r\n",
      "Requirement already satisfied: six in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (1.16.0)\r\n",
      "Requirement already satisfied: absl-py in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (1.4.0)\r\n",
      "Requirement already satisfied: wheel in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (0.41.2)\r\n",
      "Collecting wurlitzer (from tensorflow_decision_forests)\r\n",
      "  Using cached wurlitzer-3.0.3-py3-none-any.whl.metadata (1.9 kB)\r\n",
      "Requirement already satisfied: tf-keras~=2.16 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow_decision_forests) (2.16.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: astunparse>=1.6.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (1.6.3)\r\n",
      "Requirement already satisfied: flatbuffers>=23.5.26 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (24.3.25)\r\n",
      "Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (0.5.4)\r\n",
      "Requirement already satisfied: google-pasta>=0.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (0.2.0)\r\n",
      "Requirement already satisfied: h5py>=3.10.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (3.11.0)\r\n",
      "Requirement already satisfied: libclang>=13.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (18.1.1)\r\n",
      "Requirement already satisfied: ml-dtypes~=0.3.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (0.3.2)\r\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (3.3.0)\r\n",
      "Requirement already satisfied: packaging in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (24.0)\r\n",
      "Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (3.20.3)\r\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (2.31.0)\r\n",
      "Requirement already satisfied: setuptools in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (69.5.1)\r\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (2.4.0)\r\n",
      "Requirement already satisfied: typing-extensions>=3.6.6 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (4.11.0)\r\n",
      "Requirement already satisfied: wrapt>=1.11.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (1.16.0)\r\n",
      "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (1.63.0rc2)\r\n",
      "Requirement already satisfied: tensorboard<2.17,>=2.16 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (2.16.2)\r\n",
      "Requirement already satisfied: keras>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (3.2.1)\r\n",
      "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorflow~=2.16.1->tensorflow_decision_forests) (0.36.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: python-dateutil>=2.8.2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pandas->tensorflow_decision_forests) (2.9.0.post0)\r\n",
      "Requirement already satisfied: pytz>=2020.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pandas->tensorflow_decision_forests) (2024.1)\r\n",
      "Requirement already satisfied: tzdata>=2022.7 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from pandas->tensorflow_decision_forests) (2024.1)\r\n",
      "Requirement already satisfied: rich in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras>=3.0.0->tensorflow~=2.16.1->tensorflow_decision_forests) (13.7.1)\r\n",
      "Requirement already satisfied: namex in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras>=3.0.0->tensorflow~=2.16.1->tensorflow_decision_forests) (0.0.8)\r\n",
      "Requirement already satisfied: optree in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from keras>=3.0.0->tensorflow~=2.16.1->tensorflow_decision_forests) (0.11.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: charset-normalizer<4,>=2 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorflow~=2.16.1->tensorflow_decision_forests) (3.3.2)\r\n",
      "Requirement already satisfied: idna<4,>=2.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorflow~=2.16.1->tensorflow_decision_forests) (3.7)\r\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorflow~=2.16.1->tensorflow_decision_forests) (2.2.1)\r\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from requests<3,>=2.21.0->tensorflow~=2.16.1->tensorflow_decision_forests) (2024.2.2)\r\n",
      "Requirement already satisfied: markdown>=2.6.8 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.17,>=2.16->tensorflow~=2.16.1->tensorflow_decision_forests) (3.6)\r\n",
      "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.17,>=2.16->tensorflow~=2.16.1->tensorflow_decision_forests) (0.7.2)\r\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from tensorboard<2.17,>=2.16->tensorflow~=2.16.1->tensorflow_decision_forests) (3.0.2)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: importlib-metadata>=4.4 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from markdown>=2.6.8->tensorboard<2.17,>=2.16->tensorflow~=2.16.1->tensorflow_decision_forests) (7.1.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: MarkupSafe>=2.1.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from werkzeug>=1.0.1->tensorboard<2.17,>=2.16->tensorflow~=2.16.1->tensorflow_decision_forests) (2.1.5)\r\n",
      "Requirement already satisfied: markdown-it-py>=2.2.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from rich->keras>=3.0.0->tensorflow~=2.16.1->tensorflow_decision_forests) (3.0.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from rich->keras>=3.0.0->tensorflow~=2.16.1->tensorflow_decision_forests) (2.17.2)\r\n",
      "Requirement already satisfied: zipp>=0.5 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.17,>=2.16->tensorflow~=2.16.1->tensorflow_decision_forests) (3.18.1)\r\n",
      "Requirement already satisfied: mdurl~=0.1 in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (from markdown-it-py>=2.2.0->rich->keras>=3.0.0->tensorflow~=2.16.1->tensorflow_decision_forests) (0.1.2)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cached tensorflow_decision_forests-1.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.5 MB)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cached wurlitzer-3.0.3-py3-none-any.whl (7.3 kB)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: wurlitzer, tensorflow_decision_forests\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully installed tensorflow_decision_forests-1.9.0 wurlitzer-3.0.3\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: wurlitzer in /tmpfs/src/tf_docs_env/lib/python3.9/site-packages (3.0.3)\r\n"
     ]
    }
   ],
   "source": [
    "# Install TensorFlow Decision Forests.\n",
    "!pip install tensorflow_decision_forests\n",
    "\n",
    "# Use wurlitzer to show the training logs.\n",
    "!pip install wurlitzer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:41.456245Z",
     "iopub.status.busy": "2024-04-20T11:24:41.455950Z",
     "iopub.status.idle": "2024-04-20T11:24:44.147752Z",
     "shell.execute_reply": "2024-04-20T11:24:44.146995Z"
    },
    "id": "RsCV2oAS7gC_"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "# Keep using Keras 2\n",
    "os.environ['TF_USE_LEGACY_KERAS'] = '1'\n",
    "\n",
    "import tensorflow_decision_forests as tfdf\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import tf_keras\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "import collections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xV3klWJnyCgH"
   },
   "source": [
    "The hidden code cell limits the output height in colab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:44.152047Z",
     "iopub.status.busy": "2024-04-20T11:24:44.151605Z",
     "iopub.status.idle": "2024-04-20T11:24:44.156227Z",
     "shell.execute_reply": "2024-04-20T11:24:44.155617Z"
    },
    "id": "XAWSjWrQmVE0"
   },
   "outputs": [],
   "source": [
    "#@title\n",
    "\n",
    "from IPython.core.magic import register_line_magic\n",
    "from IPython.display import Javascript\n",
    "from IPython.display import display as ipy_display\n",
    "\n",
    "# Some of the model training logs can cover the full\n",
    "# screen if not compressed to a smaller viewport.\n",
    "# This magic allows setting a max height for a cell.\n",
    "@register_line_magic\n",
    "def set_cell_height(size):\n",
    "  ipy_display(\n",
    "      Javascript(\"google.colab.output.setIframeHeight(0, true, {maxHeight: \" +\n",
    "                 str(size) + \"})\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M_D4Ft4o65XT"
   },
   "source": [
    "## Train a simple Random Forest\n",
    "\n",
    "We train a Random Forest like in the [beginner colab](beginner_colab.ipynb):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:44.159522Z",
     "iopub.status.busy": "2024-04-20T11:24:44.158991Z",
     "iopub.status.idle": "2024-04-20T11:24:54.402903Z",
     "shell.execute_reply": "2024-04-20T11:24:54.402089Z"
    },
    "id": "tTW2aBiVcU3E"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  species     island  bill_length_mm  bill_depth_mm  flipper_length_mm  \\\n",
      "0  Adelie  Torgersen            39.1           18.7              181.0   \n",
      "1  Adelie  Torgersen            39.5           17.4              186.0   \n",
      "2  Adelie  Torgersen            40.3           18.0              195.0   \n",
      "\n",
      "   body_mass_g     sex  year  \n",
      "0       3750.0    male  2007  \n",
      "1       3800.0  female  2007  \n",
      "2       3250.0  female  2007  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:The `num_threads` constructor argument is not set and the number of CPU is os.cpu_count()=32 > 32. Setting num_threads to 32. Set num_threads manually to use more than 32 cpus.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Use /tmpfs/tmp/tmpadwizz7x as temporary training directory\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading training dataset...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training dataset read in 0:00:03.574049. Found 344 examples.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model trained in 0:00:00.092571\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compiling model...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO 24-04-20 11:24:50.3886 UTC kernel.cc:1233] Loading model from path /tmpfs/tmp/tmpadwizz7x/model/ with prefix 59499fe5fa654879\n",
      "[INFO 24-04-20 11:24:50.4047 UTC decision_forest.cc:734] Model loaded with 300 root(s), 5080 node(s), and 7 input feature(s).\n",
      "[INFO 24-04-20 11:24:50.4047 UTC abstract_model.cc:1344] Engine \"RandomForestGeneric\" built\n",
      "[INFO 24-04-20 11:24:50.4048 UTC kernel.cc:1061] Use fast generic engine\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model compiled.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tf_keras.src.callbacks.History at 0x7fb16472dbe0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Download the dataset\n",
    "!wget -q https://storage.googleapis.com/download.tensorflow.org/data/palmer_penguins/penguins.csv -O /tmp/penguins.csv\n",
    "\n",
    "# Load a dataset into a Pandas Dataframe.\n",
    "dataset_df = pd.read_csv(\"/tmp/penguins.csv\")\n",
    "\n",
    "# Show the first three examples.\n",
    "print(dataset_df.head(3))\n",
    "\n",
    "# Convert the pandas dataframe into a tf dataset.\n",
    "dataset_tf = tfdf.keras.pd_dataframe_to_tf_dataset(dataset_df, label=\"species\")\n",
    "\n",
    "# Train the Random Forest\n",
    "model = tfdf.keras.RandomForestModel(compute_oob_variable_importances=True)\n",
    "model.fit(x=dataset_tf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b7Xie0bhcw8_"
   },
   "source": [
    "Note the `compute_oob_variable_importances=True`\n",
    "hyper-parameter in the model constructor. This option computes the Out-of-bag (OOB)\n",
    "variable importance during training. This is a popular\n",
    "[permutation variable importance](https://christophm.github.io/interpretable-ml-book/feature-importance.html) for Random Forest models.\n",
    "\n",
    "Computing the OOB Variable importance does not impact the final model, it will slow the training on large datasets.\n",
    "\n",
    "Check the model summary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.406750Z",
     "iopub.status.busy": "2024-04-20T11:24:54.406107Z",
     "iopub.status.idle": "2024-04-20T11:24:54.417345Z",
     "shell.execute_reply": "2024-04-20T11:24:54.416704Z"
    },
    "id": "fsQYD-jFc2EH"
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "google.colab.output.setIframeHeight(0, true, {maxHeight: 300})"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"random_forest_model\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Layer (type)                Output Shape              Param #   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=================================================================\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=================================================================\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total params: 1 (1.00 Byte)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trainable params: 0 (0.00 Byte)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Non-trainable params: 1 (1.00 Byte)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type: \"RANDOM_FOREST\"\n",
      "Task: CLASSIFICATION\n",
      "Label: \"__LABEL\"\n",
      "\n",
      "Input Features (7):\n",
      "\tbill_depth_mm\n",
      "\tbill_length_mm\n",
      "\tbody_mass_g\n",
      "\tflipper_length_mm\n",
      "\tisland\n",
      "\tsex\n",
      "\tyear\n",
      "\n",
      "No weights\n",
      "\n",
      "Variable Importance: INV_MEAN_MIN_DEPTH:\n",
      "    1. \"flipper_length_mm\"  0.440513 ################\n",
      "    2.    \"bill_length_mm\"  0.438028 ###############\n",
      "    3.     \"bill_depth_mm\"  0.299751 #####\n",
      "    4.            \"island\"  0.295079 #####\n",
      "    5.       \"body_mass_g\"  0.256534 ##\n",
      "    6.               \"sex\"  0.225708 \n",
      "    7.              \"year\"  0.224020 \n",
      "\n",
      "Variable Importance: MEAN_DECREASE_IN_ACCURACY:\n",
      "    1.    \"bill_length_mm\"  0.151163 ################\n",
      "    2.            \"island\"  0.008721 #\n",
      "    3.     \"bill_depth_mm\"  0.000000 \n",
      "    4.       \"body_mass_g\"  0.000000 \n",
      "    5.               \"sex\"  0.000000 \n",
      "    6.              \"year\"  0.000000 \n",
      "    7. \"flipper_length_mm\" -0.002907 \n",
      "\n",
      "Variable Importance: MEAN_DECREASE_IN_AP_1_VS_OTHERS:\n",
      "    1.    \"bill_length_mm\"  0.083305 ################\n",
      "    2.            \"island\"  0.007664 #\n",
      "    3. \"flipper_length_mm\"  0.003400 \n",
      "    4.     \"bill_depth_mm\"  0.002741 \n",
      "    5.       \"body_mass_g\"  0.000722 \n",
      "    6.               \"sex\"  0.000644 \n",
      "    7.              \"year\"  0.000000 \n",
      "\n",
      "Variable Importance: MEAN_DECREASE_IN_AP_2_VS_OTHERS:\n",
      "    1.    \"bill_length_mm\"  0.508510 ################\n",
      "    2.            \"island\"  0.023487 \n",
      "    3.     \"bill_depth_mm\"  0.007744 \n",
      "    4. \"flipper_length_mm\"  0.006008 \n",
      "    5.       \"body_mass_g\"  0.003017 \n",
      "    6.               \"sex\"  0.001537 \n",
      "    7.              \"year\" -0.000245 \n",
      "\n",
      "Variable Importance: MEAN_DECREASE_IN_AP_3_VS_OTHERS:\n",
      "    1.            \"island\"  0.002192 ################\n",
      "    2.    \"bill_length_mm\"  0.001572 ############\n",
      "    3.     \"bill_depth_mm\"  0.000497 #######\n",
      "    4.               \"sex\"  0.000000 ####\n",
      "    5.              \"year\"  0.000000 ####\n",
      "    6.       \"body_mass_g\" -0.000053 ####\n",
      "    7. \"flipper_length_mm\" -0.000890 \n",
      "\n",
      "Variable Importance: MEAN_DECREASE_IN_AUC_1_VS_OTHERS:\n",
      "    1.    \"bill_length_mm\"  0.071306 ################\n",
      "    2.            \"island\"  0.007299 #\n",
      "    3. \"flipper_length_mm\"  0.004506 #\n",
      "    4.     \"bill_depth_mm\"  0.002124 \n",
      "    5.       \"body_mass_g\"  0.000548 \n",
      "    6.               \"sex\"  0.000480 \n",
      "    7.              \"year\"  0.000000 \n",
      "\n",
      "Variable Importance: MEAN_DECREASE_IN_AUC_2_VS_OTHERS:\n",
      "    1.    \"bill_length_mm\"  0.108642 ################\n",
      "    2.            \"island\"  0.014493 ##\n",
      "    3.     \"bill_depth_mm\"  0.007406 #\n",
      "    4. \"flipper_length_mm\"  0.005195 \n",
      "    5.       \"body_mass_g\"  0.001012 \n",
      "    6.               \"sex\"  0.000480 \n",
      "    7.              \"year\" -0.000053 \n",
      "\n",
      "Variable Importance: MEAN_DECREASE_IN_AUC_3_VS_OTHERS:\n",
      "    1.            \"island\"  0.002126 ################\n",
      "    2.    \"bill_length_mm\"  0.001393 ###########\n",
      "    3.     \"bill_depth_mm\"  0.000293 #####\n",
      "    4.               \"sex\"  0.000000 ###\n",
      "    5.              \"year\"  0.000000 ###\n",
      "    6.       \"body_mass_g\" -0.000037 ###\n",
      "    7. \"flipper_length_mm\" -0.000550 \n",
      "\n",
      "Variable Importance: MEAN_DECREASE_IN_PRAUC_1_VS_OTHERS:\n",
      "    1.    \"bill_length_mm\"  0.083122 ################\n",
      "    2.            \"island\"  0.010887 ##\n",
      "    3. \"flipper_length_mm\"  0.003425 \n",
      "    4.     \"bill_depth_mm\"  0.002731 \n",
      "    5.       \"body_mass_g\"  0.000719 \n",
      "    6.               \"sex\"  0.000641 \n",
      "    7.              \"year\"  0.000000 \n",
      "\n",
      "Variable Importance: MEAN_DECREASE_IN_PRAUC_2_VS_OTHERS:\n",
      "    1.    \"bill_length_mm\"  0.497611 ################\n",
      "    2.            \"island\"  0.024045 \n",
      "    3.     \"bill_depth_mm\"  0.007734 \n",
      "    4. \"flipper_length_mm\"  0.006017 \n",
      "    5.       \"body_mass_g\"  0.003000 \n",
      "    6.               \"sex\"  0.001528 \n",
      "    7.              \"year\" -0.000243 \n",
      "\n",
      "Variable Importance: MEAN_DECREASE_IN_PRAUC_3_VS_OTHERS:\n",
      "    1.            \"island\"  0.002187 ################\n",
      "    2.    \"bill_length_mm\"  0.001568 ############\n",
      "    3.     \"bill_depth_mm\"  0.000495 #######\n",
      "    4.               \"sex\"  0.000000 ####\n",
      "    5.              \"year\"  0.000000 ####\n",
      "    6.       \"body_mass_g\" -0.000053 ####\n",
      "    7. \"flipper_length_mm\" -0.000886 \n",
      "\n",
      "Variable Importance: NUM_AS_ROOT:\n",
      "    1. \"flipper_length_mm\" 157.000000 ################\n",
      "    2.    \"bill_length_mm\" 76.000000 #######\n",
      "    3.     \"bill_depth_mm\" 52.000000 #####\n",
      "    4.            \"island\" 12.000000 \n",
      "    5.       \"body_mass_g\"  3.000000 \n",
      "\n",
      "Variable Importance: NUM_NODES:\n",
      "    1.    \"bill_length_mm\" 778.000000 ################\n",
      "    2.     \"bill_depth_mm\" 463.000000 #########\n",
      "    3. \"flipper_length_mm\" 414.000000 ########\n",
      "    4.            \"island\" 342.000000 ######\n",
      "    5.       \"body_mass_g\" 338.000000 ######\n",
      "    6.               \"sex\" 36.000000 \n",
      "    7.              \"year\" 19.000000 \n",
      "\n",
      "Variable Importance: SUM_SCORE:\n",
      "    1.    \"bill_length_mm\" 36515.793787 ################\n",
      "    2. \"flipper_length_mm\" 35120.434174 ###############\n",
      "    3.            \"island\" 14669.408395 ######\n",
      "    4.     \"bill_depth_mm\" 14515.446617 ######\n",
      "    5.       \"body_mass_g\" 3485.330881 #\n",
      "    6.               \"sex\" 354.201073 \n",
      "    7.              \"year\" 49.737758 \n",
      "\n",
      "\n",
      "\n",
      "Winner takes all: true\n",
      "Out-of-bag evaluation: accuracy:0.976744 logloss:0.068949\n",
      "Number of trees: 300\n",
      "Total number of nodes: 5080\n",
      "\n",
      "Number of nodes by tree:\n",
      "Count: 300 Average: 16.9333 StdDev: 3.10197\n",
      "Min: 11 Max: 31 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 11, 12)  6   2.00%   2.00% #\n",
      "[ 12, 13)  0   0.00%   2.00%\n",
      "[ 13, 14) 46  15.33%  17.33% #####\n",
      "[ 14, 15)  0   0.00%  17.33%\n",
      "[ 15, 16) 70  23.33%  40.67% ########\n",
      "[ 16, 17)  0   0.00%  40.67%\n",
      "[ 17, 18) 84  28.00%  68.67% ##########\n",
      "[ 18, 19)  0   0.00%  68.67%\n",
      "[ 19, 20) 46  15.33%  84.00% #####\n",
      "[ 20, 21)  0   0.00%  84.00%\n",
      "[ 21, 22) 30  10.00%  94.00% ####\n",
      "[ 22, 23)  0   0.00%  94.00%\n",
      "[ 23, 24) 13   4.33%  98.33% ##\n",
      "[ 24, 25)  0   0.00%  98.33%\n",
      "[ 25, 26)  2   0.67%  99.00%\n",
      "[ 26, 27)  0   0.00%  99.00%\n",
      "[ 27, 28)  2   0.67%  99.67%\n",
      "[ 28, 29)  0   0.00%  99.67%\n",
      "[ 29, 30)  0   0.00%  99.67%\n",
      "[ 30, 31]  1   0.33% 100.00%\n",
      "\n",
      "Depth by leafs:\n",
      "Count: 2690 Average: 3.53271 StdDev: 1.06789\n",
      "Min: 2 Max: 7 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 2, 3) 545  20.26%  20.26% ######\n",
      "[ 3, 4) 747  27.77%  48.03% ########\n",
      "[ 4, 5) 888  33.01%  81.04% ##########\n",
      "[ 5, 6) 444  16.51%  97.55% #####\n",
      "[ 6, 7)  62   2.30%  99.85% #\n",
      "[ 7, 7]   4   0.15% 100.00%\n",
      "\n",
      "Number of training obs by leaf:\n",
      "Count: 2690 Average: 38.3643 StdDev: 44.8651\n",
      "Min: 5 Max: 155 Ignored: 0\n",
      "----------------------------------------------\n",
      "[   5,  12) 1474  54.80%  54.80% ##########\n",
      "[  12,  20)  124   4.61%  59.41% #\n",
      "[  20,  27)   48   1.78%  61.19%\n",
      "[  27,  35)   74   2.75%  63.94% #\n",
      "[  35,  42)   58   2.16%  66.10%\n",
      "[  42,  50)   85   3.16%  69.26% #\n",
      "[  50,  57)   96   3.57%  72.83% #\n",
      "[  57,  65)   87   3.23%  76.06% #\n",
      "[  65,  72)   49   1.82%  77.88%\n",
      "[  72,  80)   23   0.86%  78.74%\n",
      "[  80,  88)   30   1.12%  79.85%\n",
      "[  88,  95)   23   0.86%  80.71%\n",
      "[  95, 103)   42   1.56%  82.27%\n",
      "[ 103, 110)   62   2.30%  84.57%\n",
      "[ 110, 118)  115   4.28%  88.85% #\n",
      "[ 118, 125)  115   4.28%  93.12% #\n",
      "[ 125, 133)   98   3.64%  96.77% #\n",
      "[ 133, 140)   49   1.82%  98.59%\n",
      "[ 140, 148)   31   1.15%  99.74%\n",
      "[ 148, 155]    7   0.26% 100.00%\n",
      "\n",
      "Attribute in nodes:\n",
      "\t778 : bill_length_mm [NUMERICAL]\n",
      "\t463 : bill_depth_mm [NUMERICAL]\n",
      "\t414 : flipper_length_mm [NUMERICAL]\n",
      "\t342 : island [CATEGORICAL]\n",
      "\t338 : body_mass_g [NUMERICAL]\n",
      "\t36 : sex [CATEGORICAL]\n",
      "\t19 : year [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 0:\n",
      "\t157 : flipper_length_mm [NUMERICAL]\n",
      "\t76 : bill_length_mm [NUMERICAL]\n",
      "\t52 : bill_depth_mm [NUMERICAL]\n",
      "\t12 : island [CATEGORICAL]\n",
      "\t3 : body_mass_g [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 1:\n",
      "\t250 : bill_length_mm [NUMERICAL]\n",
      "\t244 : flipper_length_mm [NUMERICAL]\n",
      "\t183 : bill_depth_mm [NUMERICAL]\n",
      "\t170 : island [CATEGORICAL]\n",
      "\t53 : body_mass_g [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 2:\n",
      "\t462 : bill_length_mm [NUMERICAL]\n",
      "\t320 : flipper_length_mm [NUMERICAL]\n",
      "\t310 : bill_depth_mm [NUMERICAL]\n",
      "\t287 : island [CATEGORICAL]\n",
      "\t162 : body_mass_g [NUMERICAL]\n",
      "\t9 : sex [CATEGORICAL]\n",
      "\t5 : year [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 3:\n",
      "\t669 : bill_length_mm [NUMERICAL]\n",
      "\t410 : bill_depth_mm [NUMERICAL]\n",
      "\t383 : flipper_length_mm [NUMERICAL]\n",
      "\t328 : island [CATEGORICAL]\n",
      "\t286 : body_mass_g [NUMERICAL]\n",
      "\t32 : sex [CATEGORICAL]\n",
      "\t10 : year [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 5:\n",
      "\t778 : bill_length_mm [NUMERICAL]\n",
      "\t462 : bill_depth_mm [NUMERICAL]\n",
      "\t413 : flipper_length_mm [NUMERICAL]\n",
      "\t342 : island [CATEGORICAL]\n",
      "\t338 : body_mass_g [NUMERICAL]\n",
      "\t36 : sex [CATEGORICAL]\n",
      "\t19 : year [NUMERICAL]\n",
      "\n",
      "Condition type in nodes:\n",
      "\t2012 : HigherCondition\n",
      "\t378 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 0:\n",
      "\t288 : HigherCondition\n",
      "\t12 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 1:\n",
      "\t730 : HigherCondition\n",
      "\t170 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 2:\n",
      "\t1259 : HigherCondition\n",
      "\t296 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 3:\n",
      "\t1758 : HigherCondition\n",
      "\t360 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 5:\n",
      "\t2010 : HigherCondition\n",
      "\t378 : ContainsBitmapCondition\n",
      "Node format: NOT_SET\n",
      "\n",
      "Training OOB:\n",
      "\ttrees: 1, Out-of-bag evaluation: accuracy:0.964286 logloss:1.28727\n",
      "\ttrees: 13, Out-of-bag evaluation: accuracy:0.94863 logloss:1.38235\n",
      "\ttrees: 29, Out-of-bag evaluation: accuracy:0.963526 logloss:0.698239\n",
      "\ttrees: 39, Out-of-bag evaluation: accuracy:0.958824 logloss:0.37345\n",
      "\ttrees: 54, Out-of-bag evaluation: accuracy:0.973837 logloss:0.171543\n",
      "\ttrees: 72, Out-of-bag evaluation: accuracy:0.97093 logloss:0.171775\n",
      "\ttrees: 82, Out-of-bag evaluation: accuracy:0.973837 logloss:0.168111\n",
      "\ttrees: 92, Out-of-bag evaluation: accuracy:0.976744 logloss:0.167506\n",
      "\ttrees: 113, Out-of-bag evaluation: accuracy:0.976744 logloss:0.170507\n",
      "\ttrees: 124, Out-of-bag evaluation: accuracy:0.976744 logloss:0.07406\n",
      "\ttrees: 135, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0739305\n",
      "\ttrees: 145, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0741686\n",
      "\ttrees: 155, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0738562\n",
      "\ttrees: 166, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0727146\n",
      "\ttrees: 177, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0721128\n",
      "\ttrees: 195, Out-of-bag evaluation: accuracy:0.976744 logloss:0.070882\n",
      "\ttrees: 205, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0705714\n",
      "\ttrees: 216, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0697382\n",
      "\ttrees: 231, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0695581\n",
      "\ttrees: 244, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0683962\n",
      "\ttrees: 255, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0693447\n",
      "\ttrees: 267, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0689024\n",
      "\ttrees: 279, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0694214\n",
      "\ttrees: 296, Out-of-bag evaluation: accuracy:0.976744 logloss:0.0691636\n",
      "\ttrees: 300, Out-of-bag evaluation: accuracy:0.976744 logloss:0.068949\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%set_cell_height 300\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dtvAH26EfSgY"
   },
   "source": [
    "Note the multiple variable importances with name `MEAN_DECREASE_IN_*`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xTwmx8A0c4TU"
   },
   "source": [
    "## Plotting the model\n",
    "\n",
    "Next, plot the model.\n",
    "\n",
    "A Random Forest is a large model (this model has 300 trees and ~5k nodes; see the summary above). Therefore, only plot the first tree, and limit the nodes to depth 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.423361Z",
     "iopub.status.busy": "2024-04-20T11:24:54.422656Z",
     "iopub.status.idle": "2024-04-20T11:24:54.429895Z",
     "shell.execute_reply": "2024-04-20T11:24:54.429284Z"
    },
    "id": "ZRTrXDz_dIAQ"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<script src=\"https://d3js.org/d3.v6.min.js\"></script>\n",
       "<div id=\"tree_plot_5579c6bb86b247b89d8a57134acf48fe\"></div>\n",
       "<script>\n",
       "/*\n",
       " * Copyright 2021 Google LLC.\n",
       " * Licensed under the Apache License, Version 2.0 (the \"License\");\n",
       " * you may not use this file except in compliance with the License.\n",
       " * You may obtain a copy of the License at\n",
       " *\n",
       " *     https://www.apache.org/licenses/LICENSE-2.0\n",
       " *\n",
       " * Unless required by applicable law or agreed to in writing, software\n",
       " * distributed under the License is distributed on an \"AS IS\" BASIS,\n",
       " * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
       " * See the License for the specific language governing permissions and\n",
       " * limitations under the License.\n",
       " */\n",
       "\n",
       "/**\n",
       " *  Plotting of decision trees generated by TF-DF.\n",
       " *\n",
       " *  A tree is a recursive structure of node objects.\n",
       " *  A node contains one or more of the following components:\n",
       " *\n",
       " *    - A value: Representing the output of the node. If the node is not a leaf,\n",
       " *      the value is only present for analysis i.e. it is not used for\n",
       " *      predictions.\n",
       " *\n",
       " *    - A condition : For non-leaf nodes, the condition (also known as split)\n",
       " *      defines a binary test to branch to the positive or negative child.\n",
       " *\n",
       " *    - An explanation: Generally a plot showing the relation between the label\n",
       " *      and the condition to give insights about the effect of the condition.\n",
       " *\n",
       " *    - Two children : For non-leaf nodes, the children nodes. The first\n",
       " *      children (i.e. \"node.children[0]\") is the negative children (drawn in\n",
       " *      red). The second children is the positive one (drawn in green).\n",
       " *\n",
       " */\n",
       "\n",
       "/**\n",
       " * Plots a single decision tree into a DOM element.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!tree} raw_tree Recursive tree structure.\n",
       " * @param {string} canvas_id Id of the output dom element.\n",
       " */\n",
       "function display_tree(options, raw_tree, canvas_id) {\n",
       "  console.log(options);\n",
       "\n",
       "  // Determine the node placement.\n",
       "  const tree_struct = d3.tree().nodeSize(\n",
       "      [options.node_y_offset, options.node_x_offset])(d3.hierarchy(raw_tree));\n",
       "\n",
       "  // Boundaries of the node placement.\n",
       "  let x_min = Infinity;\n",
       "  let x_max = -x_min;\n",
       "  let y_min = Infinity;\n",
       "  let y_max = -x_min;\n",
       "\n",
       "  tree_struct.each(d => {\n",
       "    if (d.x > x_max) x_max = d.x;\n",
       "    if (d.x < x_min) x_min = d.x;\n",
       "    if (d.y > y_max) y_max = d.y;\n",
       "    if (d.y < y_min) y_min = d.y;\n",
       "  });\n",
       "\n",
       "  // Size of the plot.\n",
       "  const width = y_max - y_min + options.node_x_size + options.margin * 2;\n",
       "  const height = x_max - x_min + options.node_y_size + options.margin * 2 +\n",
       "      options.node_y_offset - options.node_y_size;\n",
       "\n",
       "  const plot = d3.select(canvas_id);\n",
       "\n",
       "  // Tool tip\n",
       "  options.tooltip = plot.append('div')\n",
       "                        .attr('width', 100)\n",
       "                        .attr('height', 100)\n",
       "                        .style('padding', '4px')\n",
       "                        .style('background', '#fff')\n",
       "                        .style('box-shadow', '4px 4px 0px rgba(0,0,0,0.1)')\n",
       "                        .style('border', '1px solid black')\n",
       "                        .style('font-family', 'sans-serif')\n",
       "                        .style('font-size', options.font_size)\n",
       "                        .style('position', 'absolute')\n",
       "                        .style('z-index', '10')\n",
       "                        .attr('pointer-events', 'none')\n",
       "                        .style('display', 'none');\n",
       "\n",
       "  // Create canvas\n",
       "  const svg = plot.append('svg').attr('width', width).attr('height', height);\n",
       "  const graph =\n",
       "      svg.style('overflow', 'visible')\n",
       "          .append('g')\n",
       "          .attr('font-family', 'sans-serif')\n",
       "          .attr('font-size', options.font_size)\n",
       "          .attr(\n",
       "              'transform',\n",
       "              () => `translate(${options.margin},${\n",
       "                  - x_min + options.node_y_offset / 2 + options.margin})`);\n",
       "\n",
       "  // Plot bounding box.\n",
       "  if (options.show_plot_bounding_box) {\n",
       "    svg.append('rect')\n",
       "        .attr('width', width)\n",
       "        .attr('height', height)\n",
       "        .attr('fill', 'none')\n",
       "        .attr('stroke-width', 1.0)\n",
       "        .attr('stroke', 'black');\n",
       "  }\n",
       "\n",
       "  // Draw the edges.\n",
       "  display_edges(options, graph, tree_struct);\n",
       "\n",
       "  // Draw the nodes.\n",
       "  display_nodes(options, graph, tree_struct);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Draw the nodes of the tree.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!graph} graph D3 search handle containing the graph.\n",
       " * @param {!tree_struct} tree_struct Structure of the tree (node placement,\n",
       " *     data, etc.).\n",
       " */\n",
       "function display_nodes(options, graph, tree_struct) {\n",
       "  const nodes = graph.append('g')\n",
       "                    .selectAll('g')\n",
       "                    .data(tree_struct.descendants())\n",
       "                    .join('g')\n",
       "                    .attr('transform', d => `translate(${d.y},${d.x})`);\n",
       "\n",
       "  nodes.append('rect')\n",
       "      .attr('x', 0.5)\n",
       "      .attr('y', 0.5)\n",
       "      .attr('width', options.node_x_size)\n",
       "      .attr('height', options.node_y_size)\n",
       "      .attr('stroke', 'lightgrey')\n",
       "      .attr('stroke-width', 1)\n",
       "      .attr('fill', 'white')\n",
       "      .attr('y', -options.node_y_size / 2);\n",
       "\n",
       "  // Brackets on the right of condition nodes without children.\n",
       "  non_leaf_node_without_children =\n",
       "      nodes.filter(node => node.data.condition != null && node.children == null)\n",
       "          .append('g')\n",
       "          .attr('transform', `translate(${options.node_x_size},0)`);\n",
       "\n",
       "  non_leaf_node_without_children.append('path')\n",
       "      .attr('d', 'M0,0 C 10,0 0,10 10,10')\n",
       "      .attr('fill', 'none')\n",
       "      .attr('stroke-width', 1.0)\n",
       "      .attr('stroke', '#F00');\n",
       "\n",
       "  non_leaf_node_without_children.append('path')\n",
       "      .attr('d', 'M0,0 C 10,0 0,-10 10,-10')\n",
       "      .attr('fill', 'none')\n",
       "      .attr('stroke-width', 1.0)\n",
       "      .attr('stroke', '#0F0');\n",
       "\n",
       "  const node_content = nodes.append('g').attr(\n",
       "      'transform',\n",
       "      `translate(0,${options.node_padding - options.node_y_size / 2})`);\n",
       "\n",
       "  node_content.append(node => create_node_element(options, node));\n",
       "}\n",
       "\n",
       "/**\n",
       " * Creates the D3 content for a single node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!node} node Node to draw.\n",
       " * @return {!d3} D3 content.\n",
       " */\n",
       "function create_node_element(options, node) {\n",
       "  // Output accumulator.\n",
       "  let output = {\n",
       "    // Content to draw.\n",
       "    content: d3.create('svg:g'),\n",
       "    // Vertical offset to the next element to draw.\n",
       "    vertical_offset: 0\n",
       "  };\n",
       "\n",
       "  // Conditions.\n",
       "  if (node.data.condition != null) {\n",
       "    display_condition(options, node.data.condition, output);\n",
       "  }\n",
       "\n",
       "  // Values.\n",
       "  if (node.data.value != null) {\n",
       "    display_value(options, node.data.value, output);\n",
       "  }\n",
       "\n",
       "  // Explanations.\n",
       "  if (node.data.explanation != null) {\n",
       "    display_explanation(options, node.data.explanation, output);\n",
       "  }\n",
       "\n",
       "  return output.content.node();\n",
       "}\n",
       "\n",
       "\n",
       "/**\n",
       " * Adds a single line of text inside of a node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {string} text Text to display.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_node_text(options, text, output) {\n",
       "  output.content.append('text')\n",
       "      .attr('x', options.node_padding)\n",
       "      .attr('y', output.vertical_offset)\n",
       "      .attr('alignment-baseline', 'hanging')\n",
       "      .text(text);\n",
       "  output.vertical_offset += 10;\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds a single line of text inside of a node with a tooltip.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {string} text Text to display.\n",
       " * @param {string} tooltip Text in the Tooltip.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_node_text_with_tooltip(options, text, tooltip, output) {\n",
       "  const item = output.content.append('text')\n",
       "                   .attr('x', options.node_padding)\n",
       "                   .attr('alignment-baseline', 'hanging')\n",
       "                   .text(text);\n",
       "\n",
       "  add_tooltip(options, item, () => tooltip);\n",
       "  output.vertical_offset += 10;\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds a tooltip to a dom element.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!dom} target Dom element to equip with a tooltip.\n",
       " * @param {!func} get_content Generates the html content of the tooltip.\n",
       " */\n",
       "function add_tooltip(options, target, get_content) {\n",
       "  function show(d) {\n",
       "    options.tooltip.style('display', 'block');\n",
       "    options.tooltip.html(get_content());\n",
       "  }\n",
       "\n",
       "  function hide(d) {\n",
       "    options.tooltip.style('display', 'none');\n",
       "  }\n",
       "\n",
       "  function move(d) {\n",
       "    options.tooltip.style('display', 'block');\n",
       "    options.tooltip.style('left', (d.pageX + 5) + 'px');\n",
       "    options.tooltip.style('top', d.pageY + 'px');\n",
       "  }\n",
       "\n",
       "  target.on('mouseover', show);\n",
       "  target.on('mouseout', hide);\n",
       "  target.on('mousemove', move);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds a condition inside of a node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!condition} condition Condition to display.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_condition(options, condition, output) {\n",
       "  threshold_format = d3.format('r');\n",
       "\n",
       "  if (condition.type === 'IS_MISSING') {\n",
       "    display_node_text(options, `${condition.attribute} is missing`, output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'IS_TRUE') {\n",
       "    display_node_text(options, `${condition.attribute} is true`, output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'NUMERICAL_IS_HIGHER_THAN') {\n",
       "    format = d3.format('r');\n",
       "    display_node_text(\n",
       "        options,\n",
       "        `${condition.attribute} >= ${threshold_format(condition.threshold)}`,\n",
       "        output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'CATEGORICAL_IS_IN') {\n",
       "    display_node_text_with_tooltip(\n",
       "        options, `${condition.attribute} in [...]`,\n",
       "        `${condition.attribute} in [${condition.mask}]`, output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'CATEGORICAL_SET_CONTAINS') {\n",
       "    display_node_text_with_tooltip(\n",
       "        options, `${condition.attribute} intersect [...]`,\n",
       "        `${condition.attribute} intersect [${condition.mask}]`, output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'NUMERICAL_SPARSE_OBLIQUE') {\n",
       "    display_node_text_with_tooltip(\n",
       "        options, `Sparse oblique split...`,\n",
       "        `[${condition.attributes}]*[${condition.weights}]>=${\n",
       "            threshold_format(condition.threshold)}`,\n",
       "        output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  display_node_text(\n",
       "      options, `Non supported condition ${condition.type}`, output);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds a value inside of a node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!value} value Value to display.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_value(options, value, output) {\n",
       "  if (value.type === 'PROBABILITY') {\n",
       "    const left_margin = 0;\n",
       "    const right_margin = 50;\n",
       "    const plot_width = options.node_x_size - options.node_padding * 2 -\n",
       "        left_margin - right_margin;\n",
       "\n",
       "    let cusum = Array.from(d3.cumsum(value.distribution));\n",
       "    cusum.unshift(0);\n",
       "    const distribution_plot = output.content.append('g').attr(\n",
       "        'transform', `translate(0,${output.vertical_offset + 0.5})`);\n",
       "\n",
       "    distribution_plot.selectAll('rect')\n",
       "        .data(value.distribution)\n",
       "        .join('rect')\n",
       "        .attr('height', 10)\n",
       "        .attr(\n",
       "            'x',\n",
       "            (d, i) =>\n",
       "                (cusum[i] * plot_width + left_margin + options.node_padding))\n",
       "        .attr('width', (d, i) => d * plot_width)\n",
       "        .style('fill', (d, i) => d3.schemeSet1[i]);\n",
       "\n",
       "    const num_examples =\n",
       "        output.content.append('g')\n",
       "            .attr('transform', `translate(0,${output.vertical_offset})`)\n",
       "            .append('text')\n",
       "            .attr('x', options.node_x_size - options.node_padding)\n",
       "            .attr('alignment-baseline', 'hanging')\n",
       "            .attr('text-anchor', 'end')\n",
       "            .text(`(${value.num_examples})`);\n",
       "\n",
       "    const distribution_details = d3.create('ul');\n",
       "    distribution_details.selectAll('li')\n",
       "        .data(value.distribution)\n",
       "        .join('li')\n",
       "        .append('span')\n",
       "        .text(\n",
       "            (d, i) =>\n",
       "                'class ' + i + ': ' + d3.format('.3%')(value.distribution[i]));\n",
       "\n",
       "    add_tooltip(options, distribution_plot, () => distribution_details.html());\n",
       "    add_tooltip(options, num_examples, () => 'Number of examples');\n",
       "\n",
       "    output.vertical_offset += 10;\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (value.type === 'REGRESSION') {\n",
       "    display_node_text(\n",
       "        options,\n",
       "        'value: ' + d3.format('r')(value.value) + ` (` +\n",
       "            d3.format('.6')(value.num_examples) + `)`,\n",
       "        output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (value.type === 'UPLIFT') {\n",
       "    display_node_text(\n",
       "        options,\n",
       "        'effect: ' + d3.format('r')(value.treatment_effect) + ` (` +\n",
       "            d3.format('.6')(value.num_examples) + `)`,\n",
       "        output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  display_node_text(options, `Non supported value ${value.type}`, output);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds an explanation inside of a node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!explanation} explanation Explanation to display.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_explanation(options, explanation, output) {\n",
       "  // Margin before the explanation.\n",
       "  output.vertical_offset += 10;\n",
       "\n",
       "  display_node_text(\n",
       "      options, `Non supported explanation ${explanation.type}`, output);\n",
       "}\n",
       "\n",
       "\n",
       "/**\n",
       " * Draw the edges of the tree.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!graph} graph D3 search handle containing the graph.\n",
       " * @param {!tree_struct} tree_struct Structure of the tree (node placement,\n",
       " *     data, etc.).\n",
       " */\n",
       "function display_edges(options, graph, tree_struct) {\n",
       "  // Draw an edge between a parent and a child node with a bezier.\n",
       "  function draw_single_edge(d) {\n",
       "    return 'M' + (d.source.y + options.node_x_size) + ',' + d.source.x + ' C' +\n",
       "        (d.source.y + options.node_x_size + options.edge_rounding) + ',' +\n",
       "        d.source.x + ' ' + (d.target.y - options.edge_rounding) + ',' +\n",
       "        d.target.x + ' ' + d.target.y + ',' + d.target.x;\n",
       "  }\n",
       "\n",
       "  graph.append('g')\n",
       "      .attr('fill', 'none')\n",
       "      .attr('stroke-width', 1.2)\n",
       "      .selectAll('path')\n",
       "      .data(tree_struct.links())\n",
       "      .join('path')\n",
       "      .attr('d', draw_single_edge)\n",
       "      .attr(\n",
       "          'stroke', d => (d.target === d.source.children[0]) ? '#0F0' : '#F00');\n",
       "}\n",
       "\n",
       "display_tree({\"margin\": 10, \"node_x_size\": 160, \"node_y_size\": 28, \"node_x_offset\": 180, \"node_y_offset\": 33, \"font_size\": 10, \"edge_rounding\": 20, \"node_padding\": 2, \"show_plot_bounding_box\": false}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.47093023255813954, 0.19476744186046513, 0.33430232558139533], \"num_examples\": 344.0}, \"condition\": {\"type\": \"NUMERICAL_IS_HIGHER_THAN\", \"attribute\": \"bill_length_mm\", \"threshold\": 43.25}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.005847953216374269, 0.3567251461988304, 0.6374269005847953], \"num_examples\": 171.0}, \"condition\": {\"type\": \"CATEGORICAL_IS_IN\", \"attribute\": \"island\", \"mask\": [\"Biscoe\"]}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.00909090909090909, 0.0, 0.990909090909091], \"num_examples\": 110.0}, \"condition\": {\"type\": \"NUMERICAL_IS_HIGHER_THAN\", \"attribute\": \"bill_depth_mm\", \"threshold\": 17.225584030151367}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.16666666666666666, 0.0, 0.8333333333333334], \"num_examples\": 6.0}}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.0, 0.0, 1.0], \"num_examples\": 104.0}}]}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.0, 1.0, 0.0], \"num_examples\": 61.0}}]}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.930635838150289, 0.03468208092485549, 0.03468208092485549], \"num_examples\": 173.0}, \"condition\": {\"type\": \"NUMERICAL_IS_HIGHER_THAN\", \"attribute\": \"bill_depth_mm\", \"threshold\": 15.100000381469727}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.9640718562874252, 0.03592814371257485, 0.0], \"num_examples\": 167.0}, \"condition\": {\"type\": \"NUMERICAL_IS_HIGHER_THAN\", \"attribute\": \"flipper_length_mm\", \"threshold\": 187.5}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [1.0, 0.0, 0.0], \"num_examples\": 104.0}}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.9047619047619048, 0.09523809523809523, 0.0], \"num_examples\": 63.0}, \"condition\": {\"type\": \"NUMERICAL_IS_HIGHER_THAN\", \"attribute\": \"bill_length_mm\", \"threshold\": 42.30000305175781}}]}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.0, 0.0, 1.0], \"num_examples\": 6.0}}]}]}, \"#tree_plot_5579c6bb86b247b89d8a57134acf48fe\")\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfdf.model_plotter.plot_model_in_colab(model, tree_idx=0, max_depth=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lOlieoz2c-GA"
   },
   "source": [
    "## Inspect the model structure\n",
    "\n",
    "The model structure and meta-data is\n",
    "available through the **inspector** created by `make_inspector()`.\n",
    "\n",
    "**Note:** Depending on the learning algorithm and hyper-parameters, the\n",
    "inspector will expose different specialized attributes. For examples, the\n",
    "`winner_take_all` field is specific to Random Forest models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.433296Z",
     "iopub.status.busy": "2024-04-20T11:24:54.432683Z",
     "iopub.status.idle": "2024-04-20T11:24:54.436951Z",
     "shell.execute_reply": "2024-04-20T11:24:54.436318Z"
    },
    "id": "KHc8IcW1c8ER"
   },
   "outputs": [],
   "source": [
    "inspector = model.make_inspector()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RDdUhqaSsNnQ"
   },
   "source": [
    "For our model, the available inspector fields are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.440561Z",
     "iopub.status.busy": "2024-04-20T11:24:54.439979Z",
     "iopub.status.idle": "2024-04-20T11:24:54.444995Z",
     "shell.execute_reply": "2024-04-20T11:24:54.444350Z"
    },
    "id": "jx54DFjRsA7k"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['MODEL_NAME',\n",
       " 'dataspec',\n",
       " 'directory',\n",
       " 'evaluation',\n",
       " 'export_to_tensorboard',\n",
       " 'extract_all_trees',\n",
       " 'extract_tree',\n",
       " 'features',\n",
       " 'file_prefix',\n",
       " 'header',\n",
       " 'iterate_on_nodes',\n",
       " 'label',\n",
       " 'label_classes',\n",
       " 'metadata',\n",
       " 'model_type',\n",
       " 'num_trees',\n",
       " 'objective',\n",
       " 'specialized_header',\n",
       " 'task',\n",
       " 'training_logs',\n",
       " 'tuning_logs',\n",
       " 'variable_importances',\n",
       " 'winner_take_all_inference']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[field for field in dir(inspector) if not field.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_QJFITMQsgtK"
   },
   "source": [
    "Remember to see [the API-reference](https://tensorflow.org/decision_forests/api_docs/python/tfdf/inspector/AbstractInspector) or use `?` for the builtin documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.448197Z",
     "iopub.status.busy": "2024-04-20T11:24:54.447657Z",
     "iopub.status.idle": "2024-04-20T11:24:54.487393Z",
     "shell.execute_reply": "2024-04-20T11:24:54.486705Z"
    },
    "id": "YCGkpRkssdCb"
   },
   "outputs": [],
   "source": [
    "?inspector.model_type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nd-fOgmjd1oK"
   },
   "source": [
    "Some of the model meta-data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.490987Z",
     "iopub.status.busy": "2024-04-20T11:24:54.490551Z",
     "iopub.status.idle": "2024-04-20T11:24:54.494787Z",
     "shell.execute_reply": "2024-04-20T11:24:54.494175Z"
    },
    "id": "Iu_To_z9d35G"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model type: RANDOM_FOREST\n",
      "Number of trees: 300\n",
      "Objective: Classification(label=__LABEL, class=None, num_classes=3)\n",
      "Input features: [\"bill_depth_mm\" (1; #1), \"bill_length_mm\" (1; #2), \"body_mass_g\" (1; #3), \"flipper_length_mm\" (1; #4), \"island\" (4; #5), \"sex\" (4; #6), \"year\" (1; #7)]\n"
     ]
    }
   ],
   "source": [
    "print(\"Model type:\", inspector.model_type())\n",
    "print(\"Number of trees:\", inspector.num_trees())\n",
    "print(\"Objective:\", inspector.objective())\n",
    "print(\"Input features:\", inspector.features())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zs7b8EBud9JM"
   },
   "source": [
    "`evaluate()` is the evaluation of the model computed during training. The dataset used for this evaluation depends on the algorithm. For example, it can be the validation dataset or the out-of-bag-dataset .\n",
    "\n",
    "**Note:** While computed during training, `evaluate()` is never an evaluation on the\n",
    "training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.498031Z",
     "iopub.status.busy": "2024-04-20T11:24:54.497556Z",
     "iopub.status.idle": "2024-04-20T11:24:54.501558Z",
     "shell.execute_reply": "2024-04-20T11:24:54.500992Z"
    },
    "id": "uVN-j0E4Q1T3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Evaluation(num_examples=344, accuracy=0.9767441860465116, loss=0.06894904488784283, rmse=None, ndcg=None, aucs=None, auuc=None, qini=None)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inspector.evaluation()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2r6Yrjb7f5KH"
   },
   "source": [
    "The variable importances are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.504764Z",
     "iopub.status.busy": "2024-04-20T11:24:54.504381Z",
     "iopub.status.idle": "2024-04-20T11:24:54.508511Z",
     "shell.execute_reply": "2024-04-20T11:24:54.507907Z"
    },
    "id": "qoqhhmGjf7ED"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available variable importances:\n",
      "\t MEAN_DECREASE_IN_PRAUC_3_VS_OTHERS\n",
      "\t MEAN_DECREASE_IN_PRAUC_1_VS_OTHERS\n",
      "\t INV_MEAN_MIN_DEPTH\n",
      "\t MEAN_DECREASE_IN_AUC_1_VS_OTHERS\n",
      "\t MEAN_DECREASE_IN_AP_2_VS_OTHERS\n",
      "\t MEAN_DECREASE_IN_AUC_3_VS_OTHERS\n",
      "\t MEAN_DECREASE_IN_AUC_2_VS_OTHERS\n",
      "\t MEAN_DECREASE_IN_AP_1_VS_OTHERS\n",
      "\t NUM_AS_ROOT\n",
      "\t NUM_NODES\n",
      "\t MEAN_DECREASE_IN_PRAUC_2_VS_OTHERS\n",
      "\t MEAN_DECREASE_IN_ACCURACY\n",
      "\t SUM_SCORE\n",
      "\t MEAN_DECREASE_IN_AP_3_VS_OTHERS\n"
     ]
    }
   ],
   "source": [
    "print(f\"Available variable importances:\")\n",
    "for importance in inspector.variable_importances().keys():\n",
    "  print(\"\\t\", importance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8QUW8w-UmCoW"
   },
   "source": [
    "Different variable importances have different semantics. For example, a feature\n",
    "with a **mean decrease in auc** of `0.05` means that removing this feature from\n",
    "the training dataset would reduce/hurt the AUC by 5%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.511809Z",
     "iopub.status.busy": "2024-04-20T11:24:54.511264Z",
     "iopub.status.idle": "2024-04-20T11:24:54.516008Z",
     "shell.execute_reply": "2024-04-20T11:24:54.515415Z"
    },
    "id": "OoSG5T8ShSdG"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(\"bill_length_mm\" (1; #2), 0.0713061951754389),\n",
       " (\"island\" (4; #5), 0.007298519736842035),\n",
       " (\"flipper_length_mm\" (1; #4), 0.004505893640351366),\n",
       " (\"bill_depth_mm\" (1; #1), 0.0021244517543865804),\n",
       " (\"body_mass_g\" (1; #3), 0.0005482456140351033),\n",
       " (\"sex\" (4; #6), 0.00047971491228060437),\n",
       " (\"year\" (1; #7), 0.0)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Mean decrease in AUC of the class 1 vs the others.\n",
    "inspector.variable_importances()[\"MEAN_DECREASE_IN_AUC_1_VS_OTHERS\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "afSPSBg_uJuI"
   },
   "source": [
    "Plot the variable importances from the inspector using Matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.519423Z",
     "iopub.status.busy": "2024-04-20T11:24:54.518818Z",
     "iopub.status.idle": "2024-04-20T11:24:54.779885Z",
     "shell.execute_reply": "2024-04-20T11:24:54.778963Z"
    },
    "id": "53lM0wDmuI8u"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABKIAAAGGCAYAAABIYSkNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAACDJ0lEQVR4nOzdd3yN5//H8fdJIkMmQoaGBKkYIRpEbF9pQ63YVGsF1VqpqtHWKi2lSo1WUaul1Kj2q0bNDrNGtBTFV4QSW2KP5P794ZHzcyQiSE6M1/PxuB9x7vtzX9fnus+VUz697vuYDMMwBAAAAAAAAGQzm5xOAAAAAAAAAM8GClEAAAAAAACwCgpRAAAAAAAAsAoKUQAAAAAAALAKClEAAAAAAACwCgpRAAAAAAAAsAoKUQAAAAAAALAKClEAAAAAAACwCgpRAAAAAAAAsAoKUQAAPINMJpOGDBmS02k8Efz9/dW+ffucTuOxdOnSJXXq1Ene3t4ymUyKiYl54DaGDBkik8mkM2fOZH2CD2j9+vUymUxav359TqfyVGjfvr1cXFxyOo1MMZlM6t69e06nAQDPBApRAADcZebMmTKZTDKZTPr999/THDcMQ35+fjKZTKpfv34OZIinRd++fWUymdSyZct0j6cWRhYuXJju8e7du8tkMqXZn5ycrBkzZqhmzZrKmzevHBwc5O/vrw4dOmjbtm1Zlv9HH32kmTNn6o033tDXX3+t1157LcPYJUuWZFnfz5KtW7fqzTffVGhoqHLlypXue55Trly5oiFDhjwRxbuNGzdqyJAhunDhQk6nAgDPNApRAADcg6Ojo+bOnZtm/y+//KJjx47JwcEhB7KCte3fv19Tp07N8nYNw9C3334rf39//fe//9XFixezpN2rV6+qfv366tixowzD0LvvvqsvvvhCbdu21aZNm1SxYkUdO3YsS/pau3atKlWqpMGDB+vVV19VaGjoPWMpRD28ZcuWadq0aTKZTCpSpEhOp2PhypUrGjp06BNTiBo6dCiFKADIYRSiAAC4h5dfflkLFizQrVu3LPbPnTtXoaGh8vb2zqHMnjy3bt3SjRs3cjqNh+Lg4KBcuXJlebvr16/XsWPHNH36dN26dUuLFy/OknbfeecdrVixQmPHjtUvv/yiPn36qGPHjvrggw+0Z88ejRo1Kkv6kaRTp07Jw8Mjy9pD+t544w0lJiZq27ZtevHFF3M6HTyka9euKSUlJafTAIAcRyEKAIB7aN26tc6ePatVq1aZ9924cUMLFy7UK6+8ku45KSkpGjdunEqVKiVHR0d5eXnp9ddf1/nz5y3ifvjhB9WrV0++vr5ycHBQ0aJFNWzYMCUnJ1vE1axZU6VLl9bff/+tWrVqKXfu3CpYsGCmiwnXr1/XW2+9pfz588vV1VUNGza852qYf//9Vx07dpSXl5ccHBxUqlQpTZ8+PU3ctWvXNGTIED3//PNydHSUj4+PmjRpokOHDkmS4uLiZDKZ9Mknn2jcuHEqWrSoHBwc9Pfff0uS9u3bp2bNmilv3rxydHRU+fLl9eOPP1r0ce7cOfXp00fBwcFycXGRm5ub6tatq127dqXJZ8KECSpVqpRy586tPHnyqHz58mlWsmV2bOm5+xlRqbdubtiwQb1791b+/Pnl7Oysxo0b6/Tp05lqU5LmzJmjkiVLqlatWoqIiNCcOXMyfe69HDt2TF9++aVefPHFdJ/XZGtrqz59+ui5557LsJ1Tp04pOjpaXl5ecnR0VNmyZTVr1izz8dRbBg8fPqyffvrJfCtrXFxcuu2ZTCZdvnxZs2bNMsfe/dytCxcuqH379vLw8JC7u7s6dOigK1eupGnrm2++UWhoqJycnJQ3b161atVKR48eve+1kW7Pg+joaPPvXUBAgN54440Mi6S//fabmjdvrkKFCsnBwUF+fn566623dPXqVYu4hIQEdejQQc8995wcHBzk4+OjRo0aWVyTbdu2KTIyUp6ennJyclJAQIA6dux437y9vLzk5OSUqTHerXTp0qpVq1aa/SkpKSpYsKCaNWtm3jdv3jyFhobK1dVVbm5uCg4O1meffXbPtuPi4pQ/f35J0tChQ83v7d3Pn/v3338VFRUlFxcX5c+fX3369EnzWZfZz857Wbt2rapVqyZnZ2d5eHioUaNG2rt3r/n4kCFD9M4770iSAgIC7jlnlyxZotKlS5s/J1asWJGmr8x8nqT+jsybN0/vv/++ChYsqNy5cyspKUk3b97U0KFDFRgYKEdHR+XLl09Vq1a1+G8NADzN7HI6AQAAHlf+/v4KDw/Xt99+q7p160qSli9frsTERLVq1Urjx49Pc87rr7+umTNnqkOHDurZs6cOHz6siRMnaufOndqwYYN5Zc3MmTPl4uKi3r17y8XFRWvXrtWgQYOUlJSk0aNHW7R5/vx51alTR02aNFGLFi20cOFC9evXT8HBwea87qVTp0765ptv9Morr6hy5cpau3at6tWrlybu5MmTqlSpkvmBvfnz59fy5csVHR2tpKQkc1EjOTlZ9evX15o1a9SqVSv16tVLFy9e1KpVq7R7924VLVrU3OaMGTN07do1denSRQ4ODsqbN6/27NmjKlWqqGDBgurfv7+cnZ313XffKSoqSosWLVLjxo0lSf/73/+0ZMkSNW/eXAEBATp58qS+/PJL1ahRQ3///bd8fX0lSVOnTlXPnj3VrFkz9erVS9euXdOff/6pLVu2mIuFmR3bg+rRo4fy5MmjwYMHKy4uTuPGjVP37t01f/78+557/fp1LVq0SG+//bak20XPDh06KCEh4ZFW2i1fvly3bt3K8FlN93P16lXVrFlTBw8eVPfu3RUQEKAFCxaoffv2unDhgnr16qUSJUro66+/1ltvvaXnnnvOPI7UosTdvv76a3Xq1EkVK1ZUly5dJMlirkhSixYtFBAQoBEjRmjHjh2aNm2aChQooI8//tgc8+GHH2rgwIFq0aKFOnXqpNOnT2vChAmqXr26du7cmeHqrOPHj6tixYq6cOGCunTpoqCgIP37779auHChrly5Int7+3TPW7Bgga5cuaI33nhD+fLl09atWzVhwgQdO3ZMCxYsMMc1bdpUe/bsUY8ePeTv769Tp05p1apVio+PN79+6aWXlD9/fvXv318eHh6Ki4vLspVw99KyZUsNGTIkzdz6/fffdfz4cbVq1UqStGrVKrVu3Vq1a9c2X/O9e/dqw4YN6tWrV7pt58+fX1988YXeeOMNNW7cWE2aNJEklSlTxhyTnJysyMhIhYWF6ZNPPtHq1as1ZswYFS1aVG+88YY5LrOfnelZvXq16tatqyJFimjIkCG6evWqJkyYoCpVqmjHjh3y9/dXkyZN9M8//+jbb7/V2LFj5enpaR7Dnddk8eLFevPNN+Xq6qrx48eradOmio+PV758+SQ9+OfJsGHDZG9vrz59+uj69euyt7fXkCFDNGLECPPvRFJSkrZt26YdO3aw4g3As8EAAAAWZsyYYUgy/vjjD2PixImGq6urceXKFcMwDKN58+ZGrVq1DMMwjMKFCxv16tUzn/fbb78Zkow5c+ZYtLdixYo0+1Pbu9Prr79u5M6d27h27Zp5X40aNQxJxuzZs837rl+/bnh7extNmzbNcByxsbGGJOPNN9+02P/KK68YkozBgweb90VHRxs+Pj7GmTNnLGJbtWpluLu7m/OdPn26Icn49NNP0/SXkpJiGIZhHD582JBkuLm5GadOnbKIqV27thEcHGwxxpSUFKNy5cpGYGCged+1a9eM5ORki3MPHz5sODg4GB988IF5X6NGjYxSpUpleB0yO7Z7KVy4sNGuXTvz69T5ERERYR6zYRjGW2+9Zdja2hoXLlzIsD3DMIyFCxcakowDBw4YhmEYSUlJhqOjozF27FiLuHXr1hmSjAULFqTbTrdu3Yw7/zr31ltvGZKMnTt33jeHexk3bpwhyfjmm2/M+27cuGGEh4cbLi4uRlJSknn/3b8DGXF2dra4jqkGDx5sSDI6duxosb9x48ZGvnz5zK/j4uIMW1tb48MPP7SI++uvvww7O7s0++/Wtm1bw8bGxvjjjz/SHEt9H1Ov97p168zH0psfI0aMMEwmk3HkyBHDMAzj/PnzhiRj9OjR9+z/+++/N3+uPIq73/P72b9/vyHJmDBhgsX+N99803BxcTGPr1evXoabm5tx69atB8rn9OnTaT5PUrVr186QZPE7axiGUa5cOSM0NNT8+kE+O9MTEhJiFChQwDh79qx5365duwwbGxujbdu25n2jR482JBmHDx9O04Ykw97e3jh48KBFG3dfu8x+nqTOpSJFiqSZQ2XLls307w0API24NQ8AgAy0aNFCV69e1dKlS3Xx4kUtXbr0nrflLViwQO7u7nrxxRd15swZ8xYaGioXFxetW7fOHHvnbTYXL17UmTNnVK1aNV25ckX79u2zaNfFxUWvvvqq+bW9vb0qVqyo//3vfxnmvmzZMklSz549Lfbf/X/sDcPQokWL1KBBAxmGYZF7ZGSkEhMTtWPHDknSokWL5OnpqR49eqTp7+5v8mratKnFaoNz585p7dq1atGihXnMZ86c0dmzZxUZGakDBw7o33//lXT7uUw2Nrf/mpKcnKyzZ8/KxcVFxYsXN+ciSR4eHjp27Jj++OOPdK/Bg4ztQXXp0sVizNWqVVNycrKOHDly33PnzJmj8uXLq1ixYpIkV1dX1atX75Fvz0tKSjK397CWLVsmb29vtW7d2rwvV65c6tmzpy5duqRffvnlkXK8l65du1q8rlatms6ePWse0+LFi5WSkqIWLVpYvI/e3t4KDAy0+P26W0pKipYsWaIGDRqofPnyaY5n9C10d/6uXr58WWfOnFHlypVlGIZ27txpjrG3t9f69evveStZ6mqtpUuX6ubNm/fsL6s9//zzCgkJsVipl5ycrIULF6pBgwbm8Xl4eOjy5cvZcntYeu/tnZ9fD/LZebcTJ04oNjZW7du3V968ec37y5QpoxdffNH8OZgZERERFiv1ypQpIzc3N3OuD/N50q5duzS3VXp4eGjPnj06cOBApnMDgKcJhSgAADKQP39+RUREaO7cuVq8eLGSk5MtnqlypwMHDigxMVEFChRQ/vz5LbZLly7p1KlT5tg9e/aocePGcnd3l5ubm/Lnz28uNiUmJlq0+9xzz6X5h3KePHnu++yUI0eOyMbGJs0tUMWLF7d4ffr0aV24cEFTpkxJk3eHDh0kyZz7oUOHVLx4cdnZ3f/u/oCAAIvXBw8elGEYGjhwYJp+Bg8ebNFPSkqKxo4dq8DAQDk4OMjT01P58+fXn3/+aXF9+vXrJxcXF1WsWFGBgYHq1q2bNmzY8FBje1CFChWyeJ0nTx5Juu/7cuHCBS1btkw1atTQwYMHzVuVKlW0bds2/fPPPw+VjyS5ublJ0iN9A9+RI0cUGBhoLgSmKlGihPl4drjf9Txw4IAMw1BgYGCa93Lv3r0Zvo+nT59WUlKSSpcu/cB5xcfHm4scqc84qlGjhqT//111cHDQxx9/rOXLl8vLy0vVq1fXqFGjlJCQYG6nRo0aatq0qYYOHSpPT081atRIM2bM0PXr1x84pwfVsmVLbdiwwVzoXb9+vU6dOqWWLVuaY9588009//zzqlu3rp577jl17Ngx3ecjPShHR8c0t2ze/fn1IJ+dd0udj3d/rkm35+yZM2d0+fLlTOV69xy8O9eH+Ty5+3NQkj744ANduHBBzz//vIKDg/XOO+/ozz//zFSOAPA04BlRAADcxyuvvKLOnTsrISFBdevWvedzaFJSUlSgQIF7rmpJ/cfYhQsXVKNGDbm5uemDDz5Q0aJF5ejoqB07dqhfv35pvlXJ1tY23fYMw3j4Qd2VtyS9+uqrateuXboxdz7zJbPuXgWQ2k+fPn0UGRmZ7jmpK4Q++ugjDRw4UB07dtSwYcOUN29e2djYKCYmxuL6lChRQvv379fSpUu1YsUKLVq0SJ9//rkGDRqkoUOHZtvYpId/XxYsWKDr169rzJgxGjNmTJrjc+bM0dChQyXd/ke8pDQPxk515coVc4wkBQUFSZL++usvhYSE3HcMj5P7Xc+UlBSZTCYtX7483VgXF5cszyk5OVkvvviizp07p379+ikoKEjOzs76999/1b59e4u5GBMTowYNGmjJkiVauXKlBg4cqBEjRmjt2rUqV66cTCaTFi5cqM2bN+u///2vVq5cqY4dO2rMmDHavHlztuSfqmXLlhowYIAWLFigmJgYfffdd3J3d1edOnXMMQUKFFBsbKxWrlyp5cuXa/ny5ZoxY4batm1r8aD6B3Wv9/VOmf3szG6ZmYPSg32epPeQ+erVq+vQoUP64Ycf9PPPP2vatGkaO3asJk+erE6dOj3KEADgiUAhCgCA+2jcuLFef/11bd68OcMHURctWlSrV69WlSpVMvyGq/Xr1+vs2bNavHixqlevbt5/+PDhLM27cOHCSklJMa9iSrV//36LuNRv1EtOTlZERESGbRYtWlRbtmzRzZs3M3x4cHqKFCki6fZtXvfrZ+HChapVq5a++uori/0XLlwwP2Q4lbOzs1q2bKmWLVvqxo0batKkiT788EMNGDDggcZmLXPmzFHp0qXNq8Du9OWXX2ru3LnmQlThwoUlpX3PUu3fv98cI0l169aVra2tvvnmm4d+YHnhwoX1559/KiUlxWJVVOoto3f29yAyuv0tM4oWLSrDMBQQEKDnn3/+gc7Nnz+/3NzctHv37gc676+//tI///yjWbNmqW3btub997p9rWjRonr77bf19ttv68CBAwoJCdGYMWP0zTffmGMqVaqkSpUq6cMPP9TcuXPVpk0bzZs3L1sLEAEBAapYsaLmz5+v7t27a/HixYqKipKDg4NFnL29vRo0aKAGDRooJSVFb775pr788ksNHDjQXCS+26O+r1LmPzvTk9HvyL59++Tp6SlnZ+csyTUrP0/y5s2rDh06qEOHDrp06ZKqV6+uIUOGUIgC8Ezg1jwAAO7DxcVFX3zxhYYMGaIGDRrcM65FixZKTk7WsGHD0hy7deuWLly4IOn//6/7nStnbty4oc8//zxL8079Rr27v91v3LhxFq9tbW3VtGlTLVq0KN1/qJ8+fdr856ZNm+rMmTOaOHFimrj7rQQqUKCAatasqS+//FInTpzIsB9bW9s07S1YsMB8a1Gqs2fPWry2t7dXyZIlZRiGbt68+UBjs4ajR4/q119/VYsWLdSsWbM0W4cOHXTw4EFt2bJFkuTj46OQkBB988035vmTavv27dq8ebPFNyf6+fmpc+fO+vnnnzVhwoQ0/aekpGjMmDE6duzYPXN8+eWXlZCQYFF0vXXrliZMmCAXFxfzbWkPytnZOc0YHkSTJk1ka2uroUOHppkbhmGkmQt3srGxUVRUlP773/9q27ZtaY7fa+6m97tqGIY+++wzi7grV67o2rVrFvuKFi0qV1dX861358+fT9NP6qo1a92et3nzZk2fPl1nzpyxuC1PSvu7ZGNjY17dk1F+uXPnlqRHem8z+9mZntTfkVmzZlnE7d69Wz///LNefvll877UgtTD5ppVnyd3X2sXFxcVK1bMKvMAAB4HrIgCACAT7nUbxp1q1Kih119/XSNGjFBsbKxeeukl5cqVSwcOHNCCBQv02WefqVmzZqpcubLy5Mmjdu3aqWfPnjKZTPr666+z7Fa7VCEhIWrdurU+//xzJSYmqnLlylqzZo0OHjyYJnbkyJFat26dwsLC1LlzZ5UsWVLnzp3Tjh07tHr1ap07d06S1LZtW82ePVu9e/fW1q1bVa1aNV2+fFmrV6/Wm2++qUaNGmWY06RJk1S1alUFBwerc+fOKlKkiE6ePKlNmzbp2LFj2rVrlySpfv36+uCDD9ShQwdVrlxZf/31l+bMmWNeVZXqpZdekre3t6pUqSIvLy/t3btXEydOVL169cwP7M7s2Kxh7ty5MgxDDRs2TPf4yy+/LDs7O82ZM0dhYWGSpE8//VSRkZEKCQlR+/bt5evrq71792rKlCny8fHRgAEDLNoYM2aMDh06pJ49e2rx4sWqX7++8uTJo/j4eC1YsED79u1Tq1at7pljly5d9OWXX6p9+/bavn27/P39tXDhQm3YsEHjxo176Aehh4aGavXq1fr000/l6+urgIAA8xgzo2jRoho+fLgGDBiguLg4RUVFydXVVYcPH9b333+vLl26qE+fPvc8/6OPPtLPP/+sGjVqqEuXLipRooROnDihBQsW6Pfff0/3ltugoCAVLVpUffr00b///is3NzctWrQozXPA/vnnH9WuXVstWrRQyZIlZWdnp++//14nT540X+tZs2bp888/V+PGjVW0aFFdvHhRU6dOlZubm0WxJD1HjhzR119/LUnmQtrw4cMl3V4RlJnVby1atFCfPn3Up08f5c2bN82Knk6dOuncuXP6z3/+o+eee05HjhzRhAkTFBISYn4+WHqcnJxUsmRJzZ8/X88//7zy5s2r0qVLP9DzuDL72Xkvo0ePVt26dRUeHq7o6GhdvXpVEyZMkLu7u4YMGWKOCw0NlSS99957atWqlXLlyqUGDRqYC1SZkRWfJyVLllTNmjUVGhqqvHnzatu2bVq4cKG6d++e6TwA4Ilmra/nAwDgSTFjxoxMfc36vb66fsqUKUZoaKjh5ORkuLq6GsHBwUbfvn2N48ePm2M2bNhgVKpUyXBycjJ8fX2Nvn37GitXrkzz1fE1atQwSpUqlaaPdu3aGYULF77vWK5evWr07NnTyJcvn+Hs7Gw0aNDAOHr0aLpft37y5EmjW7duhp+fn5ErVy7D29vbqF27tjFlyhSLuCtXrhjvvfeeERAQYI5r1qyZcejQIcMwDOPw4cMZfpX9oUOHjLZt2xre3t5Grly5jIIFCxr169c3Fi5caI65du2a8fbbbxs+Pj6Gk5OTUaVKFWPTpk1GjRo1jBo1apjjvvzyS6N69epGvnz5DAcHB6No0aLGO++8YyQmJj7U2NJTuHBho127dubX95ofqV/Xfuf7d7fg4GCjUKFCGfZXs2ZNo0CBAsbNmzfN+zZv3mzUr1/fyJMnj2FnZ2cULFjQ6NSpk3Hs2LF027h165Yxbdo0o1q1aoa7u7uRK1cuo3DhwkaHDh2MnTt33nfMJ0+eNDp06GB4enoa9vb2RnBwsDFjxow0cff6HUjPvn37jOrVqxtOTk6GJPM1HTx4sCHJOH36tEV86nU+fPiwxf5FixYZVatWNZydnQ1nZ2cjKCjI6Natm7F///775nDkyBGjbdu2Rv78+Q0HBwejSJEiRrdu3Yzr168bhpH+e/j3338bERERhouLi+Hp6Wl07tzZ2LVrlyHJfE3OnDljdOvWzQgKCjKcnZ0Nd3d3IywszPjuu+/M7ezYscNo3bq1UahQIcPBwcEoUKCAUb9+fWPbtm33zTs1r/S2O38f7qdKlSqGJKNTp05pji1cuNB46aWXjAIFChj29vZGoUKFjNdff904ceLEfdvduHGjERoaatjb21t8trRr185wdnZOE5/6nt8tM5+d97J69WqjSpUqhpOTk+Hm5mY0aNDA+Pvvv9PEDRs2zChYsKBhY2NjMb8kGd26dUsTf/fvv2Fk7vMk9T1bsGBBmjaHDx9uVKxY0fDw8DCcnJyMoKAg48MPPzRu3Lhx33ECwNPAZBhZ/L9fAQAAAAAAgHTwjCgAAAAAAABYBYUoAAAAAAAAWAWFKAAAAAAAAFgFhSgAAAAAAABYBYUoAAAAAAAAWAWFKAAAAAAAAFiFXU4ngKdPSkqKjh8/LldXV5lMppxOBwAAAAAAZCPDMHTx4kX5+vrKxibjNU8UopDljh8/Lj8/v5xOAwAAAAAAWNHRo0f13HPPZRhDIQpZztXVVdLtCejm5pbD2QAAAAAAgOyUlJQkPz8/cz0gIxSikOVSb8dzc3OjEAUAAAAAwDMiM4/n4WHlAAAAAAAAsAoKUQAAAAAAALAKClEAAAAAAACwCgpRAAAAAAAAsAoKUQAAAAAAALAKClEAAAAAAACwCgpRAAAAAAAAsAoKUQAAAAAAALAKClEAAAAAAACwCgpRAAAAAAAAsAoKUQAAAAAAALAKu5xOAE+v0oNXysYhd06nAQAAAADAYy1uZL2cTsFqWBEFAAAAAADwmJg0aZL8/f3l6OiosLAwbd26NcP4BQsWKCgoSI6OjgoODtayZcssjptMpnS30aNHm2M+/PBDVa5cWblz55aHh0eaPs6ePas6derI19dXDg4O8vPzU/fu3ZWUlPTA46MQBQAAAAAA8BiYP3++evfurcGDB2vHjh0qW7asIiMjderUqXTjN27cqNatWys6Olo7d+5UVFSUoqKitHv3bnPMiRMnLLbp06fLZDKpadOm5pgbN26oefPmeuONN9Ltx8bGRo0aNdKPP/6of/75RzNnztTq1avVtWvXBx6jyTAM44HPAjKQlJQkd3d3+cV8x615AAAAAADcR+qteWFhYapQoYImTpwoSUpJSZGfn5969Oih/v37pzmvZcuWunz5spYuXWreV6lSJYWEhGjy5Mnp9hUVFaWLFy9qzZo1aY7NnDlTMTExunDhwn1zHj9+vEaPHq2jR4+a6wCJiYlyc3PL8DxWRAEAAAAAAOSwGzduaPv27YqIiDDvs7GxUUREhDZt2pTuOZs2bbKIl6TIyMh7xp88eVI//fSToqOjHynX48ePa/HixapRo8YDn0shCgAAAAAAIIedOXNGycnJ8vLystjv5eWlhISEdM9JSEh4oPhZs2bJ1dVVTZo0eagcW7durdy5c6tgwYJyc3PTtGnTHrgNClEAAAAAAADPgOnTp6tNmzZydHR8qPPHjh2rHTt26IcfftChQ4fUu3fvB27D7qF6BgAAAAAAQJbx9PSUra2tTp48abH/5MmT8vb2Tvccb2/vTMf/9ttv2r9/v+bPn//QOXp7e8vb21tBQUHKmzevqlWrpoEDB8rZ2TnTbbAiCgAAAAAAIIfZ29srNDTU4iHiKSkpWrNmjcLDw9M9Jzw8PM1Dx1etWpVu/FdffaXQ0FCVLVs2S/JNSUmRJF2/fv2BzmNFFAAAAAAAwGOgd+/eateuncqXL6+KFStq3Lhxunz5sjp06CBJatu2rQoWLKgRI0ZIknr16qUaNWpozJgxqlevnubNm6dt27ZpypQpFu0mJSVpwYIFGjNmTLr9xsfH69y5c4qPj1dycrJiY2MlScWKFZOLi4uWLVumkydPqkKFCnJxcdGePXv0zjvvqEqVKvL391dSUlKmx0ghCgAAAAAA4DHQsmVLnT59WoMGDVJCQoJCQkK0YsUK8wPJ4+PjZWPz/ze3Va5cWXPnztX777+vd999V4GBgVqyZIlKly5t0e68efNkGIZat26dbr+DBg3SrFmzzK/LlSsnSVq3bp1q1qwpJycnTZ06VW+99ZauX78uPz8/NWnSRP3793/gMZoMwzAe+KxsVLNmTYWEhGjcuHHpHvf391dMTIxiYmIkSSaTSd9//72ioqIUFxengIAA7dy5UyEhIRn2s379etWqVUvnz5+Xh4dHlo7hYdxv3E+SpKQkubu7yy/mO9k45M7pdAAAAAAAeKzFjayX0yk8ktQ6QGJiotzc3DKMfeJWRP3xxx8P9BCsx83jVgADAAAAAACwlieuEJU/f/6cTgEAAAAAAAAP4bH81rxbt26pe/fucnd3l6enpwYOHKjUOwj9/f2z7fa133//XdWqVZOTk5P8/PzUs2dPXb582Xzc399fH330kTp27ChXV1cVKlQozQPANm7cqJCQEDk6Oqp8+fJasmSJTCaTYmNjFRcXp1q1akmS8uTJI5PJpPbt25vPTUlJUd++fZU3b155e3tryJAhmc7dZDLpyy+/VP369ZU7d26VKFFCmzZt0sGDB1WzZk05OzurcuXKOnTokPmcIUOGKCQkRNOnT1ehQoXk4uKiN998U8nJyRo1apS8vb1VoEABffjhhw93QQEAAAAAAO7wWBaiZs2aJTs7O23dulWfffaZPv30U02bNi1b+zx06JDq1Kmjpk2b6s8//9T8+fP1+++/q3v37hZxY8aMUfny5bVz5069+eabeuONN7R//35Jt++JbNCggYKDg7Vjxw4NGzZM/fr1M5/r5+enRYsWSZL279+vEydO6LPPPrMYt7Ozs7Zs2aJRo0bpgw8+0KpVqzI9hmHDhqlt27aKjY1VUFCQXnnlFb3++usaMGCAtm3bJsMw0ozn0KFDWr58uVasWKFvv/1WX331lerVq6djx47pl19+0ccff6z3339fW7ZsuWe/169fV1JSksUGAAAAAABwt8eyEOXn56exY8eqePHiatOmjXr06KGxY8dma58jRoxQmzZtFBMTo8DAQFWuXFnjx4/X7Nmzde3aNXPcyy+/rDfffFPFihVTv3795OnpqXXr1kmS5s6dK5PJpKlTp6pkyZKqW7eu3nnnHfO5tra2yps3rySpQIEC8vb2lru7u/l4mTJlNHjwYAUGBqpt27YqX7681qxZk+kxdOjQQS1atNDzzz+vfv36KS4uTm3atFFkZKRKlCihXr16af369RbnpKSkaPr06SpZsqQaNGigWrVqaf/+/Ro3bpyKFy+uDh06qHjx4uYx3uvaubu7mzc/P79M5wwAAAAAAJ4dj2UhqlKlSjKZTObX4eHhOnDggJKTk7Otz127dmnmzJlycXExb5GRkUpJSdHhw4fNcWXKlDH/2WQyydvbW6dOnZJ0e5VTmTJl5OjoaI6pWLFipnO4s21J8vHxMbf9oOenfrVjcHCwxb5r165ZrFjy9/eXq6urRUzJkiUtvg7Sy8srwzwGDBigxMRE83b06NFM5wwAAAAAAJ4dT9zDyrPLpUuX9Prrr6tnz55pjhUqVMj851y5clkcM5lMSklJyZIcHrXtO89PLeSlt+/ONtPr80HzcHBwkIODQ6bzBAAAAAAAz6bHshB19/OINm/erMDAQNna2mZbny+88IL+/vtvFStW7KHbKF68uL755htdv37dXJj5448/LGLs7e0lKVtXdwEAAAAAADyOHstb8+Lj49W7d2/t379f3377rSZMmKBevXpla5/9+vXTxo0b1b17d8XGxurAgQP64Ycf0jzcOyOvvPKKUlJS1KVLF+3du1crV67UJ598Iun/VyMVLlxYJpNJS5cu1enTp3Xp0qVsGQ8AAAAAAMDj5rEsRLVt21ZXr15VxYoV1a1bN/Xq1UtdunTJ1j7LlCmjX375Rf/884+qVaumcuXKadCgQfL19c10G25ubvrvf/+r2NhYhYSE6L333tOgQYMkyfzcqIIFC2ro0KHq37+/vLy8HqjQBQAAAAAA8CQzGYZh5HQST7M5c+aoQ4cOSkxMlJOTU06nYxVJSUm3vz0v5jvZOOTO6XQAAAAAAHisxY2sl9MpPJLUOkBiYqLc3NwyjH0snxH1JJs9e7aKFCmiggULateuXerXr59atGjxzBShAAAAAAAA7uWxvDUvK3Tt2lUuLi7pbl27ds22fhMSEvTqq6+qRIkSeuutt9S8eXNNmTLlkdqcM2fOPcdSqlSpLMocAAAAAAAgez21t+adOnVKSUlJ6R5zc3NTgQIFrJzRw7t48aJOnjyZ7rFcuXKpcOHCVs4oY9yaBwAAAABA5nFr3lOgQIECT1SxKSOurq5ydXXN6TQAAAAAAAAeyVN7ax4AAAAAAAAeLxSiAAAAAAAAYBUUogAAAAAAAGAVFKIAAAAAAABgFRSiAAAAAAAAYBUUogAAAAAAAGAVdjmdAJ5eu4dGys3NLafTAAAAAAAAjwlWRAEAAAAAAMAqKEQBAAAAAADAKihEAQAAAAAAwCooRAEAAAAAAMAqKEQBAAAAAADAKihEAQAAAAAAwCooRAEAAAAAAMAq7HI6ATy9Sg9eKRuH3DmdBiTFjayX0ykAAAAAAMCKKOBZM2nSJPn7+8vR0VFhYWHaunVrhvELFixQUFCQHB0dFRwcrGXLllkcNwxDgwYNko+Pj5ycnBQREaEDBw6Yj69fv14mkynd7Y8//pAk7d+/X7Vq1ZKXl5ccHR1VpEgRvf/++7p582bWXwAAAAAAQI6hEAU8Q+bPn6/evXtr8ODB2rFjh8qWLavIyEidOnUq3fiNGzeqdevWio6O1s6dOxUVFaWoqCjt3r3bHDNq1CiNHz9ekydP1pYtW+Ts7KzIyEhdu3ZNklS5cmWdOHHCYuvUqZMCAgJUvnx5SVKuXLnUtm1b/fzzz9q/f7/GjRunqVOnavDgwdl/UQAAAAAAVmMyDMPI6STwdElKSpK7u7v8Yr7j1rzHROqteWFhYapQoYImTpwoSUpJSZGfn5969Oih/v37pzmvZcuWunz5spYuXWreV6lSJYWEhGjy5MkyDEO+vr56++231adPH0lSYmKivLy8NHPmTLVq1SpNmzdv3lTBggXVo0cPDRw48J459+7dW3/88Yd+++23Rxo7AAAAACB7pdYBEhMT5ebmlmEsK6KAZ8SNGze0fft2RUREmPfZ2NgoIiJCmzZtSvecTZs2WcRLUmRkpDn+8OHDSkhIsIhxd3dXWFjYPdv88ccfdfbsWXXo0OGeuR48eFArVqxQjRo1Mj0+AAAAAMDjj0IU8Iw4c+aMkpOT5eXlZbHfy8tLCQkJ6Z6TkJCQYXzqzwdp86uvvlJkZKSee+65NMcqV64sR0dHBQYGqlq1avrggw8yNzgAAAAAwBOBQhQAqzl27JhWrlyp6OjodI/Pnz9fO3bs0Ny5c/XTTz/pk08+sXKGAAAAAIDsZJfTCQCwDk9PT9na2urkyZMW+0+ePClvb+90z/H29s4wPvXnyZMn5ePjYxETEhKSpr0ZM2YoX758atiwYbr9+fn5SZJKliyp5ORkdenSRW+//bZsbW0zN0gAAAAAwGONFVHAM8Le3l6hoaFas2aNeV9KSorWrFmj8PDwdM8JDw+3iJekVatWmeMDAgLk7e1tEZOUlKQtW7akadMwDM2YMUNt27ZVrly57ptvSkqKbt68qZSUlEyPEQAAAADweGNFFPAM6d27t9q1a6fy5curYsWKGjdunC5fvmx+cHjbtm1VsGBBjRgxQpLUq1cv1ahRQ2PGjFG9evU0b948bdu2TVOmTJEkmUwmxcTEaPjw4QoMDFRAQIAGDhwoX19fRUVFWfS9du1aHT58WJ06dUqT15w5c5QrVy4FBwfLwcFB27Zt04ABA9SyZctMFa0AAAAAAE8GClHAM6Rly5Y6ffq0Bg0apISEBIWEhGjFihXmh43Hx8fLxub/F0pWrlxZc+fO1fvvv693331XgYGBWrJkiUqXLm2O6du3ry5fvqwuXbrowoULqlq1qlasWCFHR0eLvr/66itVrlxZQUFBafKys7PTxx9/rH/++UeGYahw4cLq3r273nrrrWy6EgAAAACAnGAyDMPI6STw/2rWrKmQkBCNGzcuS+Ie1ZAhQ7RkyRLFxsZm+pykpCS5u7vLL+Y72Tjkzr7kkGlxI+vldAoAAAAAgKdUah0gMTFRbm5uGcayIuoxs3jxYm5FAgAAAAAATyUKUY+ZvHnz5nQKAAAAAAAA2YJvzXvM1KxZUzExMZKkzz//XIGBgXJ0dJSXl5eaNWt2z/O+/vprlS9fXq6urvL29tYrr7yiU6dOmY+vX79eJpNJa9asUfny5ZU7d25VrlxZ+/fvt2hn5MiR8vLykqurq6Kjo3Xt2rVsGScAAAAAAHj2UIh6TG3btk09e/bUBx98oP3792vFihWqXr36PeNv3rypYcOGadeuXVqyZIni4uLUvn37NHHvvfeexowZo23btsnOzk4dO3Y0H/vuu+80ZMgQffTRR9q2bZt8fHz0+eef3zfX69evKykpyWIDAAAAAAC4G7fmPabi4+Pl7Oys+vXry9XVVYULF1a5cuXuGX9nQalIkSIaP368KlSooEuXLsnFxcV87MMPP1SNGjUkSf3791e9evV07do1OTo6aty4cYqOjlZ0dLQkafjw4Vq9evV9V0WNGDFCQ4cOfZThAgAAAACAZwAroh5TL774ogoXLqwiRYrotdde05w5c3TlypV7xm/fvl0NGjRQoUKF5Orqai42xcfHW8SVKVPG/GcfHx9JMt/Ct3fvXoWFhVnEh4eH3zfXAQMGKDEx0bwdPXo0c4MEAAAAAADPFApRjylXV1ft2LFD3377rXx8fDRo0CCVLVtWFy5cSBN7+fJlRUZGys3NTXPmzNEff/yh77//XpJ048YNi9g7v5HPZDJJklJSUh4pVwcHB7m5uVlsAAAAAAAAd6MQ9Rizs7NTRESERo0apT///FNxcXFau3Ztmrh9+/bp7NmzGjlypKpVq6agoCCLB5VnVokSJbRlyxaLfZs3b37o/AEAAAAAAO7EM6IeU0uXLtX//vc/Va9eXXny5NGyZcuUkpKi4sWLp4ktVKiQ7O3tNWHCBHXt2lW7d+/WsGHDHrjPXr16qX379ipfvryqVKmiOXPmaM+ePSpSpEhWDAkAAAAAADzjWBH1mPLw8NDixYv1n//8RyVKlNDkyZP17bffqlSpUmli8+fPr5kzZ2rBggUqWbKkRo4cqU8++eSB+2zZsqUGDhyovn37KjQ0VEeOHNEbb7yRFcMBAAAAAACQyTAMI6eTwNMlKSlJ7u7u8ov5TjYOuXM6HUiKG1kvp1MAAAAAADylUusAiYmJ931uNCuiAAAAAAAAYBUUogAAAAAAAGAVFKIAAAAAAABgFRSiAAAAAAAAYBUUogAAAAAAAGAVFKIAAAAAAABgFRSiAAAAAAAAYBUUogAAAAAAAGAVFKIAAAAAAABgFXY5nQCeXruHRsrNzS2n0wAAAAAAAI8JVkQBAAAAAADAKihEAQAAAAAAwCooRAEAAAAAAMAqKEQBAAAAAADAKihEAQAAAAAAwCooRAEAAAAAAMAqKEQBAAAAAADAKihEAQAAAAAAwCrscjoBPL1KD14pG4fcOZ2G1cSNrJfTKQAAAAAA8FhjRRSQDSZNmiR/f385OjoqLCxMW7duzTB+wYIFCgoKkqOjo4KDg7Vs2TKL44ZhaNCgQfLx8ZGTk5MiIiJ04MCBdNu6fv26QkJCZDKZFBsba94fFxcnk8mUZtu8efMjjxcAAAAAgMygEAVksfnz56t3794aPHiwduzYobJlyyoyMlKnTp1KN37jxo1q3bq1oqOjtXPnTkVFRSkqKkq7d+82x4waNUrjx4/X5MmTtWXLFjk7OysyMlLXrl1L017fvn3l6+t7z/xWr16tEydOmLfQ0NBHHzQAAAAAAJlAIQrIYp9++qk6d+6sDh06qGTJkpo8ebJy586t6dOnpxv/2WefqU6dOnrnnXdUokQJDRs2TC+88IImTpwo6fZqqHHjxun9999Xo0aNVKZMGc2ePVvHjx/XkiVLLNpavny5fv75Z33yySf3zC9fvnzy9vY2b7ly5cqysQMAAAAAkBEKUUAWunHjhrZv366IiAjzPhsbG0VERGjTpk3pnrNp0yaLeEmKjIw0xx8+fFgJCQkWMe7u7goLC7No8+TJk+rcubO+/vpr5c5972dzNWzYUAUKFFDVqlX1448/PtQ4AQAAAAB4GBSigCx05swZJScny8vLy2K/l5eXEhIS0j0nISEhw/jUnxnFGIah9u3bq2vXripfvny6/bi4uGjMmDFasGCBfvrpJ1WtWlVRUVEUowAAAAAAVsO35gFPgQkTJujixYsaMGDAPWM8PT3Vu3dv8+sKFSro+PHjGj16tBo2bGiNNAEAAAAAzzhWRAFZyNPTU7a2tjp58qTF/pMnT8rb2zvdc7y9vTOMT/2ZUczatWu1adMmOTg4yM7OTsWKFZMklS9fXu3atbtnvmFhYTp48OADjBAAAAAAgIdHIQrIQvb29goNDdWaNWvM+1JSUrRmzRqFh4ene054eLhFvCStWrXKHB8QECBvb2+LmKSkJG3ZssUcM378eO3atUuxsbGKjY3VsmXLJN3+Br8PP/zwnvnGxsbKx8fn4QYLAAAAAMAD4tY8IIv17t1b7dq1U/ny5VWxYkWNGzdOly9fVocOHSRJbdu2VcGCBTVixAhJUq9evVSjRg2NGTNG9erV07x587Rt2zZNmTJFkmQymRQTE6Phw4crMDBQAQEBGjhwoHx9fRUVFSVJKlSokEUOLi4ukqSiRYvqueeekyTNmjVL9vb2KleunCRp8eLFmj59uqZNm5bt1wQAAAAAAOkBV0QZhqEuXboob968MplM8vDwUExMjPm4v7+/xo0bl8UpZg+TyaQlS5bkdBqSpCFDhigkJCSn00AWadmypT755BMNGjRIISEhio2N1YoVK8wPG4+Pj9eJEyfM8ZUrV9bcuXM1ZcoUlS1bVgsXLtSSJUtUunRpc0zfvn3Vo0cPdenSRRUqVNClS5e0YsUKOTo6PlBuw4YNU2hoqMLCwvTDDz9o/vz55gIZAAAAAADZzWQYhpHZ4OXLl6tRo0Zav369ihQpombNmql8+fLm4tPp06fl7Oyc4VfHPy5MJpO+//5784qSnOx3yJAhWrJkiWJjY62aS3ZJSkqSu7u7/GK+k43D4z8XskrcyHo5nQIAAAAAAFaXWgdITEyUm5tbhrEPdGveoUOH5OPjo8qVK98+2c7y9Pz58z9gqtnjxo0bsre3z+k0AAAAAAAAcIdM35rXvn179ejRQ/Hx8TKZTPL3908Tc/eteSaTSV988YXq1q0rJycnFSlSRAsXLjQfj4uLk8lk0rx581S5cmU5OjqqdOnS+uWXXyza3b17t+rWrSsXFxd5eXnptdde05kzZ8zHa9asqe7duysmJkaenp6KjIx8gEtw29GjR9WiRQt5eHgob968atSokeLi4izGHxUVpU8++UQ+Pj7Kly+funXrpps3b5pjTpw4oXr16snJyUkBAQGaO3euxTVJvWaNGzdO9xp+/fXX8vf3l7u7u1q1aqWLFy9mKveaNWuqR48eiomJUZ48eeTl5aWpU6ean0vk6uqqYsWKafny5eZz1q9fL5PJpJUrV6pcuXJycnLSf/7zH506dUrLly9XiRIl5ObmpldeeUVXrlx54OsJAAAAAABwt0wXoj777DN98MEHeu6553TixAn98ccfmTpv4MCBatq0qXbt2qU2bdqoVatW2rt3r0XMO++8o7fffls7d+5UeHi4GjRooLNnz0qSLly4oP/85z8qV66ctm3bphUrVujkyZNq0aKFRRupD2LesGGDJk+enNlhSZJu3rypyMhIubq66rffftOGDRvk4uKiOnXq6MaNG+a4devW6dChQ1q3bp1mzZqlmTNnaubMmebjbdu21fHjx7V+/XotWrRIU6ZM0alTp8zHU6/ZjBkz0lzDQ4cOacmSJVq6dKmWLl2qX375RSNHjsz0GGbNmiVPT09t3bpVPXr00BtvvKHmzZurcuXK2rFjh1566SW99tpraYpKQ4YM0cSJE7Vx40ZzMW7cuHGaO3eufvrpJ/3888+aMGHCA11PAAAAAACA9GS6EOXu7i5XV1fZ2trK29s707fhNW/eXJ06ddLzzz+vYcOGqXz58mkKG927d1fTpk1VokQJffHFF3J3d9dXX30lSZo4caLKlSunjz76SEFBQSpXrpymT5+udevW6Z9//jG3ERgYqFGjRql48eIqXrx4Zocl6fZX3KekpGjatGkKDg5WiRIlNGPGDMXHx2v9+vXmuDx58mjixIkKCgpS/fr1Va9ePa1Zs0aStG/fPq1evVpTp05VWFiYXnjhBU2bNk1Xr141n596zTw8PNJcw5SUFM2cOVOlS5dWtWrV9Nprr5nbzoyyZcvq/fffV2BgoAYMGCBHR0d5enqqc+fOCgwM1KBBg3T27Fn9+eefFucNHz5cVapUUbly5RQdHa1ffvlFX3zxhcqVK6dq1aqpWbNmWrduXYZ9X79+XUlJSRYbAAAAAADA3R7oW/MeRnh4eJrXd6+IujPGzs5O5cuXN8fs2rVL69atk4uLi3kLCgqSdHsVUarQ0NCHznHXrl06ePCgXF1dzX3kzZtX165ds+ijVKlSsrW1Nb/28fExr3jav3+/7Ozs9MILL5iPFytWTHny5MlUDv7+/nJ1dU237cwoU6aM+c+2trbKly+fgoODzftSv7Ht7jbvPM/Ly0u5c+dWkSJFLPbdL48RI0bI3d3dvPn5+WU6bwAAAAAA8Ox4oIeV54RLly6pQYMG+vjjj9Mc8/HxMf/Z2dn5kfoIDQ3VnDlz0hy7c9VSrly5LI6ZTCalpKQ8dL93etS20zv/zn0mk0mS0rR5d8zD5DFgwAD17t3b/DopKYliFAAAAAAASCPbV0Rt3rw5zesSJUrcM+bWrVvavn27OeaFF17Qnj175O/vr2LFillsj1J8utMLL7ygAwcOqECBAmn6cHd3z1QbxYsX161bt7Rz507zvoMHD+r8+fMWcbly5VJycnKW5P24cHBwkJubm8UGAAAAAABwt2wvRC1YsEDTp0/XP//8o8GDB2vr1q3q3r27RcykSZP0/fffa9++ferWrZvOnz+vjh07SpK6deumc+fOqXXr1vrjjz906NAhrVy5Uh06dMiygk6bNm3k6empRo0a6bffftPhw4e1fv169ezZU8eOHctUG0FBQYqIiFCXLl20detW7dy5U126dJGTk5N5NZJ0+xa8NWvWKCEhIU2RCgAAAAAA4GmW7YWooUOHat68eSpTpoxmz56tb7/9ViVLlrSIGTlypEaOHKmyZcvq999/148//ihPT09Jkq+vrzZs2KDk5GS99NJLCg4OVkxMjDw8PGRjkzXp586dW7/++qsKFSqkJk2aqESJEoqOjta1a9ceaHXP7Nmz5eXlperVq6tx48bq3LmzXF1d5ejoaI4ZM2aMVq1aJT8/P5UrVy5L8gcAAAAAAHgSmAzDMLKtcZNJ33//vaKiotI9HhcXp4CAAO3cuVMhISHZlUaOOXbsmPz8/LR69WrVrl07p9OxmqSkpNsPLY/5TjYOuXM6HauJG1kvp1MAAAAAAMDqUusAiYmJ913Q89g/rPxJsnbtWl26dEnBwcE6ceKE+vbtK39/f1WvXj2nUwMAAAAAAMhx2X5rXk6YM2eOXFxc0t1KlSqVbf3evHlT7777rkqVKqXGjRsrf/78Wr9+fZpvonsQ8fHx9xyLi4uL4uPjs3AEAAAAAAAA2Sdbb83LKRcvXtTJkyfTPZYrVy4VLlzYyhk9vFu3bikuLu6ex/39/WVn93gtbOPWPAAAAAAAnh3P/K15rq6ucnV1zek0soSdnZ2KFSuW02kAAAAAAAA8sqfy1jwAAAAAAAA8fihEAQAAAAAAwCooRAEAAAAAAMAqKEQBAAAAAADAKihEAQAAAAAAwCooRAEAAAAAAMAq7HI6ATy9dg+NlJubW06nAQAAAAAAHhOsiAIAAAAAAIBVUIgCAAAAAACAVVCIAgAAAAAAgFVQiAIAAAAAAIBVUIgCAAAAAACAVVCIAgAAAAAAgFVQiAIAAAAAAIBV2OV0Anh6lR68UjYOua3WX9zIelbrCwAAAAAAPDhWROGpNGnSJPn7+8vR0VFhYWHaunVrhvELFixQUFCQHB0dFRwcrGXLllkcNwxDgwYNko+Pj5ycnBQREaEDBw6Yj8fFxSk6OloBAQFycnJS0aJFNXjwYN24ccMcc+3aNbVv317BwcGys7NTVFRUlo4ZAAAAAIDHHYUoPHXmz5+v3r17a/DgwdqxY4fKli2ryMhInTp1Kt34jRs3qnXr1oqOjtbOnTsVFRWlqKgo7d692xwzatQojR8/XpMnT9aWLVvk7OysyMhIXbt2TZK0b98+paSk6Msvv9SePXs0duxYTZ48We+++665jeTkZDk5Oalnz56KiIjI3osAAAAAAMBjyGQYhpHTSeDpkpSUJHd3d/nFfJcjt+aFhYWpQoUKmjhxoiQpJSVFfn5+6tGjh/r375/mvJYtW+ry5ctaunSpeV+lSpUUEhKiyZMnyzAM+fr66u2331afPn0kSYmJifLy8tLMmTPVqlWrdPMZPXq0vvjiC/3vf/9Lc6x9+/a6cOGClixZ8qjDBgAAAAAgR6XWARITE+Xm5pZhLCui8FS5ceOGtm/fbrHiyMbGRhEREdq0aVO652zatCnNCqXIyEhz/OHDh5WQkGAR4+7urrCwsHu2Kd0uVuXNm/dRhgMAAAAAwFOFQhSeKmfOnFFycrK8vLws9nt5eSkhISHdcxISEjKMT/35IG0ePHhQEyZM0Ouvv/5Q4wAAAAAA4GlEIQrIYv/++6/q1Kmj5s2bq3PnzjmdDgAAAAAAjw0KUXiqeHp6ytbWVidPnrTYf/LkSXl7e6d7jre3d4bxqT8z0+bx48dVq1YtVa5cWVOmTHmksQAAAAAA8LShEIWnir29vUJDQ7VmzRrzvpSUFK1Zs0bh4eHpnhMeHm4RL0mrVq0yxwcEBMjb29siJikpSVu2bLFo899//1XNmjUVGhqqGTNmyMaGXy8AAAAAAO5kl9MJAFmtd+/eateuncqXL6+KFStq3Lhxunz5sjp06CBJatu2rQoWLKgRI0ZIknr16qUaNWpozJgxqlevnubNm6dt27aZVzSZTCbFxMRo+PDhCgwMVEBAgAYOHChfX19FRUVJ+v8iVOHChfXJJ5/o9OnT5nzuXDX1999/68aNGzp37pwuXryo2NhYSVJISEj2XxgAAAAAAHIYhSg8dVq2bKnTp09r0KBBSkhIUEhIiFasWGF+2Hh8fLzFaqXKlStr7ty5ev/99/Xuu+8qMDBQS5YsUenSpc0xffv21eXLl9WlSxdduHBBVatW1YoVK+To6Cjp9gqqgwcP6uDBg3ruuecs8jEMw/znl19+WUeOHDG/LleuXJoYAAAAAACeVibjMf4XcM2aNRUSEqJx48ale9zf318xMTGKiYmRdHvlyvfff6+oqCjFxcUpICBAO3fufKjVJuvXr1etWrV0/vx5eXh4PPQYMmPIkCFasmSJeXXMky4pKUnu7u7yi/lONg65rdZv3Mh6VusLAAAAAADclloHSExMlJubW4axT/SKqD/++EPOzs45ncYDubNYBgAAAAAA8Cx5ogtR+fPnz+kUAAAAAAAAkEmP/dd63bp1S927d5e7u7s8PT01cOBA8/N0/P3973nb3oNatmyZnn/+eTk5OalWrVqKi4tLE/P777+rWrVqcnJykp+fn3r27KnLly+bj/v7+2vYsGFq3bq1nJ2dVbBgQU2aNMniuCQ1btxYJpPJ/DrV119/LX9/f7m7u6tVq1a6ePFipnKvWbOmevTooZiYGOXJk0deXl6aOnWq+QHdrq6uKlasmJYvX24+Z/369TKZTFq5cqXKlSsnJycn/ec//9GpU6e0fPlylShRQm5ubnrllVd05cqVzF9IAAAAAACAe3jsC1GzZs2SnZ2dtm7dqs8++0yffvqppk2blqV9HD16VE2aNFGDBg0UGxurTp06qX///hYxhw4dUp06ddS0aVP9+eefmj9/vn7//Xd1797dIm706NEqW7asdu7cqf79+6tXr15atWqVpNu3EkrSjBkzdOLECfPr1PaXLFmipUuXaunSpfrll180cuTITI9h1qxZ8vT01NatW9WjRw+98cYbat68uSpXrqwdO3bopZde0muvvZamqDRkyBBNnDhRGzdu1NGjR9WiRQuNGzdOc+fO1U8//aSff/5ZEyZMeKDrCQAAAAAAkJ7HvhDl5+ensWPHqnjx4mrTpo169OihsWPHZmkfX3zxhYoWLaoxY8aY+2nfvr1FzIgRI9SmTRvFxMQoMDBQlStX1vjx4zV79mxdu3bNHFelShX1799fzz//vHr06KFmzZqZ8029ldDDw0Pe3t4WtxampKRo5syZKl26tKpVq6bXXntNa9asyfQYypYtq/fff1+BgYEaMGCAHB0d5enpqc6dOyswMFCDBg3S2bNn9eeff1qcN3z4cFWpUkXlypVTdHS0fvnlF33xxRcqV66cqlWrpmbNmmndunUZ9n39+nUlJSVZbAAAAAAAAHd77AtRlSpVkslkMr8ODw/XgQMHlJycnGV97N27V2FhYRb7wsPDLV7v2rVLM2fOlIuLi3mLjIxUSkqKDh8+fM/zwsPDtXfv3vvm4O/vL1dXV/NrHx8fnTp1KtNjKFOmjPnPtra2ypcvn4KDg837vLy8JClNm3ee5+Xlpdy5c6tIkSIW++6Xx4gRI+Tu7m7e/Pz8Mp03AAAAAAB4djzRDyu3pkuXLun1119Xz5490xwrVKjQI7efK1cui9cmk0kpKSmPdP6d+1KLeXe3eXfMw+QxYMAA9e7d2/w6KSmJYhQAAAAAAEjjsS9EbdmyxeL15s2bFRgYKFtb2yzro0SJEvrxxx/T9HOnF154QX///beKFSuWYVt3n7d582aVKFHC/DpXrlxZuprrceDg4CAHB4ecTgMAAAAAADzmHvtb8+Lj49W7d2/t379f3377rSZMmKBevXplaR9du3bVgQMH9M4772j//v2aO3euZs6caRHTr18/bdy4Ud27d1dsbKwOHDigH374Ic3Dyjds2KBRo0bpn3/+0aRJk7RgwQKLfP39/bVmzRolJCTo/PnzWToOAAAAAACAx9ljX4hq27atrl69qooVK6pbt27q1auXunTpkqV9FCpUSIsWLdKSJUtUtmxZTZ48WR999JFFTJkyZfTLL7/on3/+UbVq1VSuXDkNGjRIvr6+FnFvv/22tm3bpnLlymn48OH69NNPFRkZaT4+ZswYrVq1Sn5+fipXrlyWjgMAAAAAAOBxZjIMw8jpJJ4W/v7+iomJUUxMTE6nkqOSkpJuP7Q85jvZOOS2Wr9xI+tZrS8AAAAAAHBbah0gMTFRbm5uGcY+9iuiAAAAAAAA8HR4JgpRXbt2lYuLS7pb165dczq9DMXHx98zdxcXF8XHx+d0igAAAAAAAJny2H9rXlb44IMP1KdPn3SP3W/J2IOIi4vLsrZS+fr6KjY2NsPjAAAAAAAAT4JnohBVoEABFShQIKfTeCh2dnYqVqxYTqcBAAAAAADwyJ6JW/MAAAAAAACQ8yhEAQAAAAAAwCooRAEAAAAAAMAqKEQBAAAAAADAKihEAQAAAAAAwCooRAEAAAAAAMAq7HI6ATy9dg+NlJubW06nAQAAAAAAHhOsiAIAAAAAAIBVUIgCAAAAAACAVVCIAgAAAAAAgFVQiAIAAAAAAIBVUIgCAAAAAACAVVCIAgAAAAAAgFVQiAIAAAAAAIBV2OV0Anh6lR68UjYOudPsjxtZLweyAQAAAAAAOY0VUchRkyZNkr+/vxwdHRUWFqatW7dmGL9gwQIFBQXJ0dFRwcHBWrZsmcVxwzA0aNAg+fj4yMnJSRERETpw4IBFzLlz59SmTRu5ubnJw8ND0dHRunTpkvl4XFycTCZTmm3z5s1ZN3AAAAAAAJ5BFKKQY+bPn6/evXtr8ODB2rFjh8qWLavIyEidOnUq3fiNGzeqdevWio6O1s6dOxUVFaWoqCjt3r3bHDNq1CiNHz9ekydP1pYtW+Ts7KzIyEhdu3bNHNOmTRvt2bNHq1at0tKlS/Xrr7+qS5cuafpbvXq1Tpw4Yd5CQ0Oz/iIAAAAAAPAMMRmGYeR0Eni6JCUlyd3dXX4x32V4a15YWJgqVKigiRMnSpJSUlLk5+enHj16qH///mnOa9mypS5fvqylS5ea91WqVEkhISGaPHmyDMOQr6+v3n77bfXp00eSlJiYKC8vL82cOVOtWrXS3r17VbJkSf3xxx8qX768JGnFihV6+eWXdezYMfn6+iouLk4BAQHauXOnQkJCsvryAAAAAADwVEmtAyQmJsrNzS3DWFZEIUfcuHFD27dvV0REhHmfjY2NIiIitGnTpnTP2bRpk0W8JEVGRprjDx8+rISEBIsYd3d3hYWFmWM2bdokDw8PcxFKkiIiImRjY6MtW7ZYtN2wYUMVKFBAVatW1Y8//vhoAwYAAAAAABSikDPOnDmj5ORkeXl5Wez38vJSQkJCuuckJCRkGJ/6834xBQoUsDhuZ2envHnzmmNcXFw0ZswYLViwQD/99JOqVq2qqKgoilEAAAAAADwivjUPuIunp6d69+5tfl2hQgUdP35co0ePVsOGDXMwMwAAAAAAnmysiEKO8PT0lK2trU6ePGmx/+TJk/L29k73HG9v7wzjU3/eL+buh6HfunVL586du2e/0u3nWR08eDATIwMAAAAAAPdCIQo5wt7eXqGhoVqzZo15X0pKitasWaPw8PB0zwkPD7eIl6RVq1aZ4wMCAuTt7W0Rk5SUpC1btphjwsPDdeHCBW3fvt0cs3btWqWkpCgsLOye+cbGxsrHx+fBBwoAAAAAAMy4NQ85pnfv3mrXrp3Kly+vihUraty4cbp8+bI6dOggSWrbtq0KFiyoESNGSJJ69eqlGjVqaMyYMapXr57mzZunbdu2acqUKZIkk8mkmJgYDR8+XIGBgQoICNDAgQPl6+urqKgoSVKJEiVUp04dde7cWZMnT9bNmzfVvXt3tWrVSr6+vpKkWbNmyd7eXuXKlZMkLV68WNOnT9e0adOsfIUAAAAAAHi6UIhCjmnZsqVOnz6tQYMGKSEhQSEhIVqxYoX5YePx8fGysfn/RXuVK1fW3Llz9f777+vdd99VYGCglixZotKlS5tj+vbtq8uXL6tLly66cOGCqlatqhUrVsjR0dEcM2fOHHXv3l21a9eWjY2NmjZtqvHjx1vkNmzYMB05ckR2dnYKCgrS/Pnz1axZs2y+IgAAAAAAPN1MhmEYOZ3Eo6hZs6ZCQkI0bty4LGtz5syZiomJ0YULF7KszWdJUlKS3N3d5RfznWwccqc5HjeyXg5kBQAAAAAAskNqHSAxMVFubm4ZxvKMKAAAAAAAAFgFhSgAAAAAAABYxVNRiLp165a6d+8ud3d3eXp6auDAgUq94/D8+fNq27at8uTJo9y5c6tu3bo6cOCAxfkzZ85UoUKFlDt3bjVu3Fhnz541H4uLi5ONjY22bdtmcc64ceNUuHBhpaSkZJjb+vXrZTKZtHLlSpUrV05OTk76z3/+o1OnTmn58uUqUaKE3Nzc9Morr+jKlSvm81asWKGqVavKw8ND+fLlU/369XXo0CHz8Rs3bqh79+7y8fGRo6OjChcubH6ot2EYGjJkiAoVKiQHBwf5+vqqZ8+embqWJ06cUL169eTk5KSAgADNnTtX/v7+WXrrIwAAAAAAeDY9FYWoWbNmyc7OTlu3btVnn32mTz/91PwNZ+3bt9e2bdv0448/atOmTTIMQy+//LJu3rwpSdqyZYuio6PVvXt3xcbGqlatWho+fLi5bX9/f0VERGjGjBkWfc6YMUPt27e3eJh2RoYMGaKJEydq48aNOnr0qFq0aKFx48Zp7ty5+umnn/Tzzz9rwoQJ5vjLly+rd+/e2rZtm9asWSMbGxs1btzYXPgaP368fvzxR3333Xfav3+/5syZI39/f0nSokWLNHbsWH355Zc6cOCAlixZouDg4Ezl2bZtWx0/flzr16/XokWLNGXKFJ06dSrDc65fv66kpCSLDQAAAAAA4G5Pxbfm+fn5aezYsTKZTCpevLj++usvjR07VjVr1tSPP/6oDRs2qHLlypJuf2Oan5+flixZoubNm+uzzz5TnTp11LdvX0nS888/r40bN2rFihXm9jt16qSuXbvq008/lYODg3bs2KG//vpLP/zwQ6ZzHD58uKpUqSJJio6O1oABA3To0CEVKVJEktSsWTOtW7dO/fr1kyQ1bdrU4vzp06crf/78+vvvv1W6dGnFx8crMDBQVatWlclkUuHChc2x8fHx8vb2VkREhHLlyqVChQqpYsWK981x3759Wr16tf744w+VL19ekjRt2jQFBgZmeN6IESM0dOjQTF8LAAAAAADwbHoqVkRVqlRJJpPJ/Do8PFwHDhzQ33//LTs7O4WFhZmP5cuXT8WLF9fevXslSXv37rU4nnr+naKiomRra6vvv/9e0u1b+WrVqmVegZQZZcqUMf/Zy8tLuXPnNhehUvfdufLowIEDat26tYoUKSI3NzdzX/Hx8ZJur/SKjY1V8eLF1bNnT/3888/mc5s3b66rV6+qSJEi6ty5s77//nvdunXrvjnu379fdnZ2euGFF8z7ihUrpjx58mR43oABA5SYmGjejh49et++AAAAAADAs+epKERlN3t7e7Vt21YzZszQjRs3NHfuXHXs2PGB2siVK5f5zyaTyeJ16r47nzfVoEEDnTt3TlOnTtWWLVu0ZcsWSbefDSVJL7zwgg4fPqxhw4bp6tWratGihZo1aybp9gqx/fv36/PPP5eTk5PefPNNVa9e3Xw7YlZzcHCQm5ubxQYAAAAAAHC3p6IQlVqkSbV582YFBgaqZMmSunXrlsXxs2fPav/+/SpZsqQkqUSJEumef7dOnTpp9erV+vzzz3Xr1i01adIkG0ZimeP777+v2rVrq0SJEjp//nyaODc3N7Vs2VJTp07V/PnztWjRIp07d06S5OTkpAYNGmj8+PFav369Nm3apL/++ivDfosXL65bt25p586d5n0HDx5Mt28AAAAAAIAH9VQ8Iyo+Pl69e/fW66+/rh07dmjChAkaM2aMAgMD1ahRI3Xu3FlffvmlXF1d1b9/fxUsWFCNGjWSJPXs2VNVqlTRJ598okaNGmnlypUWz4dKVaJECVWqVEn9+vVTx44d5eTklG3jyZMnj/Lly6cpU6bIx8dH8fHx6t+/v0XMp59+Kh8fH5UrV042NjZasGCBvL295eHhoZkzZyo5OVlhYWHKnTu3vvnmGzk5OVk8Ryo9QUFBioiIUJcuXfTFF18oV65cevvtt+Xk5GRx6yMAAAAAAMDDeCpWRLVt21ZXr15VxYoV1a1bN/Xq1UtdunSRdPvb7UJDQ1W/fn2Fh4fLMAwtW7bMfGtcpUqVNHXqVH322WcqW7asfv75Z73//vvp9hMdHa0bN2488G15D8rGxkbz5s3T9u3bVbp0ab311lsaPXq0RYyrq6tGjRql8uXLq0KFCoqLi9OyZctkY2MjDw8PTZ06VVWqVFGZMmW0evVq/fe//1W+fPnu2/fs2bPl5eWl6tWrq3HjxurcubNcXV3l6OiYXcMFAAAAAADPCJNhGEZOJ/GkGDZsmBYsWKA///wzp1OxmmPHjsnPz0+rV69W7dq1M3VOUlKS3N3d5RfznWwccqc5HjeyXlanCQAAAAAAckhqHSAxMfG+z41+Km7Ny26XLl1SXFycJk6cqOHDh+d0Otlq7dq1unTpkoKDg3XixAn17dtX/v7+ql69ek6nBgAAAAAAnnBPxa152a179+4KDQ1VzZo109yW17VrV7m4uKS7de3aNYcyTt9vv/12z1xdXFwkSTdv3tS7776rUqVKqXHjxsqfP7/Wr1+f5lv+AAAAAAAAHhS35j2iU6dOKSkpKd1jbm5uKlCggJUzurerV6/q33//vefxYsWKZUk/3JoHAAAAAMCzg1vzrKhAgQKPVbEpI05OTllWbAIAAAAAAHhQ3JoHAAAAAAAAq6AQBQAAAAAAAKugEAUAAAAAAACroBAFAAAAAAAAq6AQBQAAAAAAAKvgW/OQbXYPjbzv1zYCAAAAAIBnByuiAAAAAAAAYBUUogAAAAAAAGAVFKIAAAAAAABgFRSiAAAAAAAAYBUUogAAAAAAAGAVFKIAAAAAAABgFRSiAAAAAAAAYBUUogAAAAAAAGAVdjmdAJ5epQevlI1Dbot9cSPr5VA2AAAAAAAgp7EiCjlm0qRJ8vf3l6Ojo8LCwrR169YM4xcsWKCgoCA5OjoqODhYy5YtszhuGIYGDRokHx8fOTk5KSIiQgcOHLCIOXfunNq0aSM3Nzd5eHgoOjpaly5dMh+Pi4uTyWRKs23evDnrBg4AAAAAwDOKQhRyxPz589W7d28NHjxYO3bsUNmyZRUZGalTp06lG79x40a1bt1a0dHR2rlzp6KiohQVFaXdu3ebY0aNGqXx48dr8uTJ2rJli5ydnRUZGalr166ZY9q0aaM9e/Zo1apVWrp0qX799Vd16dIlTX+rV6/WiRMnzFtoaGjWXwQAAAAAAJ4xJsMwjJxOAk+XpKQkubu7yy/mu3vemhcWFqYKFSpo4sSJkqSUlBT5+fmpR48e6t+/f5o2W7ZsqcuXL2vp0qXmfZUqVVJISIgmT54swzDk6+urt99+W3369JEkJSYmysvLSzNnzlSrVq20d+9elSxZUn/88YfKly8vSVqxYoVefvllHTt2TL6+voqLi1NAQIB27typkJCQ7Lg8AAAAAAA8VVLrAImJiXJzc8swlhVRsLobN25o+/btioiIMO+zsbFRRESENm3alO45mzZtsoiXpMjISHP84cOHlZCQYBHj7u6usLAwc8ymTZvk4eFhLkJJUkREhGxsbLRlyxaLths2bKgCBQqoatWq+vHHHx9twAAAAAAAQBKFKOSAM2fOKDk5WV5eXhb7vby8lJCQkO45CQkJGcan/rxfTIECBSyO29nZKW/evOYYFxcXjRkzRgsWLNBPP/2kqlWrKioqimIUAAAAAABZgG/NA+7g6emp3r17m19XqFBBx48f1+jRo9WwYcMczAwAAAAAgCcfK6JgdZ6enrK1tdXJkyct9p88eVLe3t7pnuPt7Z1hfOrP+8Xc/TD0W7du6dy5c/fsV7r9PKuDBw9mYmQAAAAAACAjFKJgdfb29goNDdWaNWvM+1JSUrRmzRqFh4ene054eLhFvCStWrXKHB8QECBvb2+LmKSkJG3ZssUcEx4ergsXLmj79u3mmLVr1yolJUVhYWH3zDc2NlY+Pj4PPlAAAAAAAGCBW/OQI3r37q127dqpfPnyqlixosaNG6fLly+rQ4cOkqS2bduqYMGCGjFihCSpV69eqlGjhsaMGaN69epp3rx52rZtm6ZMmSJJMplMiomJ0fDhwxUYGKiAgAANHDhQvr6+ioqKkiSVKFFCderUUefOnTV58mTdvHlT3bt3V6tWreTr6ytJmjVrluzt7VWuXDlJ0uLFizV9+nRNmzbNylcIAAAAAICnD4Uo5IiWLVvq9OnTGjRokBISEhQSEqIVK1aYHzYeHx8vG5v/X7BXuXJlzZ07V++//77effddBQYGasmSJSpdurQ5pm/fvrp8+bK6dOmiCxcuqGrVqlqxYoUcHR3NMXPmzFH37t1Vu3Zt2djYqGnTpho/frxFbsOGDdORI0dkZ2enoKAgzZ8/X82aNcvmKwIAAAAAwNPPZBiGkdNJ4OmSlJQkd3d3+cV8JxuH3BbH4kbWy6GsAAAAAABAdkitAyQmJsrNzS3DWJ4RBQAAAAAAAKugEPWUWrhwoYKDg+Xk5KR8+fIpIiJCly9fliRNmzZNJUqUkKOjo4KCgvT555+bz+vYsaPKlCmj69evS5Ju3LihcuXKqW3btjkyDgAAAAAA8PSgEPUUOnHihFq3bq2OHTtq7969Wr9+vZo0aSLDMDRnzhwNGjRIH374ofbu3auPPvpIAwcO1KxZsyRJ48eP1+XLl9W/f39J0nvvvacLFy5o4sSJOTkkAAAAAADwFOBh5U+hEydO6NatW2rSpIkKFy4sSQoODpYkDR48WGPGjFGTJk0kSQEBAfr777/15Zdfql27dnJxcdE333yjGjVqyNXVVePGjdO6desyvMfz+vXr5hVU0u17QwEAAAAAAO5GIeopVLZsWdWuXVvBwcGKjIzUSy+9pGbNmsne3l6HDh1SdHS0OnfubI6/deuW3N3dza/Dw8PVp08fDRs2TP369VPVqlUz7G/EiBEaOnRoto0HAAAAAAA8Hbg17ylka2urVatWafny5SpZsqQmTJig4sWLa/fu3ZKkqVOnKjY21rzt3r1bmzdvNp+fkpKiDRs2yNbWVgcPHrxvfwMGDFBiYqJ5O3r0aLaNDQAAAAAAPLlYEfWUMplMqlKliqpUqaJBgwapcOHC2rBhg3x9ffW///1Pbdq0uee5o0eP1r59+/TLL78oMjJSM2bMUIcOHe4Z7+DgIAcHh+wYBgAAAAAAeIpQiHoKbdmyRWvWrNFLL72kAgUKaMuWLTp9+rRKlCihoUOHqmfPnnJ3d1edOnV0/fp1bdu2TefPn1fv3r21c+dODRo0SAsXLlSVKlX06aefqlevXqpRo4aKFCmS00MDAAAAAABPMApRTyE3Nzf9+uuvGjdunJKSklS4cGGNGTNGdevWlSTlzp1bo0eP1jvvvCNnZ2cFBwcrJiZG165d06uvvqr27durQYMGkqQuXbrop59+0muvvaZff/1Vtra2OTk0AAAAAADwBDMZhmHkdBJ4uiQlJcnd3V1+Md/JxiG3xbG4kfVyKCsAAAAAAJAdUusAiYmJcnNzyzCWh5UDAAAAAADAKihEAQAAAAAAwCooRAEAAAAAAMAqKEQBAAAAAADAKihEAQAAAAAAwCooRAEAAAAAAMAqKEQBAAAAAADAKihEAQAAAAAAwCooRAEAAAAAAMAqKEQBAAAAAADAKuxyOgE8vXYPjZSbm1tOpwEAAAAAAB4TrIgCAAAAAACAVVCIAgAAAAAAgFVQiAIAAAAAAIBVUIgCAAAAAACAVVCIAgAAAAAAgFVQiAIAAAAAAIBVUIgCAAAAAACAVVCIAgAAAAAAgFVQiEK2mzRpkvz9/eXo6KiwsDBt3bo1w/gFCxYoKChIjo6OCg4O1rJlyyyOG4ahQYMGycfHR05OToqIiNCBAwcsYs6dO6c2bdrIzc1NHh4eio6O1qVLlyxi/vzzT1WrVk2Ojo7y8/PTqFGjsmbAAAAAAAAgXRSikK3mz5+v3r17a/DgwdqxY4fKli2ryMhInTp1Kt34jRs3qnXr1oqOjtbOnTsVFRWlqKgo7d692xwzatQojR8/XpMnT9aWLVvk7OysyMhIXbt2zRzTpk0b7dmzR6tWrdLSpUv166+/qkuXLubjSUlJeumll1S4cGFt375do0eP1pAhQzRlypTsuxgAAAAAADzjTIZhGDmdBJ4uSUlJcnd3V2Jiol588UVVqFBBEydOlCSlpKTIz89PPXr0UP/+/dOc27JlS12+fFlLly4176tUqZJCQkI0efJkGYYhX19fvf322+rTp48kKTExUV5eXpo5c6ZatWqlvXv3qmTJkvrjjz9Uvnx5SdKKFSv08ssv69ixY/L19dUXX3yh9957TwkJCbK3t5ck9e/fX0uWLNG+ffuy+xIBAAAAAPDUuLMO4ObmlmEsK6KQbW7cuKHt27crIiLCvM/GxkYRERHatGlTuuds2rTJIl6SIiMjzfGHDx9WQkKCRYy7u7vCwsLMMZs2bZKHh4e5CCVJERERsrGx0ZYtW8wx1atXNxehUvvZv3+/zp8//4gjBwAAAAAA6aEQhWxz9uxZJScny8vLy2K/l5eXEhIS0j0nISEhw/jUn/eLKVCggMVxOzs75c2b1yImvTbu7AMAAAAAAGQtClEAAAAAAACwCgpRyDb58uWTra2tTp48abH/5MmT8vb2Tvccb2/vDONTf94v5u6Hod+6dUvnzp2ziEmvjTv7AAAAAAAAWYtCFLKNvb29QkNDtWbNGvO+lJQUrVmzRuHh4emeEx4ebhEvSatWrTLHBwQEyNvb2yImKSlJW7ZsMceEh4frwoUL2r59uzlm7dq1SklJUVhYmDnm119/1c2bNy36KV68uPLkyfOIIwcAAAAAAOmhEIVs1bt3b02dOlWzZs3S3r179cYbb+jy5cvq0KGDJKlt27YaMGCAOb5Xr15asWKFxowZo3379mnIkCHatm2bunfvLkkymUyKiYnR8OHD9eOPP+qvv/5S27Zt5evrq6ioKElSiRIlVKdOHXXu3Flbt27Vhg0b1L17d7Vq1Uq+vr6SpFdeeUX29vaKjo7Wnj17NH/+fH322Wfq3bu3dS8QAAAAAADPELucTgBPt5YtW+r06dMaNGiQEhISFBISohUrVpgfDB4fHy8bm/+vh1auXFlz587V+++/r3fffVeBgYFasmSJSpcubY7p27evLl++rC5duujChQuqWrWqVqxYIUdHR3PMnDlz1L17d9WuXVs2NjZq2rSpxo8fbz7u7u6un3/+Wd26dVNoaKg8PT01aNAgdenSxQpXBQAAAACAZ5PJMAwjp5PA0yUpKUnu7u5KTEyUm5tbTqcDAAAAAACy0YPUAbg1DwAAAAAAAFZBIQqZkpycrJSUlJxOAwAAAAAAPMEoRD2BZs+erXz58un69esW+6OiovTaa69Jkn744Qe98MILcnR0VJEiRTR06FDdunXLHPvpp58qODhYzs7O8vPz05tvvqlLly6Zj8+cOVMeHh768ccfVbJkSTk4OCg+Pt46AwQAAAAAAE8lClFPoObNmys5OVk//vijed+pU6f0008/qWPHjvrtt9/Utm1b9erVS3///be+/PJLzZw5Ux9++KE53sbGRuPHj9eePXs0a9YsrV27Vn379rXo58qVK/r44481bdo07dmzRwUKFLDaGAEAAAAAwNOHh5U/od58803FxcVp2bJlkm6vcJo0aZIOHjyoF198UbVr19aAAQPM8d9884369u2r48ePp9vewoUL1bVrV505c0bS7RVRHTp0UGxsrMqWLZthLtevX7dYnZWUlCQ/Pz8eVg4AAAAAwDPgQR5WbmelnJDFOnfurAoVKujff/9VwYIFNXPmTLVv314mk0m7du3Shg0bLFZAJScn69q1a7py5Ypy586t1atXa8SIEdq3b5+SkpJ069Yti+OSZG9vrzJlytw3lxEjRmjo0KHZNlYAAAAAAPB04Na8J1S5cuVUtmxZzZ49W9u3b9eePXvUvn17SdKlS5c0dOhQxcbGmre//vpLBw4ckKOjo+Li4lS/fn2VKVNGixYt0vbt2zVp0iRJ0o0bN8x9ODk5yWQy3TeXAQMGKDEx0bwdPXo0W8YMAAAAAACebKyIeoJ16tRJ48aN07///quIiAj5+flJkl544QXt379fxYoVS/e87du3KyUlRWPGjJGNze1a5HfffffQeTg4OMjBweGhzwcAAAAAAM8GClFPsFdeeUV9+vTR1KlTNXv2bPP+QYMGqX79+ipUqJCaNWsmGxsb7dq1S7t379bw4cNVrFgx3bx5UxMmTFCDBg20YcMGTZ48OQdHAgAAAAAAngXcmvcEc3d3V9OmTeXi4qKoqCjz/sjISC1dulQ///yzKlSooEqVKmns2LEqXLiwJKls2bL69NNP9fHHH6t06dKaM2eORowYkUOjAAAAAAAAzwq+Ne8JV7t2bZUqVUrjx4/P6VTMHuRp+QAAAAAA4MnGt+Y9A86fP6/169dr/fr1+vzzz3M6HQAAAAAAgPuiEPWEKleunM6fP6+PP/5YxYsXz+l0AAAAAAAA7otC1BMqLi4up1MAAAAAAAB4IDysHAAAAAAAAFZBIQoAAAAAAABWQSEKAAAAAAAAVkEhCgAAAAAAAFZBIQoAAAAAAABWQSEKAAAAAAAAVkEhCgAAAAAAAFZBIQoAAAAAAABWQSEKAAAAAAAAVkEhCgAAAAAAAFZBIQoAAAAAAABWQSEKAAAAAAAAVkEhCgAAAAAAAFZBIQoAAAAAAABWQSEKAAAAAAAAVkEhCgAAAAAAAFZBIQoAAAAAAABWQSEKAAAAAAAAVkEhCgAAAAAAAFZBIQoAAAAAAABWYZfTCeDpYxiGJCkpKSmHMwEAAAAAANkt9d//qfWAjFCIQpY7e/asJMnPzy+HMwEAAAAAANZy8eJFubu7ZxhDIQpZLm/evJKk+Pj4+05APJuSkpLk5+eno0ePys3NLafTwWOIOYL7YY7gfpgjuB/mCO6HOYL7YY78P8MwdPHiRfn6+t43lkIUspyNze1Hj7m7uz/zv4zImJubG3MEGWKO4H6YI7gf5gjuhzmC+2GO4H6YI7dldiEKDysHAAAAAACAVVCIAgAAAAAAgFVQiEKWc3Bw0ODBg+Xg4JDTqeAxxRzB/TBHcD/MEdwPcwT3wxzB/TBHcD/MkYdjMjLz3XoAAAAAAADAI2JFFAAAAAAAAKyCQhQAAAAAAACsgkIUAAAAAAAArIJCFO5r0qRJ8vf3l6Ojo8LCwrR169YM4xcsWKCgoCA5OjoqODhYy5YtszhuGIYGDRokHx8fOTk5KSIiQgcOHMjOISCbZfUcWbx4sV566SXly5dPJpNJsbGx2Zg9rCEr58jNmzfVr18/BQcHy9nZWb6+vmrbtq2OHz+e3cNANsvqz5IhQ4YoKChIzs7OypMnjyIiIrRly5bsHAKyWVbPkTt17dpVJpNJ48aNy+KsYU1ZPUfat28vk8lksdWpUyc7h4Bslh2fI3v37lXDhg3l7u4uZ2dnVahQQfHx8dk1BGSzrJ4jd3+GpG6jR4/OzmE83gwgA/PmzTPs7e2N6dOnG3v27DE6d+5seHh4GCdPnkw3fsOGDYatra0xatQo4++//zbef/99I1euXMZff/1ljhk5cqTh7u5uLFmyxNi1a5fRsGFDIyAgwLh69aq1hoUslB1zZPbs2cbQoUONqVOnGpKMnTt3Wmk0yA5ZPUcuXLhgREREGPPnzzf27dtnbNq0yahYsaIRGhpqzWEhi2XHZ8mcOXOMVatWGYcOHTJ2795tREdHG25ubsapU6esNSxkoeyYI6kWL15slC1b1vD19TXGjh2bzSNBdsmOOdKuXTujTp06xokTJ8zbuXPnrDUkZLHsmCMHDx408ubNa7zzzjvGjh07jIMHDxo//PDDPdvE4y075sidnx8nTpwwpk+fbphMJuPQoUPWGtZjh0IUMlSxYkWjW7du5tfJycmGr6+vMWLEiHTjW7RoYdSrV89iX1hYmPH6668bhmEYKSkphre3tzF69Gjz8QsXLhgODg7Gt99+mw0jQHbL6jlyp8OHD1OIegpk5xxJtXXrVkOSceTIkaxJGlZnjXmSmJhoSDJWr16dNUnDqrJrjhw7dswoWLCgsXv3bqNw4cIUop5g2TFH2rVrZzRq1Chb8oX1ZcccadmypfHqq69mT8KwOmv8faRRo0bGf/7zn6xJ+AnFrXm4pxs3bmj79u2KiIgw77OxsVFERIQ2bdqU7jmbNm2yiJekyMhIc/zhw4eVkJBgEePu7q6wsLB7tonHV3bMETxdrDVHEhMTZTKZ5OHhkSV5w7qsMU9u3LihKVOmyN3dXWXLls265GEV2TVHUlJS9Nprr+mdd95RqVKlsid5WEV2fo6sX79eBQoUUPHixfXGG2/o7NmzWT8AZLvsmCMpKSn66aef9PzzzysyMlIFChRQWFiYlixZkm3jQPaxxt9HTp48qZ9++knR0dFZl/gTiEIU7unMmTNKTk6Wl5eXxX4vLy8lJCSke05CQkKG8ak/H6RNPL6yY47g6WKNOXLt2jX169dPrVu3lpubW9YkDqvKznmydOlSubi4yNHRUWPHjtWqVavk6emZtQNAtsuuOfLxxx/Lzs5OPXv2zPqkYVXZNUfq1Kmj2bNna82aNfr444/1yy+/qG7dukpOTs76QSBbZcccOXXqlC5duqSRI0eqTp06+vnnn9W4cWM1adJEv/zyS/YMBNnGGn9vnTVrllxdXdWkSZOsSfoJZZfTCQAA8LBu3rypFi1ayDAMffHFFzmdDh5DtWrVUmxsrM6cOaOpU6eqRYsW2rJliwoUKJDTqSGHbd++XZ999pl27Nghk8mU0+ngMdWqVSvzn4ODg1WmTBkVLVpU69evV+3atXMwMzwOUlJSJEmNGjXSW2+9JUkKCQnRxo0bNXnyZNWoUSMn08NjaPr06WrTpo0cHR1zOpUcxYoo3JOnp6dsbW118uRJi/0nT56Ut7d3uud4e3tnGJ/680HaxOMrO+YIni7ZOUdSi1BHjhzRqlWrWA31BMvOeeLs7KxixYqpUqVK+uqrr2RnZ6evvvoqaweAbJcdc+S3337TqVOnVKhQIdnZ2cnOzk5HjhzR22+/LX9//2wZB7KPtf5OUqRIEXl6eurgwYOPnjSsKjvmiKenp+zs7FSyZEmLmBIlSvCteU+g7P4c+e2337R//3516tQp65J+QlGIwj3Z29srNDRUa9asMe9LSUnRmjVrFB4enu454eHhFvGStGrVKnN8QECAvL29LWKSkpK0ZcuWe7aJx1d2zBE8XbJrjqQWoQ4cOKDVq1crX7582TMAWIU1P0tSUlJ0/fr1R08aVpUdc+S1117Tn3/+qdjYWPPm6+urd955RytXrsy+wSBbWOtz5NixYzp79qx8fHyyJnFYTXbMEXt7e1WoUEH79++3iPnnn39UuHDhLB4Bslt2f4589dVXCg0N5VmVkvjWPGRo3rx5hoODgzFz5kzj77//Nrp06WJ4eHgYCQkJhmEYxmuvvWb079/fHL9hwwbDzs7O+OSTT4y9e/cagwcPTvP1lSNHjjQ8PDyMH374wfjzzz+NRo0aGQEBAcbVq1etPj48uuyYI2fPnjV27txp/PTTT4YkY968ecbOnTuNEydOWH18eHRZPUdu3LhhNGzY0HjuueeM2NhYi6/DvX79eo6MEY8uq+fJpUuXjAEDBhibNm0y4uLijG3bthkdOnQwHBwcjN27d+fIGPFosuO/N3fjW/OebFk9Ry5evGj06dPH2LRpk3H48GFj9erVxgsvvGAEBgYa165dy5Ex4tFkx+fI4sWLjVy5chlTpkwxDhw4YEyYMMGwtbU1fvvtN6uPD48uu/5bk5iYaOTOndv44osvrDqexxWFKNzXhAkTjEKFChn29vZGxYoVjc2bN5uP1ahRw2jXrp1F/HfffWc8//zzhr29vVGqVCnjp59+sjiekpJiDBw40PDy8jIcHByM2rVrG/v377fGUJBNsnqOzJgxw5CUZhs8eLAVRoPskJVz5PDhw+nOD0nGunXrrDQiZIesnCdXr141GjdubPj6+hr29vaGj4+P0bBhQ2Pr1q3WGg6yQVb/9+ZuFKKefFk5R65cuWK89NJLRv78+Y1cuXIZhQsXNjp37mz+BymeTNnxOfLVV18ZxYoVMxwdHY2yZcsaS5Ysye5hIBtlxxz58ssvDScnJ+PChQvZnf4TwWQYhpEza7EAAAAAAADwLOEZUQAAAAAAALAKClEAAAAAAACwCgpRAAAAAAAAsAoKUQAAAAAAALAKClEAAAAAAACwCgpRAAAAAAAAsAoKUQAAAAAAALAKClEAAAAAAACwCgpRAAAAAAAAsAoKUQAA4J7at28vk8mkrl27pjnWrVs3mUwmtW/f3iL27q1OnTppzh0xYoRsbW01evToNMdmzpyZ7nkXLlyQyWTS+vXrM5X7nTk4OzsrMDBQ7du31/bt2y3i1q9fn27eJpNJCQkJ5rikpCS99957CgoKkqOjo7y9vRUREaHFixfLMAxJUs2aNc3nOjo66vnnn9eIESPMxyUpLi7unv1t3rzZIrerV68qb9688vT01PXr19OMcdeuXWrYsKEKFCggR0dH+fv7q2XLljp16tQD95WemTNnysPDw+J1Vrw3qV5//XXZ2tpqwYIFaY61b99eUVFRafanvl8XLlww77tx44ZGjRqlsmXLKnfu3PL09FSVKlU0Y8YM3bx58755/Prrr2rQoIF8fX1lMpm0ZMmSTI8hODg43d8PSfr666/l4OCgM2fOSJKmTp2qsmXLysXFRR4eHipXrpxGjBiR6b7OnTunmJgYFS5cWPb29vL19VXHjh0VHx9vjrnX+526DRkyxDwvYmNj0/RRs2ZNxcTEWLxOr507x3znfjc3N1WoUEE//PCDRbvJyckaOXKkgoKC5OTkpLx58yosLEzTpk3L9PgBAE8HClEAACBDfn5+mjdvnq5evWred+3aNc2dO1eFChWyiK1Tp45OnDhhsX377bdp2pw+fbr69u2r6dOnp9unnZ2dVq9erXXr1j1S7jNmzNCJEye0Z88eTZo0SZcuXVJYWJhmz56dJnb//v1pci9QoICk24WWypUra/bs2RowYIB27NihX3/9VS1btlTfvn2VmJhobqdz5846ceKE9u/frwEDBmjQoEGaPHlymv5Wr16dpr/Q0FCLmEWLFqlUqVIKCgpKUxw5ffq0ateurbx582rlypXau3evZsyYIV9fX12+fPmB+8qsrHpvrly5onnz5mU4DzLjxo0bioyM1MiRI9WlSxdt3LhRW7duVbdu3TRhwgTt2bPnvm1cvnxZZcuW1aRJkx64/+jo6DS/H6lmzJihhg0bytPTU9OnT1dMTIx69uyp2NhYbdiwQX379tWlS5cy1c+5c+dUqVIlrV69WpMnT9bBgwc1b948HTx4UBUqVND//vc/SbJ4j8eNGyc3NzeLfX369HngMabO6Tu3UaNGpRnriRMntG3bNlWpUkXNmjXTX3/9ZT4+dOhQjR07VsOGDdPff/+tdevWqUuXLhYFRQDAs8EupxMAAACPtxdeeEGHDh3S4sWL1aZNG0nS4sWLVahQIQUEBFjEOjg4yNvbO8P2fvnlF129elUffPCBZs+erY0bN6py5coWMc7OzmrRooX69++vLVu2PHTuHh4e5nz8/f310ksvqV27durevbsaNGigPHnymGMLFChgsfrnTu+++67i4uL0zz//yNfX17z/+eefV+vWreXo6Gjelzt3bnOfHTp00MSJE7Vq1Sq98cYbFm3my5fvvtfqq6++0quvvirDMPTVV1+pZcuW5mMbNmxQYmKipk2bJju723+lCwgIUK1atdK0k5m+Miur3psFCxaoZMmS6t+/v3x9fXX06FH5+fk9cDvjxo3Tr7/+qm3btqlcuXLm/UWKFFHz5s1148aN+7ZRt25d1a1b94H7lqRXX31V/fr106JFi/Tqq6+a9x8+fFjr16/XsmXLJEk//vijWrRooejoaHNMqVKlMt3Pe++9p+PHj+vgwYPm97JQoUJauXKlAgMD1a1bNy1fvtzifXZ3d5fJZErz3qeu0MqsO+f0vaT+rnl7e2vYsGH67LPPtG7dOgUHB0u6Pf4333xTzZs3N59TtmzZB8oDAPB0YEUUAAC4r44dO2rGjBnm19OnT1eHDh0eqq2vvvpKrVu3Vq5cudS6dWt99dVX6cYNGTJEf/31lxYuXPhQ/dzLW2+9pYsXL2rVqlWZik9JSdG8efPUpk0biyJUKhcXF3Mh6E6GYei3337Tvn37ZG9v/8B5Hjp0SJs2bVKLFi3UokUL/fbbbzpy5Ij5uLe3t27duqXvv//e4tY/a8iK9ya1yObu7q66detq5syZD9XOnDlzFBERYVGESpUrVy45Ozs/dI6Z4enpqUaNGqVZ1TVz5kw999xzeumllyTdfr82b95s8R5m1p1z8O6CkJOTk958802tXLlS586de/iBZJFbt26Zf6fvnPfe3t5au3atTp8+nVOpAQAeExSiAADAfb366qv6/fffdeTIER05ckQbNmywWP2RaunSpXJxcbHYPvroI/PxpKQkLVy40Hzuq6++qu+++y7d25N8fX3Vq1cvvffee7p161aWjSUoKEjS7ecn3em5556zyDt1tcqZM2d0/vx583n38/nnn8vFxUUODg6qXr26UlJS1LNnzzRxlStXTnOt7jR9+nTVrVtXefLkUd68eRUZGWlRDKxUqZLeffddvfLKK/L09FTdunU1evRonTx58oH7elCP+t4cOHBAmzdvNq/wevXVVzVjxoyHKqgdOHAg0+9NdomOjtb69et1+PBhSbeLkLNmzVK7du1kY3P7r9uDBw+Wh4eH/P39Vbx4cbVv317fffedUlJS7tv+6dOndeHCBZUoUSLd4yVKlJBhGDp48OAD5Z3evPjtt9/SxKXO6Tu3OXPmWMS0bt3aPO/feust+fv7q0WLFubjn376qU6fPi1vb2+VKVNGXbt21fLlyx8oXwDA04FCFAAAuK/8+fOrXr16mjlzpmbMmKF69erJ09MzTVytWrUUGxtrsd35UONvv/1WRYsWNd+SExISosKFC2v+/Pnp9tuvXz+dPn36kZ4hdLfUYofJZLLY/9tvv1nknXpL1YMWR9q0aWN+BlDdunX13nvvpbn1UJLmz5+f5lqlSk5O1qxZsyyKfa+++qpmzpxpUbj48MMPlZCQoMmTJ6tUqVKaPHmygoKCLJ7Nc7++HtajvDfTp09XZGSkeQ69/PLLSkxM1Nq1ax+4LWuvBkvPiy++qOeee85cKFyzZo3i4+MtVg36+Pho06ZN+uuvv9SrVy/dunVL7dq1U506dTJVjJKyfqzpzYvy5cuniUud03duDRs2tIgZO3asYmNjtXz5cpUsWVLTpk1T3rx5zcdLliyp3bt3a/PmzerYsaNOnTqlBg0aqFOnTlk6JgDA449nRAEAgEzp2LGjunfvLkn3fKizs7OzihUrds82vvrqK+3Zs8fiVraUlBRNnz7d4tk5qTw8PDRgwAANHTpU9evXf8QR3LZ3715JSvN8q4CAgHSfEZU/f355eHho3759mWrf3d3dfA2+++47FStWTJUqVVJERIRFnJ+f3z2v1cqVK/Xvv/9aPBNKul2gWrNmjV588UXzvnz58ql58+Zq3ry5PvroI5UrV06ffPKJZs2alam+HtbDvjepRbaEhASLeZCc/H/t3V1Ik18cB/Bv06kjzaGZpiWZNkMXFTWjrEQNLLsQS2IUqV2UqYgGIUVdVKaidZE3QcVaRWpEuUjyKtAS8iLLKCoKYmpUUiSCks6Zv/9FuH/z2XzpZVZ8P7AL9zvPefdih/Oc8xUXL15EamoqAGDOnDkuX2Pr6+uDl5eX45U7nU435bH5XVQqFXJzc3H58mUcO3YMZrMZycnJWLx4sSKtXq+HXq9HQUEB9u/fjw0bNuDevXsuz/YaMzYHx+bueC9fvsSsWbOmPcau5oVGo1Gk+35OuxMWFoaYmBjExMTAbDYjPT0dL168cBz4D3zrJ4PBAIPBgJKSEly9ehW7d+/GkSNHFP+PRET07+KOKCIiIpqSzZs3Y3h4GHa7HWlpadN+/tmzZ2hvb0dLS4vTzoqWlha0tbW5XUwoKiqCSqVCTU3NzzYBABw3iY1fGHJHpVLBaDSitrYW79+/V8QHBgbcvp7m7++P4uJiHDx4cFq7WUwmE4xGo2IXitFodHumFvDtTJ7o6GjFrXm/y4+MTVNTE/r7+9HR0eHUtvr6ejQ0NDhuUYuNjcXz589hs9mcnn/8+DGioqKgVqsBADt37sTdu3fR0dGhKMtut3usL/bs2YO3b9+ioaEBFovF5cLqeHFxcQAwaR1VKhV27NiBuro69PT0OMUGBwdx9uxZpKWlOe1AmkkJCQlYtWoVysvLJ0w31fYTEdG/hTuiiIiIaEq8vLwcOzK8vLxcprHZbIofyt7e3pg7dy5MJhMSEhKwceNGxXMGgwEmkwmnTp1SxPz8/HD8+HEUFhZOu859fX3o6emBzWbD69evce7cOdy6dQtXrlxR7H76+PEjhoaGnL4LDg6GWq1GeXk5WlpasGbNGpSXl2P16tVQq9VobW1FZWUlHj586PbGvby8PJSVleHmzZvIyspyfP/582dFX2m1WvT396OxsRG3b9+GXq93imdnZyMzMxO9vb148OABrl27BqPRCJ1OBxFBY2MjmpqanM6Smqis72/7+xE/MjYmkwlbt25V3JgWFxeHAwcOoLa2FoWFhdi1axdOnDiB7OxslJaWIjAwEPfv38eZM2dQXV3teK6kpAR37txBamoqysrKsH79egQEBKC9vR1VVVUwmUxYsWLFhHUaGBhwOl/JarXiyZMnCAoKQmRk5JTaFRUVhZSUFOzbtw++vr7Ytm2bUzw/Px/h4eFISUnBggUL8OHDB5w8eRIhISFYu3btpPlXVFQ4dsNVV1dDr9fDarXi6NGjsNvtbncp/gpfvnxRzB9fX1+nWyfHKykpQWZmJkpLSxEREYGsrCwkJiZi3bp1CAsLg9VqxeHDh6HT6Wb8jC8iIvIwISIiInIjJydHMjIy3MYzMjIkJyfHkRaA4hMbGys2m02Cg4OlurraZT5VVVUyb948GR4eFrPZLIGBgU7xkZERiYuLEwDS3Nw8pbp/Xwc/Pz+Jjo6WnJwcefTokVO65uZml/UGIG1tbY50fX19cujQIVmyZIn4+PhIaGiobNq0SSwWi4yOjoqISFJSkhQXFyvqkpeXJ/Hx8fL161exWq1uy6uvr5fTp0+LVquV4eFhRT42m020Wq3U1NTImzdvZO/evaLT6USj0YhWqxWDwSBms9mRfrKyJjN+LH52bHp6esTb21uuX7/uMp6fny8rV650/P3q1SvJzMyU8PBwmT17tixfvlwuXLjg6O8xQ0NDUllZKcuWLRM/Pz8JCgqSxMREuXTpktjt9knb6W4OjM3tqaqrqxMAUlBQoIjduHFD0tPTZf78+eLj4yPh4eGyfft2efr06ZTz//TpkxQVFcnChQtFrVZLaGio5ObmSldXl8v0rsZL5P950dHRoYiNn8NJSUku+yYtLc2RBoBYLBanfEZHR2Xp0qWSn58vIiLnz5+X5ORkCQkJER8fH4mMjJTc3Fzp7OyccvuJiOjfMEvkDzjhkYiIiIiIiIiI/nk8I4qIiIiIiIiIiDyCC1FERET016moqIC/v7/Lz5YtW2a6en+N+Ph4t/1YW1v7Q3n+aWPT3d3ttj7+/v7o7u6e8PnW1tYJn/9VJiqjtbX1l5VDREQ00/hqHhEREf11ent70dvb6zKm0WgQERHh4Rr9nbq6umC3213GQkNDERAQMO08/7SxGRkZQWdnp9v4okWL4O3t/v6ewcFBvHv3zm08JibmZ6rn8P1h6eNFRERAo9H8knKIiIhmGheiiIiIiIiIiIjII/hqHhEREREREREReQQXooiIiIiIiIiIyCO4EEVERERERERERB7BhSgiIiIiIiIiIvIILkQREREREREREZFHcCGKiIiIiIiIiIg8ggtRRERERERERETkEVyIIiIiIiIiIiIij/gPVBmhUB5bdwsAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1200x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(12, 4))\n",
    "\n",
    "# Mean decrease in AUC of the class 1 vs the others.\n",
    "variable_importance_metric = \"MEAN_DECREASE_IN_AUC_1_VS_OTHERS\"\n",
    "variable_importances = inspector.variable_importances()[variable_importance_metric]\n",
    "\n",
    "# Extract the feature name and importance values.\n",
    "#\n",
    "# `variable_importances` is a list of <feature, importance> tuples.\n",
    "feature_names = [vi[0].name for vi in variable_importances]\n",
    "feature_importances = [vi[1] for vi in variable_importances]\n",
    "# The feature are ordered in decreasing importance value.\n",
    "feature_ranks = range(len(feature_names))\n",
    "\n",
    "bar = plt.barh(feature_ranks, feature_importances, label=[str(x) for x in feature_ranks])\n",
    "plt.yticks(feature_ranks, feature_names)\n",
    "plt.gca().invert_yaxis()\n",
    "\n",
    "# TODO: Replace with \"plt.bar_label()\" when available.\n",
    "# Label each bar with values\n",
    "for importance, patch in zip(feature_importances, bar.patches):\n",
    "  plt.text(patch.get_x() + patch.get_width(), patch.get_y(), f\"{importance:.4f}\", va=\"top\")\n",
    "\n",
    "plt.xlabel(variable_importance_metric)\n",
    "plt.title(\"Mean decrease in AUC of the class 1 vs the others\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q_-kLTNjhaQo"
   },
   "source": [
    "Finally, access the actual tree structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.783723Z",
     "iopub.status.busy": "2024-04-20T11:24:54.783235Z",
     "iopub.status.idle": "2024-04-20T11:24:54.789974Z",
     "shell.execute_reply": "2024-04-20T11:24:54.789148Z"
    },
    "id": "l4N_heuzhcUS"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Tree(root=NonLeafNode(condition=(bill_length_mm >= 43.25; miss=True, score=0.5482327342033386), pos_child=NonLeafNode(condition=(island in ['Biscoe']; miss=True, score=0.6515106558799744), pos_child=NonLeafNode(condition=(bill_depth_mm >= 17.225584030151367; miss=False, score=0.027205035090446472), pos_child=LeafNode(value=ProbabilityValue([0.16666666666666666, 0.0, 0.8333333333333334],n=6.0), idx=7), neg_child=LeafNode(value=ProbabilityValue([0.0, 0.0, 1.0],n=104.0), idx=6), value=ProbabilityValue([0.00909090909090909, 0.0, 0.990909090909091],n=110.0)), neg_child=LeafNode(value=ProbabilityValue([0.0, 1.0, 0.0],n=61.0), idx=5), value=ProbabilityValue([0.005847953216374269, 0.3567251461988304, 0.6374269005847953],n=171.0)), neg_child=NonLeafNode(condition=(bill_depth_mm >= 15.100000381469727; miss=True, score=0.150658518075943), pos_child=NonLeafNode(condition=(flipper_length_mm >= 187.5; miss=True, score=0.036139510571956635), pos_child=LeafNode(value=ProbabilityValue([1.0, 0.0, 0.0],n=104.0), idx=4), neg_child=NonLeafNode(condition=(bill_length_mm >= 42.30000305175781; miss=True, score=0.23430533707141876), pos_child=LeafNode(value=ProbabilityValue([0.0, 1.0, 0.0],n=5.0), idx=3), neg_child=NonLeafNode(condition=(bill_length_mm >= 40.55000305175781; miss=True, score=0.043961383402347565), pos_child=LeafNode(value=ProbabilityValue([0.8, 0.2, 0.0],n=5.0), idx=2), neg_child=LeafNode(value=ProbabilityValue([1.0, 0.0, 0.0],n=53.0), idx=1), value=ProbabilityValue([0.9827586206896551, 0.017241379310344827, 0.0],n=58.0)), value=ProbabilityValue([0.9047619047619048, 0.09523809523809523, 0.0],n=63.0)), value=ProbabilityValue([0.9640718562874252, 0.03592814371257485, 0.0],n=167.0)), neg_child=LeafNode(value=ProbabilityValue([0.0, 0.0, 1.0],n=6.0), idx=0), value=ProbabilityValue([0.930635838150289, 0.03468208092485549, 0.03468208092485549],n=173.0)), value=ProbabilityValue([0.47093023255813954, 0.19476744186046513, 0.33430232558139533],n=344.0)), label_classes=None)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inspector.extract_tree(tree_idx=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B8u_0p80hoeP"
   },
   "source": [
    "Extracting a tree is not efficient. If speed is important, the model inspection can be done with the `iterate_on_nodes()` method instead. This method is a Depth First Pre-order traversals iterator on all the nodes of the model.\n",
    "\n",
    "**Note:** `extract_tree()` is implemented using `iterate_on_nodes()`.\n",
    "\n",
    "For following example computes how many times each feature is used (this is a\n",
    "kind of structural variable importance):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.793602Z",
     "iopub.status.busy": "2024-04-20T11:24:54.793006Z",
     "iopub.status.idle": "2024-04-20T11:24:54.908214Z",
     "shell.execute_reply": "2024-04-20T11:24:54.907390Z"
    },
    "id": "OUEpes34iHg8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of condition nodes per features:\n",
      "\t bill_length_mm : 778\n",
      "\t bill_depth_mm : 463\n",
      "\t flipper_length_mm : 414\n",
      "\t island : 342\n",
      "\t body_mass_g : 338\n",
      "\t year : 19\n",
      "\t sex : 36\n"
     ]
    }
   ],
   "source": [
    "# number_of_use[F] will be the number of node using feature F in its condition.\n",
    "number_of_use = collections.defaultdict(lambda: 0)\n",
    "\n",
    "# Iterate over all the nodes in a Depth First Pre-order traversals.\n",
    "for node_iter in inspector.iterate_on_nodes():\n",
    "\n",
    "  if not isinstance(node_iter.node, tfdf.py_tree.node.NonLeafNode):\n",
    "    # Skip the leaf nodes\n",
    "    continue\n",
    "\n",
    "  # Iterate over all the features used in the condition.\n",
    "  # By default, models are \"oblique\" i.e. each node tests a single feature.\n",
    "  for feature in node_iter.node.condition.features():\n",
    "    number_of_use[feature] += 1\n",
    "\n",
    "print(\"Number of condition nodes per features:\")\n",
    "for feature, count in number_of_use.items():\n",
    "  print(\"\\t\", feature.name, \":\", count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CD39OmGbnPww"
   },
   "source": [
    "## Creating a model by hand\n",
    "\n",
    "In this section you will create a small Random Forest model by hand. To make it\n",
    "extra easy, the model will only contain one simple tree:\n",
    "\n",
    "```\n",
    "3 label classes: Red, blue and green.\n",
    "2 features: f1 (numerical) and f2 (string categorical)\n",
    "\n",
    "f1>=1.5\n",
    "    ├─(pos)─ f2 in [\"cat\",\"dog\"]\n",
    "    │         ├─(pos)─ value: [0.8, 0.1, 0.1]\n",
    "    │         └─(neg)─ value: [0.1, 0.8, 0.1]\n",
    "    └─(neg)─ value: [0.1, 0.1, 0.8]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.911704Z",
     "iopub.status.busy": "2024-04-20T11:24:54.911246Z",
     "iopub.status.idle": "2024-04-20T11:24:54.915974Z",
     "shell.execute_reply": "2024-04-20T11:24:54.915135Z"
    },
    "id": "fGGe5IxdnuEa"
   },
   "outputs": [],
   "source": [
    "# Create the model builder\n",
    "builder = tfdf.builder.RandomForestBuilder(\n",
    "    path=\"/tmp/manual_model\",\n",
    "    objective=tfdf.py_tree.objective.ClassificationObjective(\n",
    "        label=\"color\", classes=[\"red\", \"blue\", \"green\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DRnJ2u-Moqbf"
   },
   "source": [
    "Each tree is added one by one.\n",
    "\n",
    "**Note:** The tree object (`tfdf.py_tree.tree.Tree`) is the same as the one returned by `extract_tree()` in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.919723Z",
     "iopub.status.busy": "2024-04-20T11:24:54.919301Z",
     "iopub.status.idle": "2024-04-20T11:24:54.926196Z",
     "shell.execute_reply": "2024-04-20T11:24:54.925349Z"
    },
    "id": "cmAddPhAo0tG"
   },
   "outputs": [],
   "source": [
    "# So alias\n",
    "Tree = tfdf.py_tree.tree.Tree\n",
    "SimpleColumnSpec = tfdf.py_tree.dataspec.SimpleColumnSpec\n",
    "ColumnType = tfdf.py_tree.dataspec.ColumnType\n",
    "# Nodes\n",
    "NonLeafNode = tfdf.py_tree.node.NonLeafNode\n",
    "LeafNode = tfdf.py_tree.node.LeafNode\n",
    "# Conditions\n",
    "NumericalHigherThanCondition = tfdf.py_tree.condition.NumericalHigherThanCondition\n",
    "CategoricalIsInCondition = tfdf.py_tree.condition.CategoricalIsInCondition\n",
    "# Leaf values\n",
    "ProbabilityValue = tfdf.py_tree.value.ProbabilityValue\n",
    "\n",
    "builder.add_tree(\n",
    "    Tree(\n",
    "        NonLeafNode(\n",
    "            condition=NumericalHigherThanCondition(\n",
    "                feature=SimpleColumnSpec(name=\"f1\", type=ColumnType.NUMERICAL),\n",
    "                threshold=1.5,\n",
    "                missing_evaluation=False),\n",
    "            pos_child=NonLeafNode(\n",
    "                condition=CategoricalIsInCondition(\n",
    "                    feature=SimpleColumnSpec(name=\"f2\",type=ColumnType.CATEGORICAL),\n",
    "                    mask=[\"cat\", \"dog\"],\n",
    "                    missing_evaluation=False),\n",
    "                pos_child=LeafNode(value=ProbabilityValue(probability=[0.8, 0.1, 0.1], num_examples=10)),\n",
    "                neg_child=LeafNode(value=ProbabilityValue(probability=[0.1, 0.8, 0.1], num_examples=20))),\n",
    "            neg_child=LeafNode(value=ProbabilityValue(probability=[0.1, 0.1, 0.8], num_examples=30)))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DjWdgRNNqEAD"
   },
   "source": [
    "Conclude the tree writing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:54.929661Z",
     "iopub.status.busy": "2024-04-20T11:24:54.929193Z",
     "iopub.status.idle": "2024-04-20T11:24:55.946091Z",
     "shell.execute_reply": "2024-04-20T11:24:55.945132Z"
    },
    "id": "cJqn4khxqH6t"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO 24-04-20 11:24:54.9480 UTC kernel.cc:1233] Loading model from path /tmp/manual_model/tmp/ with prefix f938aac6d7ed44f5\n",
      "[INFO 24-04-20 11:24:54.9483 UTC decision_forest.cc:734] Model loaded with 1 root(s), 5 node(s), and 2 input feature(s).\n",
      "[INFO 24-04-20 11:24:54.9483 UTC kernel.cc:1061] Use fast generic engine\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /tmp/manual_model/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /tmp/manual_model/assets\n"
     ]
    }
   ],
   "source": [
    "builder.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_oxxXAn7qK-z"
   },
   "source": [
    "Now you can open the model as a regular keras model, and make predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:55.949899Z",
     "iopub.status.busy": "2024-04-20T11:24:55.949356Z",
     "iopub.status.idle": "2024-04-20T11:24:56.134350Z",
     "shell.execute_reply": "2024-04-20T11:24:56.133412Z"
    },
    "id": "ETwjOJ5uqP5i"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO 24-04-20 11:24:56.1029 UTC kernel.cc:1233] Loading model from path /tmp/manual_model/assets/ with prefix f938aac6d7ed44f5\n",
      "[INFO 24-04-20 11:24:56.1032 UTC decision_forest.cc:734] Model loaded with 1 root(s), 5 node(s), and 2 input feature(s).\n",
      "[INFO 24-04-20 11:24:56.1032 UTC kernel.cc:1061] Use fast generic engine\n"
     ]
    }
   ],
   "source": [
    "manual_model = tf_keras.models.load_model(\"/tmp/manual_model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:56.137655Z",
     "iopub.status.busy": "2024-04-20T11:24:56.137384Z",
     "iopub.status.idle": "2024-04-20T11:24:56.784427Z",
     "shell.execute_reply": "2024-04-20T11:24:56.783451Z"
    },
    "id": "qlC4N-LuqWWR"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "1/2 [==============>...............] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "2/2 [==============================] - 1s 3ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predictions:\n",
      " [[0.1 0.1 0.8]\n",
      " [0.8 0.1 0.1]\n",
      " [0.1 0.8 0.1]]\n"
     ]
    }
   ],
   "source": [
    "examples = tf.data.Dataset.from_tensor_slices({\n",
    "        \"f1\": [1.0, 2.0, 3.0],\n",
    "        \"f2\": [\"cat\", \"cat\", \"bird\"]\n",
    "    }).batch(2)\n",
    "\n",
    "predictions = manual_model.predict(examples)\n",
    "\n",
    "print(\"predictions:\\n\",predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mxJyp1mKFPXb"
   },
   "source": [
    "Access the structure:\n",
    "\n",
    "**Note:** Because the model is serialized-and-deserialized, you need to use an alternative but equivalent form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:56.788532Z",
     "iopub.status.busy": "2024-04-20T11:24:56.787846Z",
     "iopub.status.idle": "2024-04-20T11:24:56.810667Z",
     "shell.execute_reply": "2024-04-20T11:24:56.809680Z"
    },
    "id": "IjcyMHJUFO_B"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "yggdrasil_model_path: /tmp/manual_model/assets/\n",
      "Input features: [\"f1\" (1; #1), \"f2\" (4; #2)]\n"
     ]
    }
   ],
   "source": [
    "yggdrasil_model_path = manual_model.yggdrasil_model_path_tensor().numpy().decode(\"utf-8\")\n",
    "print(\"yggdrasil_model_path:\",yggdrasil_model_path)\n",
    "\n",
    "inspector = tfdf.inspector.make_inspector(yggdrasil_model_path)\n",
    "print(\"Input features:\", inspector.features())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "muW1hgmotx8J"
   },
   "source": [
    "And of course, you can plot this manually constructed model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-20T11:24:56.814141Z",
     "iopub.status.busy": "2024-04-20T11:24:56.813597Z",
     "iopub.status.idle": "2024-04-20T11:24:56.820910Z",
     "shell.execute_reply": "2024-04-20T11:24:56.820084Z"
    },
    "id": "bqahDVg3t1xM"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<script src=\"https://d3js.org/d3.v6.min.js\"></script>\n",
       "<div id=\"tree_plot_bff81f58c9724a39b1c6d1c39f9b5e00\"></div>\n",
       "<script>\n",
       "/*\n",
       " * Copyright 2021 Google LLC.\n",
       " * Licensed under the Apache License, Version 2.0 (the \"License\");\n",
       " * you may not use this file except in compliance with the License.\n",
       " * You may obtain a copy of the License at\n",
       " *\n",
       " *     https://www.apache.org/licenses/LICENSE-2.0\n",
       " *\n",
       " * Unless required by applicable law or agreed to in writing, software\n",
       " * distributed under the License is distributed on an \"AS IS\" BASIS,\n",
       " * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
       " * See the License for the specific language governing permissions and\n",
       " * limitations under the License.\n",
       " */\n",
       "\n",
       "/**\n",
       " *  Plotting of decision trees generated by TF-DF.\n",
       " *\n",
       " *  A tree is a recursive structure of node objects.\n",
       " *  A node contains one or more of the following components:\n",
       " *\n",
       " *    - A value: Representing the output of the node. If the node is not a leaf,\n",
       " *      the value is only present for analysis i.e. it is not used for\n",
       " *      predictions.\n",
       " *\n",
       " *    - A condition : For non-leaf nodes, the condition (also known as split)\n",
       " *      defines a binary test to branch to the positive or negative child.\n",
       " *\n",
       " *    - An explanation: Generally a plot showing the relation between the label\n",
       " *      and the condition to give insights about the effect of the condition.\n",
       " *\n",
       " *    - Two children : For non-leaf nodes, the children nodes. The first\n",
       " *      children (i.e. \"node.children[0]\") is the negative children (drawn in\n",
       " *      red). The second children is the positive one (drawn in green).\n",
       " *\n",
       " */\n",
       "\n",
       "/**\n",
       " * Plots a single decision tree into a DOM element.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!tree} raw_tree Recursive tree structure.\n",
       " * @param {string} canvas_id Id of the output dom element.\n",
       " */\n",
       "function display_tree(options, raw_tree, canvas_id) {\n",
       "  console.log(options);\n",
       "\n",
       "  // Determine the node placement.\n",
       "  const tree_struct = d3.tree().nodeSize(\n",
       "      [options.node_y_offset, options.node_x_offset])(d3.hierarchy(raw_tree));\n",
       "\n",
       "  // Boundaries of the node placement.\n",
       "  let x_min = Infinity;\n",
       "  let x_max = -x_min;\n",
       "  let y_min = Infinity;\n",
       "  let y_max = -x_min;\n",
       "\n",
       "  tree_struct.each(d => {\n",
       "    if (d.x > x_max) x_max = d.x;\n",
       "    if (d.x < x_min) x_min = d.x;\n",
       "    if (d.y > y_max) y_max = d.y;\n",
       "    if (d.y < y_min) y_min = d.y;\n",
       "  });\n",
       "\n",
       "  // Size of the plot.\n",
       "  const width = y_max - y_min + options.node_x_size + options.margin * 2;\n",
       "  const height = x_max - x_min + options.node_y_size + options.margin * 2 +\n",
       "      options.node_y_offset - options.node_y_size;\n",
       "\n",
       "  const plot = d3.select(canvas_id);\n",
       "\n",
       "  // Tool tip\n",
       "  options.tooltip = plot.append('div')\n",
       "                        .attr('width', 100)\n",
       "                        .attr('height', 100)\n",
       "                        .style('padding', '4px')\n",
       "                        .style('background', '#fff')\n",
       "                        .style('box-shadow', '4px 4px 0px rgba(0,0,0,0.1)')\n",
       "                        .style('border', '1px solid black')\n",
       "                        .style('font-family', 'sans-serif')\n",
       "                        .style('font-size', options.font_size)\n",
       "                        .style('position', 'absolute')\n",
       "                        .style('z-index', '10')\n",
       "                        .attr('pointer-events', 'none')\n",
       "                        .style('display', 'none');\n",
       "\n",
       "  // Create canvas\n",
       "  const svg = plot.append('svg').attr('width', width).attr('height', height);\n",
       "  const graph =\n",
       "      svg.style('overflow', 'visible')\n",
       "          .append('g')\n",
       "          .attr('font-family', 'sans-serif')\n",
       "          .attr('font-size', options.font_size)\n",
       "          .attr(\n",
       "              'transform',\n",
       "              () => `translate(${options.margin},${\n",
       "                  - x_min + options.node_y_offset / 2 + options.margin})`);\n",
       "\n",
       "  // Plot bounding box.\n",
       "  if (options.show_plot_bounding_box) {\n",
       "    svg.append('rect')\n",
       "        .attr('width', width)\n",
       "        .attr('height', height)\n",
       "        .attr('fill', 'none')\n",
       "        .attr('stroke-width', 1.0)\n",
       "        .attr('stroke', 'black');\n",
       "  }\n",
       "\n",
       "  // Draw the edges.\n",
       "  display_edges(options, graph, tree_struct);\n",
       "\n",
       "  // Draw the nodes.\n",
       "  display_nodes(options, graph, tree_struct);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Draw the nodes of the tree.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!graph} graph D3 search handle containing the graph.\n",
       " * @param {!tree_struct} tree_struct Structure of the tree (node placement,\n",
       " *     data, etc.).\n",
       " */\n",
       "function display_nodes(options, graph, tree_struct) {\n",
       "  const nodes = graph.append('g')\n",
       "                    .selectAll('g')\n",
       "                    .data(tree_struct.descendants())\n",
       "                    .join('g')\n",
       "                    .attr('transform', d => `translate(${d.y},${d.x})`);\n",
       "\n",
       "  nodes.append('rect')\n",
       "      .attr('x', 0.5)\n",
       "      .attr('y', 0.5)\n",
       "      .attr('width', options.node_x_size)\n",
       "      .attr('height', options.node_y_size)\n",
       "      .attr('stroke', 'lightgrey')\n",
       "      .attr('stroke-width', 1)\n",
       "      .attr('fill', 'white')\n",
       "      .attr('y', -options.node_y_size / 2);\n",
       "\n",
       "  // Brackets on the right of condition nodes without children.\n",
       "  non_leaf_node_without_children =\n",
       "      nodes.filter(node => node.data.condition != null && node.children == null)\n",
       "          .append('g')\n",
       "          .attr('transform', `translate(${options.node_x_size},0)`);\n",
       "\n",
       "  non_leaf_node_without_children.append('path')\n",
       "      .attr('d', 'M0,0 C 10,0 0,10 10,10')\n",
       "      .attr('fill', 'none')\n",
       "      .attr('stroke-width', 1.0)\n",
       "      .attr('stroke', '#F00');\n",
       "\n",
       "  non_leaf_node_without_children.append('path')\n",
       "      .attr('d', 'M0,0 C 10,0 0,-10 10,-10')\n",
       "      .attr('fill', 'none')\n",
       "      .attr('stroke-width', 1.0)\n",
       "      .attr('stroke', '#0F0');\n",
       "\n",
       "  const node_content = nodes.append('g').attr(\n",
       "      'transform',\n",
       "      `translate(0,${options.node_padding - options.node_y_size / 2})`);\n",
       "\n",
       "  node_content.append(node => create_node_element(options, node));\n",
       "}\n",
       "\n",
       "/**\n",
       " * Creates the D3 content for a single node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!node} node Node to draw.\n",
       " * @return {!d3} D3 content.\n",
       " */\n",
       "function create_node_element(options, node) {\n",
       "  // Output accumulator.\n",
       "  let output = {\n",
       "    // Content to draw.\n",
       "    content: d3.create('svg:g'),\n",
       "    // Vertical offset to the next element to draw.\n",
       "    vertical_offset: 0\n",
       "  };\n",
       "\n",
       "  // Conditions.\n",
       "  if (node.data.condition != null) {\n",
       "    display_condition(options, node.data.condition, output);\n",
       "  }\n",
       "\n",
       "  // Values.\n",
       "  if (node.data.value != null) {\n",
       "    display_value(options, node.data.value, output);\n",
       "  }\n",
       "\n",
       "  // Explanations.\n",
       "  if (node.data.explanation != null) {\n",
       "    display_explanation(options, node.data.explanation, output);\n",
       "  }\n",
       "\n",
       "  return output.content.node();\n",
       "}\n",
       "\n",
       "\n",
       "/**\n",
       " * Adds a single line of text inside of a node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {string} text Text to display.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_node_text(options, text, output) {\n",
       "  output.content.append('text')\n",
       "      .attr('x', options.node_padding)\n",
       "      .attr('y', output.vertical_offset)\n",
       "      .attr('alignment-baseline', 'hanging')\n",
       "      .text(text);\n",
       "  output.vertical_offset += 10;\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds a single line of text inside of a node with a tooltip.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {string} text Text to display.\n",
       " * @param {string} tooltip Text in the Tooltip.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_node_text_with_tooltip(options, text, tooltip, output) {\n",
       "  const item = output.content.append('text')\n",
       "                   .attr('x', options.node_padding)\n",
       "                   .attr('alignment-baseline', 'hanging')\n",
       "                   .text(text);\n",
       "\n",
       "  add_tooltip(options, item, () => tooltip);\n",
       "  output.vertical_offset += 10;\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds a tooltip to a dom element.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!dom} target Dom element to equip with a tooltip.\n",
       " * @param {!func} get_content Generates the html content of the tooltip.\n",
       " */\n",
       "function add_tooltip(options, target, get_content) {\n",
       "  function show(d) {\n",
       "    options.tooltip.style('display', 'block');\n",
       "    options.tooltip.html(get_content());\n",
       "  }\n",
       "\n",
       "  function hide(d) {\n",
       "    options.tooltip.style('display', 'none');\n",
       "  }\n",
       "\n",
       "  function move(d) {\n",
       "    options.tooltip.style('display', 'block');\n",
       "    options.tooltip.style('left', (d.pageX + 5) + 'px');\n",
       "    options.tooltip.style('top', d.pageY + 'px');\n",
       "  }\n",
       "\n",
       "  target.on('mouseover', show);\n",
       "  target.on('mouseout', hide);\n",
       "  target.on('mousemove', move);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds a condition inside of a node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!condition} condition Condition to display.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_condition(options, condition, output) {\n",
       "  threshold_format = d3.format('r');\n",
       "\n",
       "  if (condition.type === 'IS_MISSING') {\n",
       "    display_node_text(options, `${condition.attribute} is missing`, output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'IS_TRUE') {\n",
       "    display_node_text(options, `${condition.attribute} is true`, output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'NUMERICAL_IS_HIGHER_THAN') {\n",
       "    format = d3.format('r');\n",
       "    display_node_text(\n",
       "        options,\n",
       "        `${condition.attribute} >= ${threshold_format(condition.threshold)}`,\n",
       "        output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'CATEGORICAL_IS_IN') {\n",
       "    display_node_text_with_tooltip(\n",
       "        options, `${condition.attribute} in [...]`,\n",
       "        `${condition.attribute} in [${condition.mask}]`, output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'CATEGORICAL_SET_CONTAINS') {\n",
       "    display_node_text_with_tooltip(\n",
       "        options, `${condition.attribute} intersect [...]`,\n",
       "        `${condition.attribute} intersect [${condition.mask}]`, output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'NUMERICAL_SPARSE_OBLIQUE') {\n",
       "    display_node_text_with_tooltip(\n",
       "        options, `Sparse oblique split...`,\n",
       "        `[${condition.attributes}]*[${condition.weights}]>=${\n",
       "            threshold_format(condition.threshold)}`,\n",
       "        output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  display_node_text(\n",
       "      options, `Non supported condition ${condition.type}`, output);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds a value inside of a node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!value} value Value to display.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_value(options, value, output) {\n",
       "  if (value.type === 'PROBABILITY') {\n",
       "    const left_margin = 0;\n",
       "    const right_margin = 50;\n",
       "    const plot_width = options.node_x_size - options.node_padding * 2 -\n",
       "        left_margin - right_margin;\n",
       "\n",
       "    let cusum = Array.from(d3.cumsum(value.distribution));\n",
       "    cusum.unshift(0);\n",
       "    const distribution_plot = output.content.append('g').attr(\n",
       "        'transform', `translate(0,${output.vertical_offset + 0.5})`);\n",
       "\n",
       "    distribution_plot.selectAll('rect')\n",
       "        .data(value.distribution)\n",
       "        .join('rect')\n",
       "        .attr('height', 10)\n",
       "        .attr(\n",
       "            'x',\n",
       "            (d, i) =>\n",
       "                (cusum[i] * plot_width + left_margin + options.node_padding))\n",
       "        .attr('width', (d, i) => d * plot_width)\n",
       "        .style('fill', (d, i) => d3.schemeSet1[i]);\n",
       "\n",
       "    const num_examples =\n",
       "        output.content.append('g')\n",
       "            .attr('transform', `translate(0,${output.vertical_offset})`)\n",
       "            .append('text')\n",
       "            .attr('x', options.node_x_size - options.node_padding)\n",
       "            .attr('alignment-baseline', 'hanging')\n",
       "            .attr('text-anchor', 'end')\n",
       "            .text(`(${value.num_examples})`);\n",
       "\n",
       "    const distribution_details = d3.create('ul');\n",
       "    distribution_details.selectAll('li')\n",
       "        .data(value.distribution)\n",
       "        .join('li')\n",
       "        .append('span')\n",
       "        .text(\n",
       "            (d, i) =>\n",
       "                'class ' + i + ': ' + d3.format('.3%')(value.distribution[i]));\n",
       "\n",
       "    add_tooltip(options, distribution_plot, () => distribution_details.html());\n",
       "    add_tooltip(options, num_examples, () => 'Number of examples');\n",
       "\n",
       "    output.vertical_offset += 10;\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (value.type === 'REGRESSION') {\n",
       "    display_node_text(\n",
       "        options,\n",
       "        'value: ' + d3.format('r')(value.value) + ` (` +\n",
       "            d3.format('.6')(value.num_examples) + `)`,\n",
       "        output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (value.type === 'UPLIFT') {\n",
       "    display_node_text(\n",
       "        options,\n",
       "        'effect: ' + d3.format('r')(value.treatment_effect) + ` (` +\n",
       "            d3.format('.6')(value.num_examples) + `)`,\n",
       "        output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  display_node_text(options, `Non supported value ${value.type}`, output);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds an explanation inside of a node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!explanation} explanation Explanation to display.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_explanation(options, explanation, output) {\n",
       "  // Margin before the explanation.\n",
       "  output.vertical_offset += 10;\n",
       "\n",
       "  display_node_text(\n",
       "      options, `Non supported explanation ${explanation.type}`, output);\n",
       "}\n",
       "\n",
       "\n",
       "/**\n",
       " * Draw the edges of the tree.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!graph} graph D3 search handle containing the graph.\n",
       " * @param {!tree_struct} tree_struct Structure of the tree (node placement,\n",
       " *     data, etc.).\n",
       " */\n",
       "function display_edges(options, graph, tree_struct) {\n",
       "  // Draw an edge between a parent and a child node with a bezier.\n",
       "  function draw_single_edge(d) {\n",
       "    return 'M' + (d.source.y + options.node_x_size) + ',' + d.source.x + ' C' +\n",
       "        (d.source.y + options.node_x_size + options.edge_rounding) + ',' +\n",
       "        d.source.x + ' ' + (d.target.y - options.edge_rounding) + ',' +\n",
       "        d.target.x + ' ' + d.target.y + ',' + d.target.x;\n",
       "  }\n",
       "\n",
       "  graph.append('g')\n",
       "      .attr('fill', 'none')\n",
       "      .attr('stroke-width', 1.2)\n",
       "      .selectAll('path')\n",
       "      .data(tree_struct.links())\n",
       "      .join('path')\n",
       "      .attr('d', draw_single_edge)\n",
       "      .attr(\n",
       "          'stroke', d => (d.target === d.source.children[0]) ? '#0F0' : '#F00');\n",
       "}\n",
       "\n",
       "display_tree({\"margin\": 10, \"node_x_size\": 160, \"node_y_size\": 28, \"node_x_offset\": 180, \"node_y_offset\": 33, \"font_size\": 10, \"edge_rounding\": 20, \"node_padding\": 2, \"show_plot_bounding_box\": false, \"labels\": \"[\\\"red\\\", \\\"blue\\\", \\\"green\\\"]\"}, {\"condition\": {\"type\": \"NUMERICAL_IS_HIGHER_THAN\", \"attribute\": \"f1\", \"threshold\": 1.5}, \"children\": [{\"condition\": {\"type\": \"CATEGORICAL_IS_IN\", \"attribute\": \"f2\", \"mask\": [\"cat\", \"dog\"]}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.8, 0.1, 0.1], \"num_examples\": 10.0}}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.1, 0.8, 0.1], \"num_examples\": 20.0}}]}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.1, 0.1, 0.8], \"num_examples\": 30.0}}]}, \"#tree_plot_bff81f58c9724a39b1c6d1c39f9b5e00\")\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfdf.model_plotter.plot_model_in_colab(manual_model)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "advanced_colab.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}