Post_Training_Quantization.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "_DDaAex5Q7u-"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Y8E0lw5eYWm"
      },
      "source": [
        "# Post-training Quantization using Tensorflow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BTC1rDAuei_1"
      },
      "source": [
        "## Overview\n",
        "\n",
        "Integer quantization is an optimization strategy that converts 32-bit floating-point numbers (such as weights and activation outputs) to the nearest 8-bit fixed-point numbers. This results in a smaller model and increased inferencing speed, which is valuable for low-power devices such as microcontrollers.\n",
        "\n",
        "In this tutorial, you'll train an MNIST model, convert it into a Tensorflow Lite file, and quantize it using post-training quantization. Finally, you'll check the accuracy of the converted model and compare it to the original float model.\n",
        "\n",
        "\n",
        "\n",
        "You actually have several options as to how much you want to quantize a model. In this tutorial, you'll perform \"full integer quantization,\" which converts all weights and activation outputs into 8-bit integer data—whereas other strategies may leave some amount of data in floating-point.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dDqqUIZjZjac"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I0nR5AMEWq0H"
      },
      "source": [
        "In order to quantize both the input and output tensors, we need to use APIs added in TensorFlow r2.3:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WsN6s5L1ieNl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0226b6b5-2420-4619-f6fb-918849382a39"
      },
      "source": [
        "import logging\n",
        "logging.getLogger(\"tensorflow\").setLevel(logging.DEBUG)\n",
        "\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "print(\"TensorFlow version: \", tf.__version__)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TensorFlow version:  2.15.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2XsEP17Zelz9"
      },
      "source": [
        "## Generate a TensorFlow Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5NMaNZQCkW9X"
      },
      "source": [
        "We'll build a simple model to classify numbers from the [MNIST dataset](https://www.tensorflow.org/datasets/catalog/mnist).\n",
        "\n",
        "This training won't take long because you're training the model for just a 5 epochs, which trains to about ~98% accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eMsw_6HujaqM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "88e9b604-46f5-4b9c-ec80-8a9790239e66"
      },
      "source": [
        "# Load MNIST dataset\n",
        "mnist = tf.keras.datasets.mnist\n",
        "(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n",
        "\n",
        "# Normalize the input image so that each pixel value is between 0 to 1.\n",
        "train_images = train_images.astype(np.float32) / 255.0\n",
        "test_images = test_images.astype(np.float32) / 255.0\n",
        "\n",
        "# Define the model architecture\n",
        "model = tf.keras.Sequential([\n",
        "  tf.keras.layers.InputLayer(input_shape=(28, 28)),\n",
        "  tf.keras.layers.Reshape(target_shape=(28, 28, 1)),\n",
        "  tf.keras.layers.Conv2D(filters=12, kernel_size=(3, 3), activation='relu'),\n",
        "  tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),\n",
        "  tf.keras.layers.Flatten(),\n",
        "  tf.keras.layers.Dense(10)\n",
        "])\n",
        "\n",
        "# Train the digit classification model\n",
        "model.compile(optimizer='adam',\n",
        "              loss=tf.keras.losses.SparseCategoricalCrossentropy(\n",
        "                  from_logits=True),\n",
        "              metrics=['accuracy'])\n",
        "model.fit(\n",
        "  train_images,\n",
        "  train_labels,\n",
        "  epochs=5,\n",
        "  validation_data=(test_images, test_labels)\n",
        ")"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11490434/11490434 [==============================] - 1s 0us/step\n",
            "Epoch 1/5\n",
            "1875/1875 [==============================] - 9s 3ms/step - loss: 0.2941 - accuracy: 0.9170 - val_loss: 0.1369 - val_accuracy: 0.9613\n",
            "Epoch 2/5\n",
            "1875/1875 [==============================] - 5s 3ms/step - loss: 0.1199 - accuracy: 0.9664 - val_loss: 0.0998 - val_accuracy: 0.9692\n",
            "Epoch 3/5\n",
            "1875/1875 [==============================] - 5s 3ms/step - loss: 0.0866 - accuracy: 0.9748 - val_loss: 0.0776 - val_accuracy: 0.9758\n",
            "Epoch 4/5\n",
            "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0708 - accuracy: 0.9797 - val_loss: 0.0673 - val_accuracy: 0.9778\n",
            "Epoch 5/5\n",
            "1875/1875 [==============================] - 5s 3ms/step - loss: 0.0602 - accuracy: 0.9828 - val_loss: 0.0644 - val_accuracy: 0.9783\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.src.callbacks.History at 0x7ae8ac6b10f0>"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KuTEoGFYd8aM"
      },
      "source": [
        "## Convert to a TensorFlow Lite model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xl8_fzVAZwOh"
      },
      "source": [
        "Now you can convert the trained model to TensorFlow Lite format using the [`TFLiteConverter`](https://www.tensorflow.org/lite/convert/python_api) API, and apply varying degrees of quantization.\n",
        "\n",
        "First, here's a converted model with no quantization:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_i8B2nDZmAgQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "02c07cde-4a36-48a0-b748-76ad71930bdf"
      },
      "source": [
        "converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
        "\n",
        "tflite_model = converter.convert()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:tensorflow:Assets written to: /tmp/tmp65qs94bn/assets\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7BONhYtYocQY"
      },
      "source": [
        "It's now a TensorFlow Lite model, but it's still using 32-bit float values for all parameter data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FQgTqbvPvxGJ"
      },
      "source": [
        "### Convert using integer-only quantization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mwR9keYAwArA"
      },
      "source": [
        "To quantize the input and output tensors, and make the converter throw an error if it encounters an operation it cannot quantize, convert the model again with some additional parameters:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kzjEjcDs3BHa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8c346e21-e68b-4a4d-fef6-02cf8352695b"
      },
      "source": [
        "def representative_data_gen():\n",
        "  for input_value in tf.data.Dataset.from_tensor_slices(train_images).batch(1).take(100):\n",
        "    yield [input_value]\n",
        "\n",
        "converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
        "converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
        "converter.representative_dataset = representative_data_gen\n",
        "# Ensure that if any ops can't be quantized, the converter throws an error\n",
        "converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]\n",
        "# Set the input and output tensors to uint8 (APIs added in r2.3)\n",
        "converter.inference_input_type = tf.uint8\n",
        "converter.inference_output_type = tf.uint8\n",
        "\n",
        "tflite_model_quant = converter.convert()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:tensorflow:Assets written to: /tmp/tmp6ytl4dzm/assets\n",
            "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert.py:953: UserWarning: Statistics for quantized inputs were expected, but not specified; continuing anyway.\n",
            "  warnings.warn(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wYd6NxD03yjB"
      },
      "source": [
        "The internal quantization remains the same as above, but you can see the input and output tensors are now integer format:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PaNkOS-twz4k",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f131ae20-f727-4fed-e1c0-7950b10b98e2"
      },
      "source": [
        "interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)\n",
        "input_type = interpreter.get_input_details()[0]['dtype']\n",
        "print('input: ', input_type)\n",
        "output_type = interpreter.get_output_details()[0]['dtype']\n",
        "print('output: ', output_type)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "input:  <class 'numpy.uint8'>\n",
            "output:  <class 'numpy.uint8'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TO17AP84wzBb"
      },
      "source": [
        "Now you have an integer quantized model that uses integer data for the model's input and output tensors, so it's compatible with integer-only hardware such as the [Edge TPU](https://coral.ai)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sse224YJ4KMm"
      },
      "source": [
        "### Save the models as files"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4_9nZ4nv4b9P"
      },
      "source": [
        "You'll need a `.tflite` file to deploy your model on other devices. So let's save the converted models to files and then load them when we run inferences below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BEY59dC14uRv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dbf312cc-aa2d-4d83-8e48-b9e6eea17674"
      },
      "source": [
        "import pathlib\n",
        "\n",
        "tflite_models_dir = pathlib.Path(\"/tmp/mnist_tflite_models/\")\n",
        "tflite_models_dir.mkdir(exist_ok=True, parents=True)\n",
        "\n",
        "# Save the unquantized/float model:\n",
        "tflite_model_file = tflite_models_dir/\"mnist_model.tflite\"\n",
        "tflite_model_file.write_bytes(tflite_model)\n",
        "# Save the quantized model:\n",
        "tflite_model_quant_file = tflite_models_dir/\"mnist_model_quant.tflite\"\n",
        "tflite_model_quant_file.write_bytes(tflite_model_quant)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "24608"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9t9yaTeF9fyM"
      },
      "source": [
        "## Run the TensorFlow Lite models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L8lQHMp_asCq"
      },
      "source": [
        "Now we'll run inferences using the TensorFlow Lite [`Interpreter`](https://www.tensorflow.org/api_docs/python/tf/lite/Interpreter) to compare the model accuracies.\n",
        "\n",
        "First, we need a function that runs inference with a given model and images, and then returns the predictions:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X092SbeWfd1A"
      },
      "source": [
        "# Helper function to run inference on a TFLite model\n",
        "def run_tflite_model(tflite_file, test_image_indices):\n",
        "  global test_images\n",
        "\n",
        "  # Initialize the interpreter\n",
        "  interpreter = tf.lite.Interpreter(model_path=str(tflite_file))\n",
        "  interpreter.allocate_tensors()\n",
        "\n",
        "  input_details = interpreter.get_input_details()[0]\n",
        "  output_details = interpreter.get_output_details()[0]\n",
        "\n",
        "  predictions = np.zeros((len(test_image_indices),), dtype=int)\n",
        "  for i, test_image_index in enumerate(test_image_indices):\n",
        "    test_image = test_images[test_image_index]\n",
        "    test_label = test_labels[test_image_index]\n",
        "\n",
        "    # Check if the input type is quantized, then rescale input data to uint8\n",
        "    if input_details['dtype'] == np.uint8:\n",
        "      input_scale, input_zero_point = input_details[\"quantization\"]\n",
        "      test_image = test_image / input_scale + input_zero_point\n",
        "\n",
        "    test_image = np.expand_dims(test_image, axis=0).astype(input_details[\"dtype\"])\n",
        "    interpreter.set_tensor(input_details[\"index\"], test_image)\n",
        "    interpreter.invoke()\n",
        "    output = interpreter.get_tensor(output_details[\"index\"])[0]\n",
        "\n",
        "    predictions[i] = output.argmax()\n",
        "\n",
        "  return predictions\n"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2opUt_JTdyEu"
      },
      "source": [
        "### Test the models on one image\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QpPpFPaz7eEM"
      },
      "source": [
        "Now we'll compare the performance of the float model and quantized model:\n",
        "+ `tflite_model_file` is the original TensorFlow Lite model with floating-point data.\n",
        "+ `tflite_model_quant_file` is the last model we converted using integer-only quantization (it uses uint8 data for input and output).\n",
        "\n",
        "Let's create another function to print our predictions:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zR2cHRUcUZ6e"
      },
      "source": [
        "import matplotlib.pylab as plt\n",
        "\n",
        "# Change this to test a different image\n",
        "test_image_index = 1\n",
        "\n",
        "## Helper function to test the models on one image\n",
        "def test_model(tflite_file, test_image_index, model_type):\n",
        "  global test_labels\n",
        "\n",
        "  predictions = run_tflite_model(tflite_file, [test_image_index])\n",
        "\n",
        "  plt.imshow(test_images[test_image_index])\n",
        "  template = model_type + \" Model \\n True:{true}, Predicted:{predict}\"\n",
        "  _ = plt.title(template.format(true= str(test_labels[test_image_index]), predict=str(predictions[0])))\n",
        "  plt.grid(False)"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A5OTJ_6Vcslt"
      },
      "source": [
        "Now test the float model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iTK0x980coto",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 473
        },
        "outputId": "229a6da7-d961-45be-d530-a0da397f3e7f"
      },
      "source": [
        "test_model(tflite_model_file, test_image_index, model_type=\"Float\")"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAHICAYAAAAIkT5uAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAs4ElEQVR4nO3deXxU1f3/8fcQkmExJMTsJUDCIrJFGyUgQkJJgeACiAsuFRBwaZAirlhlKdaotGhRKg/st4AKLqhISxV+sgRcAgiKFBEaMCwWEgKSBBIJWc7vD77M1yEhMMOEk4TX8/G4D5l7z5n7met98ObOPXOuwxhjBADABdbAdgEAgIsTAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAYR6q3Xr1hoxYoTtMmqF8zkWDodDU6ZM8Wk9gEQAoQ6aN2+eHA5HlcsTTzxxweooLi7WlClTlJGRcU7tMzIyXHW++eabVbbp2bOnHA6HOnfu7MNKgdqpoe0CAG/94Q9/UGxsrNu6C/kXd3FxsaZOnSpJSk5OPud+jRo10sKFC3XXXXe5rd+9e7e++OILNWrUyJdlArUWAYQ6KzU1VVdddZXtMjw2cOBA/eMf/9ChQ4cUGhrqWr9w4UJFRESoXbt2OnLkiMUKgQuDr+BwUfn+++91yy23KCQkRE2aNFH37t31r3/9y63NiRMnNGnSJCUkJCgoKEhNmzZVr169tHr1aleb3bt3KywsTJI0depU11dr53KvZNCgQXI6nVq0aJHb+oULF+rWW2+Vn59fpT5lZWWaNm2a2rRpI6fTqdatW+vJJ59USUmJWztjjJ555hm1aNFCTZo0UZ8+ffTtt99WWUd+fr7Gjx+vmJgYOZ1OtW3bVs8//7wqKirO+hkAXyCAUGcVFBTo0KFDbkt1cnNzdc0112j58uX67W9/qz/+8Y86fvy4brzxRi1evNjVrrCwUH/729+UnJys559/XlOmTFFeXp769++vzZs3S5LCwsL06quvSpKGDBmiN954Q2+88YZuuumms9bdpEkTDRo0SG+99ZZr3TfffKNvv/1Wd9xxR5V9Ro8erUmTJumXv/ylXnzxRSUlJSk9PV3Dhg1zazdp0iQ9/fTTio+P1/Tp0xUXF6d+/fqpqKjIrV1xcbGSkpL05ptv6u6779bMmTPVs2dPTZw4URMmTDjrZwB8wgB1zNy5c42kKpefa9WqlRk+fLjr9fjx440k8+mnn7rWHT161MTGxprWrVub8vJyY4wxZWVlpqSkxO29jhw5YiIiIsw999zjWpeXl2ckmcmTJ59T3atXrzaSzKJFi8zSpUuNw+Ewe/fuNcYY8+ijj5q4uDhjjDFJSUmmU6dOrn6bN282kszo0aPd3u+RRx4xksyqVauMMcYcPHjQBAQEmOuuu85UVFS42j355JNGktuxmDZtmmnatKn5z3/+4/aeTzzxhPHz83PVZYzx6DMCnuAKCHXWrFmz9Mknn7gt1fnoo4/UrVs3XXvtta51l1xyie69917t3r1b27ZtkyT5+fkpICBAklRRUaEff/xRZWVluuqqq/TVV1/5pPZ+/fopJCREb7/9towxevvtt3X77befsW5Jla5MHn74YUlyfYW4YsUKnThxQg8++KAcDoer3fjx4yu956JFi9SrVy81b97c7QoyJSVF5eXlWrt2rS8+JlAtBiGgzurWrZtHgxD27NmjxMTESusvv/xy1/ZTo+jmz5+vP//5z9q+fbtKS0tdbU8fdectf39/3XLLLVq4cKG6deumffv2nfHrtz179qhBgwZq27at2/rIyEgFBwdrz549rnaS1K5dO7d2YWFhat68udu6rKwsbdmyxXUf63QHDx706nMBniCAgNO8+eabGjFihAYPHqxHH31U4eHh8vPzU3p6unbt2uWz/dxxxx2aPXu2pkyZovj4eHXs2LHa9j+/qjlfFRUV+vWvf63HHnusyu3t27f32b6AMyGAcNFo1aqVduzYUWn99u3bXdsl6b333lNcXJw++OADt7/0J0+e7NbvfAPh2muvVcuWLZWRkaHnn3++2rorKiqUlZXlulqTTg6qyM/Pd9V96r9ZWVmKi4tztcvLy6s0rLtNmzY6duyYUlJSzuszAOeDe0C4aAwcOFAbNmxQZmama11RUZHmzJmj1q1bu65ATg2DNsa42q1fv96tn3RyNJt0cjizNxwOh2bOnKnJkyfrN7/5TbV1S9JLL73ktn7GjBmSpOuuu06SlJKSIn9/f7388stutZ/eT5JuvfVWZWZmavny5ZW25efnq6yszNOPA3iMKyBcNJ544gm99dZbSk1N1bhx4xQSEqL58+crOztb77//vho0OPnvseuvv14ffPCBhgwZouuuu07Z2dmaPXu2OnbsqGPHjrner3HjxurYsaPeeecdtW/fXiEhIercubNHszEMGjRIgwYNqrZNfHy8hg8frjlz5ig/P19JSUnasGGD5s+fr8GDB6tPnz6STt7reeSRR5Senq7rr79eAwcO1Ndff62PP/7Y7QevkvToo4/qH//4h66//nqNGDFCCQkJKioq0r///W+999572r17d6U+gM9ZHoUHeOzUMOwvv/yy2nanD8M2xphdu3aZm2++2QQHB5tGjRqZbt26maVLl7q1qaioMM8++6xp1aqVcTqd5sorrzRLly41w4cPN61atXJr+8UXX5iEhAQTEBBw1uHKPx+GXZ3Th2EbY0xpaamZOnWqiY2NNf7+/iYmJsZMnDjRHD9+3K1deXm5mTp1qomKijKNGzc2ycnJZuvWrVUei6NHj5qJEyeatm3bmoCAABMaGmquueYa86c//cmcOHHC1e5snwvwlsOYn12rAwBwgXAPCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggoB7IyMiQw+FQRkaGa92IESPUunVrazWdrqoacXEjgOATycnJrqeCVrecyxNDa9Lhw4c1ffp09e7dW2FhYQoODlb37t31zjvvnNf7nv75Q0JCdPXVV+vvf/97nXvC6LPPPqsPP/zQ2v6Li4s1a9Ys9evXT1FRUQoMDNSVV16pV199VeXl5dbqgu8xFQ984ve//71Gjx7tev3ll19q5syZevLJJ90m0OzatauN8lwyMzP1+9//XgMHDtRTTz2lhg0b6v3339ewYcO0bds2TZ061ev3btGihdLT0yWdnAD09ddf16hRo/Sf//xHzz33nK8+wjl77bXXvAq/Z599VjfffLMGDx7s+6LOwffff68HH3xQffv21YQJE9SsWTPXU2zXrVun+fPnW6kLNcD2VAyonxYtWmQkmdWrV1fb7tixYxemoP/1/fffm927d7utq6ioML/61a+M0+n0up6qps8pKioyLVq0ME2bNnWb2ubnysvLzU8//eTVPn/u1DQ/Zzve56Jp06aVpu3xhXOtMS8vz2zdurXS+pEjRxpJJisry+e1wQ6+gsMFM2XKFDkcDm3btk133HGHmjdv7no6aXJyspKTkyv1qeo+RkVFhV566SV16tRJjRo1UkREhO67775KjxwoKCjQ9u3bVVBQ4FoXGxvremzBKQ6HQ4MHD1ZJSYm+//5733xYnZwtu3v37ioqKlJeXp5rX2PHjtWCBQvUqVMnOZ1OLVu2TJL03//+V/fcc48iIiLkdDrVqVMn/f3vf6/0vj/88IMGDx6spk2bKjw8XA899JBKSkoqtTvTsfvLX/6iLl26qFGjRgoLC9OAAQO0ceNGV31FRUWaP3++6+vEESNGuPr7usbi4mJt375dhw4dcq0LDQ1Vp06dKrUdMmSIJOm7776rtA11E1/B4YK75ZZb1K5dOz377LNujw04V/fdd5/mzZunkSNHaty4ccrOztYrr7yir7/+Wp9//rn8/f0lSYsXL9bIkSM1d+5ct79Eq5KTkyNJPp8B+vvvv5efn5+Cg4Nd61atWqV3331XY8eOVWhoqFq3bq3c3Fx1797dFVBhYWH6+OOPNWrUKBUWFroeq/3TTz+pb9++2rt3r8aNG6fo6Gi98cYbWrVq1TnVM2rUKM2bN0+pqakaPXq0ysrK9Omnn2rdunW66qqr9MYbb2j06NHq1q2b7r33Xkknnx0kqUZq3LBhg/r06aPJkyef9f5gTf0/gkW2L8FQP1X1FdzkyZONJHP77bdXap+UlGSSkpIqrT99BupPP/3USDILFixwa7ds2bJK60/Nmj137txqaz18+LAJDw83vXr1OqfPVpWkpCTToUMHk5eXZ/Ly8sx3331nxo0bZySZG264wdVOkmnQoIH59ttv3fqPGjXKREVFmUOHDrmtHzZsmAkKCjLFxcXGGGNeeuklI8m8++67rjZFRUWmbdu2lY736cdu1apVRpIZN25cpforKipcfz7TV3A1UeOpr+XONtt2SUmJ6dixo4mNjTWlpaXVtkXdwVdwuODuv/9+r/suWrRIQUFB+vWvf61Dhw65loSEBF1yySVavXq1q+2IESNkjKn26qeiokJ33nmn8vPz9fLLL3tdl3TyyaphYWEKCwvT5ZdfrpdfflnXXXddpa+okpKS3B6/bYzR+++/rxtuuEHGGLfP1b9/fxUUFOirr76SJH300UeKiorSzTff7OrfpEkT19VKdd5//305HI5KT3aVzv5015qqMTk5WcaYs179jB07Vtu2bdMrr7yihg354qa+4P8kLrjY2Fiv+2ZlZamgoEDh4eFVbj948KBH7/fggw9q2bJlev311xUfH+91XZLUunVrvfbaa3I4HGrUqJHatWtXZZ2nf/68vDzl5+drzpw5mjNnTpXvfepz7dmzR23btq0UGJdddtlZ69u1a5eio6MVEhJyrh/pgtdYlenTp+u1117TtGnTXE+HRf1AAOGCa9y4caV1DoejyvtBp//uo6KiQuHh4VqwYEGV7x0WFnbOdUydOlV//etf9dxzz1X7SOxz1bRpU6WkpJy13emf/9RQ6bvuukvDhw+vso/t4eu2apw3b54ef/xx3X///Xrqqad8/v6wiwBCrdC8efMqR6Dt2bPH7XWbNm20YsUK9ezZs8ogO1ezZs3SlClTNH78eD3++ONev48vhIWFKTAwUOXl5WcNsFatWmnr1q0yxrhdYezYseOs+2nTpo2WL1+uH3/8sdqroKq+jrtQNf7ckiVLNHr0aN10002aNWuWR31RN3APCLVCmzZttH37dtdwZUn65ptv9Pnnn7u1u/XWW1VeXq5p06ZVeo+ysjLl5+e7Xlc1DFuS3nnnHY0bN0533nmnZsyY4dsP4gU/Pz8NHTpU77//vrZu3Vpp+8+PycCBA7V//3699957rnXFxcVn/Frs54YOHSpjTJU/tv351WfTpk3djmNN1ljVMGxJWrt2rYYNG6bevXtrwYIFatCAv6rqI66AUCvcc889mjFjhvr3769Ro0bp4MGDmj17tjp16qTCwkJXu6SkJN13331KT0/X5s2b1a9fP/n7+ysrK0uLFi3SX/7yF9fN76qGYW/YsEF33323Lr30UvXt27fSV3nXXHON4uLiXK8dDoeSkpJqfP6y5557TqtXr1ZiYqLGjBmjjh076scff9RXX32lFStW6Mcff5QkjRkzRq+88oruvvtubdq0SVFRUXrjjTfUpEmTs+6jT58++s1vfqOZM2cqKytLAwYMUEVFhT799FP16dNHY8eOlSQlJCRoxYoVmjFjhqKjoxUbG6vExMQaqbGqYdh79uzRjTfeKIfDoZtvvlmLFi1y69O1a1frX0nCRyyNvkM9V90w7Ly8vCr7vPnmmyYuLs4EBASYK664wixfvrzSUOJT5syZYxISEkzjxo1NYGCg6dKli3nsscfM/v37XW2qGoZ9at2Zlp+3PXr0qJFkhg0bdtbPW9VMCFWRZNLS0qrclpuba9LS0kxMTIzx9/c3kZGRpm/fvmbOnDlu7fbs2WNuvPFG06RJExMaGmp+97vfuYahVzcM2xhjysrKzPTp002HDh1MQECACQsLM6mpqWbTpk2uNtu3bze9e/c2jRs3NpLchmT7usaqhmGfWnem5WxDtlF3OIzx4peAwEXgo48+0vXXX69vvvlGXbp0sV0OUO/wxSpwBqtXr9awYcMIH6CGcAUEALCCKyAAgBUEEADACgIIAGAFAQQAsKLW/RC1oqJC+/fvV2Bg4Fln6AUA1D7GGB09elTR0dHVzmJR6wJo//79iomJsV0GAOA87du3Ty1atDjj9loXQIGBgZKkazVQDeVvuRoAgKfKVKrP9JHr7/MzqbEAmjVrlqZPn66cnBzFx8fr5ZdfVrdu3c7a79TXbg3lr4YOAggA6pz//XXp2W6j1MgghHfeeUcTJkzQ5MmT9dVXXyk+Pl79+/f3+GFhAID6q0YCaMaMGRozZoxGjhypjh07avbs2WrSpEmlRxMDAC5ePg+gEydOaNOmTW4PrWrQoIFSUlKUmZlZqX1JSYkKCwvdFgBA/efzADp06JDKy8sVERHhtj4iIkI5OTmV2qenpysoKMi1MAIOAC4O1n+IOnHiRBUUFLiWffv22S4JAHAB+HwUXGhoqPz8/JSbm+u2Pjc3V5GRkZXaO51OOZ1OX5cBAKjlfH4FFBAQoISEBK1cudK1rqKiQitXrlSPHj18vTsAQB1VI78DmjBhgoYPH66rrrpK3bp100svvaSioiKNHDmyJnYHAKiDaiSAbrvtNuXl5WnSpEnKycnRFVdcoWXLllUamAAAuHjVuieiFhYWKigoSMkaxEwIAFAHlZlSZWiJCgoK1KxZszO2sz4KDgBwcSKAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWNLRdAHA2u5/p4XGf8kbGq32FdcrzuE9m/Pte7ctTbVaN9LhP4IbGXu0rYuYXXvUDPMEVEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYwWSkuKCO/Kudx322XvFKDVTiO6XezXvqse19/uZxnwVXRXm1r3c/SfK4T/l3WV7tCxcvroAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAomI4XXvJlY9PMr3q6BSnxndn6cx31mZP7a4z6tW+V53Of/dfzA4z53Bh7wuI8k/XFEqMd94h5nMlJ4hisgAIAVBBAAwAqfB9CUKVPkcDjclg4dOvh6NwCAOq5G7gF16tRJK1as+L+dNORWEwDAXY0kQ8OGDRUZGVkTbw0AqCdq5B5QVlaWoqOjFRcXpzvvvFN79+49Y9uSkhIVFha6LQCA+s/nAZSYmKh58+Zp2bJlevXVV5Wdna1evXrp6NGjVbZPT09XUFCQa4mJifF1SQCAWsjnAZSamqpbbrlFXbt2Vf/+/fXRRx8pPz9f7777bpXtJ06cqIKCAteyb98+X5cEAKiFanx0QHBwsNq3b6+dO3dWud3pdMrpdNZ0GQCAWqbGfwd07Ngx7dq1S1FRUTW9KwBAHeLzAHrkkUe0Zs0a7d69W1988YWGDBkiPz8/3X777b7eFQCgDvP5V3A//PCDbr/9dh0+fFhhYWG69tprtW7dOoWFhfl6VwCAOsznAfT227V7sklUVtY3wat+q+JnedHL3+MeLx1p73Gf1bdd5XEfSdL+gx53aX9ko8d9GjRq5HGfZ9d38bjPk6H/9riPJJU1L/OqH+AJ5oIDAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACtq/IF0qP2O/SLAq34NvPj3izcTi2bc6PkknOXf7/C4z4W0c+qVHvdZGPJnL/bk3cMeWyzj36aoeZxlAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsILZsKHg1zO96nfzxrs87uM4Uuhxn7IDuz3uU9uNHrjC4z6XNPBuZmugtuIKCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsYDJSeK18239sl1Ar7P5jD4/7jAr+kxd7auRxj4cPdPdiP1Lgiu887lPu1Z5wMeMKCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsYDJS4Gfyf+P5xKKf3+35xKJBDTyfWDSzxM/jPpufudLjPpLUuHCDV/0AT3AFBACwggACAFjhcQCtXbtWN9xwg6Kjo+VwOPThhx+6bTfGaNKkSYqKilLjxo2VkpKirKwsX9ULAKgnPA6goqIixcfHa9asWVVuf+GFFzRz5kzNnj1b69evV9OmTdW/f38dP378vIsFANQfHg9CSE1NVWpqapXbjDF66aWX9NRTT2nQoEGSpNdff10RERH68MMPNWzYsPOrFgBQb/j0HlB2drZycnKUkpLiWhcUFKTExERlZmZW2aekpESFhYVuCwCg/vNpAOXk5EiSIiIi3NZHRES4tp0uPT1dQUFBriUmJsaXJQEAainro+AmTpyogoIC17Jv3z7bJQEALgCfBlBkZKQkKTc31219bm6ua9vpnE6nmjVr5rYAAOo/nwZQbGysIiMjtXLlSte6wsJCrV+/Xj16eP4LcwBA/eXxKLhjx45p586drtfZ2dnavHmzQkJC1LJlS40fP17PPPOM2rVrp9jYWD399NOKjo7W4MGDfVk3AKCO8ziANm7cqD59+rheT5gwQZI0fPhwzZs3T4899piKiop07733Kj8/X9dee62WLVumRo08n/sKAFB/OYwxxnYRP1dYWKigoCAla5AaOvxtl4OLzM4Xu3vcZ/utVf8o29faL7/P8z73bKyBSoDqlZlSZWiJCgoKqr2vb30UHADg4kQAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVHj+OAagLTnzSyqt+mR3+7EUvzx81Ep853OM+lz+8y+M+5R73AC4croAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAomI0Wt1zCutcd9prVd5NW+mjfwfGLRTSWe76fVNM+nCS0/csTzHQG1GFdAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFk5Gi1mvz7n897nNlwIX7t9XtK+/3uE/7b76sgUqAuoUrIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwgslIcUEdGd7D4z5TI/7sxZ6cXvSRhu9O8bjP5Y/t9LhPucc9gPqHKyAAgBUEEADACo8DaO3atbrhhhsUHR0th8OhDz/80G37iBEj5HA43JYBAwb4ql4AQD3hcQAVFRUpPj5es2bNOmObAQMG6MCBA67lrbfeOq8iAQD1j8eDEFJTU5WamlptG6fTqcjISK+LAgDUfzVyDygjI0Ph4eG67LLL9MADD+jw4cNnbFtSUqLCwkK3BQBQ//k8gAYMGKDXX39dK1eu1PPPP681a9YoNTVV5eVVDzxNT09XUFCQa4mJifF1SQCAWsjnvwMaNmyY689dunRR165d1aZNG2VkZKhv376V2k+cOFETJkxwvS4sLCSEAOAiUOPDsOPi4hQaGqqdO6v+sZ7T6VSzZs3cFgBA/VfjAfTDDz/o8OHDioqKquldAQDqEI+/gjt27Jjb1Ux2drY2b96skJAQhYSEaOrUqRo6dKgiIyO1a9cuPfbYY2rbtq369+/v08IBAHWbxwG0ceNG9enTx/X61P2b4cOH69VXX9WWLVs0f/585efnKzo6Wv369dO0adPkdHo3NxcAoH7yOICSk5NljDnj9uXLl59XQag7Gv4i2uM+vcat97jPJQ0u3D9eMre19bhP+yNf1kAlQP3HXHAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwwueP5MbF47snPX90+oeR/6yBSirr8+9bvOp3+WNVP7m3OuVe7QkAV0AAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAWTkcJrm2580YteTp/XUZWg31Z41a/syBEfVwLgTLgCAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArmIwU9VJpRJBX/fxP/MLHldhVnnfIq36mpMTjPg6n5xPN+oWFetzHG+VhwV71y3o4wLeF+JApd3jVr8ODOz3uU15Y6NW+zoYrIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwgslIUS/9672/2y6hVrjm69u96ncot5nHfZqHHfW4z/qEhR73wfnp+NRYj/vEPZZZA5VwBQQAsIQAAgBY4VEApaen6+qrr1ZgYKDCw8M1ePBg7dixw63N8ePHlZaWpksvvVSXXHKJhg4dqtzcXJ8WDQCo+zwKoDVr1igtLU3r1q3TJ598otLSUvXr109FRUWuNg899JD++c9/atGiRVqzZo3279+vm266yeeFAwDqNo8GISxbtszt9bx58xQeHq5Nmzapd+/eKigo0P/8z/9o4cKF+tWvfiVJmjt3ri6//HKtW7dO3bt3913lAIA67bzuARUUFEiSQkJCJEmbNm1SaWmpUlJSXG06dOigli1bKjOz6lEUJSUlKiwsdFsAAPWf1wFUUVGh8ePHq2fPnurcubMkKScnRwEBAQoODnZrGxERoZycnCrfJz09XUFBQa4lJibG25IAAHWI1wGUlpamrVu36u233z6vAiZOnKiCggLXsm/fvvN6PwBA3eDVD1HHjh2rpUuXau3atWrRooVrfWRkpE6cOKH8/Hy3q6Dc3FxFRkZW+V5Op1NOp9ObMgAAdZhHV0DGGI0dO1aLFy/WqlWrFBsb67Y9ISFB/v7+WrlypWvdjh07tHfvXvXo0cM3FQMA6gWProDS0tK0cOFCLVmyRIGBga77OkFBQWrcuLGCgoI0atQoTZgwQSEhIWrWrJkefPBB9ejRgxFwAAA3HgXQq6++KklKTk52Wz937lyNGDFCkvTiiy+qQYMGGjp0qEpKStS/f3/99a9/9UmxAID6w2GMMbaL+LnCwkIFBQUpWYPU0OFvuxxU46flsWdvdJqVnd+rgUpwMSk2JzzuU2oqaqCSqg3cMsLjPgWbQ31fyBlEfVbmcR/nx1961L7MlCpDS1RQUKBmzc48sS1zwQEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKr56ICkhS4/7ZHvfp9OxYj/uYWn6WBnb40eM+6xMW1kAlvtPp05Ee9zF7m9ZAJZXFvXfM804b/u37Qs6gubIuSJ/6gCsgAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCilk/ziPom9slM2yXUCtcrwXYJ1YrVFtsl4CLAFRAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWOFRAKWnp+vqq69WYGCgwsPDNXjwYO3YscOtTXJyshwOh9ty//33+7RoAEDd51EArVmzRmlpaVq3bp0++eQTlZaWql+/fioqKnJrN2bMGB04cMC1vPDCCz4tGgBQ9zX0pPGyZcvcXs+bN0/h4eHatGmTevfu7VrfpEkTRUZG+qZCAEC9dF73gAoKCiRJISEhbusXLFig0NBQde7cWRMnTlRxcfEZ36OkpESFhYVuCwCg/vPoCujnKioqNH78ePXs2VOdO3d2rb/jjjvUqlUrRUdHa8uWLXr88ce1Y8cOffDBB1W+T3p6uqZOneptGQCAOsphjDHedHzggQf08ccf67PPPlOLFi3O2G7VqlXq27evdu7cqTZt2lTaXlJSopKSEtfrwsJCxcTEKFmD1NDh701pAACLykypMrREBQUFatas2RnbeXUFNHbsWC1dulRr166tNnwkKTExUZLOGEBOp1NOp9ObMgAAdZhHAWSM0YMPPqjFixcrIyNDsbGxZ+2zefNmSVJUVJRXBQIA6iePAigtLU0LFy7UkiVLFBgYqJycHElSUFCQGjdurF27dmnhwoUaOHCgLr30Um3ZskUPPfSQevfura5du9bIBwAA1E0e3QNyOBxVrp87d65GjBihffv26a677tLWrVtVVFSkmJgYDRkyRE899VS13wP+XGFhoYKCgrgHBAB1VI3cAzpbVsXExGjNmjWevCUA4CLFXHAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsa2i7gdMYYSVKZSiVjuRgAgMfKVCrp//4+P5NaF0BHjx6VJH2mjyxXAgA4H0ePHlVQUNAZtzvM2SLqAquoqND+/fsVGBgoh8Phtq2wsFAxMTHat2+fmjVrZqlC+zgOJ3EcTuI4nMRxOKk2HAdjjI4eParo6Gg1aHDmOz217gqoQYMGatGiRbVtmjVrdlGfYKdwHE7iOJzEcTiJ43CS7eNQ3ZXPKQxCAABYQQABAKyoUwHkdDo1efJkOZ1O26VYxXE4ieNwEsfhJI7DSXXpONS6QQgAgItDnboCAgDUHwQQAMAKAggAYAUBBACwos4E0KxZs9S6dWs1atRIiYmJ2rBhg+2SLrgpU6bI4XC4LR06dLBdVo1bu3atbrjhBkVHR8vhcOjDDz90226M0aRJkxQVFaXGjRsrJSVFWVlZdoqtQWc7DiNGjKh0fgwYMMBOsTUkPT1dV199tQIDAxUeHq7Bgwdrx44dbm2OHz+utLQ0XXrppbrkkks0dOhQ5ebmWqq4ZpzLcUhOTq50Ptx///2WKq5anQigd955RxMmTNDkyZP11VdfKT4+Xv3799fBgwdtl3bBderUSQcOHHAtn332me2SalxRUZHi4+M1a9asKre/8MILmjlzpmbPnq3169eradOm6t+/v44fP36BK61ZZzsOkjRgwAC38+Ott966gBXWvDVr1igtLU3r1q3TJ598otLSUvXr109FRUWuNg899JD++c9/atGiRVqzZo3279+vm266yWLVvncux0GSxowZ43Y+vPDCC5YqPgNTB3Tr1s2kpaW5XpeXl5vo6GiTnp5usaoLb/LkySY+Pt52GVZJMosXL3a9rqioMJGRkWb69Omudfn5+cbpdJq33nrLQoUXxunHwRhjhg8fbgYNGmSlHlsOHjxoJJk1a9YYY07+v/f39zeLFi1ytfnuu++MJJOZmWmrzBp3+nEwxpikpCTzu9/9zl5R56DWXwGdOHFCmzZtUkpKimtdgwYNlJKSoszMTIuV2ZGVlaXo6GjFxcXpzjvv1N69e22XZFV2drZycnLczo+goCAlJiZelOdHRkaGwsPDddlll+mBBx7Q4cOHbZdUowoKCiRJISEhkqRNmzaptLTU7Xzo0KGDWrZsWa/Ph9OPwykLFixQaGioOnfurIkTJ6q4uNhGeWdU6yYjPd2hQ4dUXl6uiIgIt/URERHavn27parsSExM1Lx583TZZZfpwIEDmjp1qnr16qWtW7cqMDDQdnlW5OTkSFKV58epbReLAQMG6KabblJsbKx27dqlJ598UqmpqcrMzJSfn5/t8nyuoqJC48ePV8+ePdW5c2dJJ8+HgIAABQcHu7Wtz+dDVcdBku644w61atVK0dHR2rJlix5//HHt2LFDH3zwgcVq3dX6AML/SU1Ndf25a9euSkxMVKtWrfTuu+9q1KhRFitDbTBs2DDXn7t06aKuXbuqTZs2ysjIUN++fS1WVjPS0tK0devWi+I+aHXOdBzuvfde15+7dOmiqKgo9e3bV7t27VKbNm0udJlVqvVfwYWGhsrPz6/SKJbc3FxFRkZaqqp2CA4OVvv27bVz507bpVhz6hzg/KgsLi5OoaGh9fL8GDt2rJYuXarVq1e7Pb4lMjJSJ06cUH5+vlv7+no+nOk4VCUxMVGSatX5UOsDKCAgQAkJCVq5cqVrXUVFhVauXKkePXpYrMy+Y8eOadeuXYqKirJdijWxsbGKjIx0Oz8KCwu1fv36i/78+OGHH3T48OF6dX4YYzR27FgtXrxYq1atUmxsrNv2hIQE+fv7u50PO3bs0N69e+vV+XC241CVzZs3S1LtOh9sj4I4F2+//bZxOp1m3rx5Ztu2bebee+81wcHBJicnx3ZpF9TDDz9sMjIyTHZ2tvn8889NSkqKCQ0NNQcPHrRdWo06evSo+frrr83XX39tJJkZM2aYr7/+2uzZs8cYY8xzzz1ngoODzZIlS8yWLVvMoEGDTGxsrPnpp58sV+5b1R2Ho0ePmkceecRkZmaa7Oxss2LFCvPLX/7StGvXzhw/ftx26T7zwAMPmKCgIJORkWEOHDjgWoqLi11t7r//ftOyZUuzatUqs3HjRtOjRw/To0cPi1X73tmOw86dO80f/vAHs3HjRpOdnW2WLFli4uLiTO/evS1X7q5OBJAxxrz88sumZcuWJiAgwHTr1s2sW7fOdkkX3G233WaioqJMQECA+cUvfmFuu+02s3PnTttl1bjVq1cbSZWW4cOHG2NODsV++umnTUREhHE6naZv375mx44ddouuAdUdh+LiYtOvXz8TFhZm/P39TatWrcyYMWPq3T/Sqvr8kszcuXNdbX766Sfz29/+1jRv3tw0adLEDBkyxBw4cMBe0TXgbMdh7969pnfv3iYkJMQ4nU7Ttm1b8+ijj5qCggK7hZ+GxzEAAKyo9feAAAD1EwEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCs+P9+df1YP2FQMwAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o3N6-UGl1dfE"
      },
      "source": [
        "And test the quantized model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rc1i9umMcp0t",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 473
        },
        "outputId": "14257656-43d4-4ca5-a5ca-14771aad0be2"
      },
      "source": [
        "test_model(tflite_model_quant_file, test_image_index, model_type=\"Quantized\")"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAHICAYAAAAIkT5uAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAvpUlEQVR4nO3deXhUVZ7/8U8RSLEYwpJdAiSAIGGxpQVxYTE0EJRNsAVEASOCJiKgrYPSAm2PQZkBGkFs/LVEFFQQgcZBVJaAOEEFpWkaYQBZhYTNVEKCAZLz+4OhhiIJUGWKk4T363nuQ+rec+p+61rmk1v31LkOY4wRAADXWCXbBQAArk8EEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEHAVOnXqpE6dOl3TfaalpcnhcCgtLe2a7tcbDodDEydO9Lrfvn375HA4lJqaWuo1ofwggOB3//rXvzR48GDdeOONcjqdioqK0uDBg7V9+3bbpXnYvn27Jk6cqH379tkuxSupqalyOBxyOBzasGFDke3GGEVHR8vhcOi+++6zUCFQPAIIfvXxxx/r1ltv1erVqzVs2DC98cYbSkxM1Jo1a3Trrbdq2bJltkt02759uyZNmlRsAH3++ef6/PPPr31RXqhataoWLFhQZP26det06NAhOZ1OC1UBJatsuwBUXHv27NHDDz+s2NhYrV+/XqGhoe5tTz/9tO6++24NHjxYW7duVUxMjMVKrywwMNB2CVfUo0cPLVq0SDNmzFDlyv/3v/aCBQvUpk0bHT9+3GJ1QFGcAcFvpkyZory8PM2ZM8cjfCQpJCREf/3rX3Xq1ClNmTLFvX7o0KFq2LBhkeeaOHGiHA6Hx7q5c+fqnnvuUVhYmJxOp5o3b67Zs2cX6duwYUPdd9992rBhg9q2bauqVasqNjZW8+bNc7dJTU3VAw88IEnq3Lmz+yOtC9dfLr0G1LBhQ3ebS5eLr9n89NNPevTRRxUeHi6n06m4uDi9/fbbRWo8dOiQ+vTpoxo1aigsLExjxoxRfn5+ice2OAMHDtSJEyf0xRdfuNedOXNGH330kQYNGlRsn9zcXD3zzDOKjo6W0+lU06ZN9R//8R+6dJL8/Px8jRkzRqGhoQoKClKvXr106NChYp/zal8zwBkQ/Gb58uVq2LCh7r777mK3d+jQQQ0bNtTy5cv1xhtveP38s2fPVlxcnHr16qXKlStr+fLlevLJJ1VYWKikpCSPtrt371b//v2VmJioIUOG6O2339bQoUPVpk0bxcXFqUOHDho1apRmzJihF154QTfffLMkuf+91PTp03Xq1CmPddOmTdOWLVtUt25dSVJmZqZuv/12ORwOJScnKzQ0VJ9++qkSExOVnZ2t0aNHS5JOnz6t+Ph4HThwQKNGjVJUVJTeffddrVmzxqvj0bBhQ7Vv317vv/++EhISJEmffvqpXC6XBgwYoBkzZni0N8aoV69eWrt2rRITE3XLLbfos88+0x/+8Af99NNPmjZtmrvtY489pvfee0+DBg3SHXfcoTVr1ujee+8tUsPVvmZAkmQAP8jKyjKSTO/evS/brlevXkaSyc7ONsYYM2TIENOgQYMi7SZMmGAufbvm5eUVadetWzcTGxvrsa5BgwZGklm/fr173dGjR43T6TTPPPOMe92iRYuMJLN27doiz9uxY0fTsWPHEl/HwoULjSTzpz/9yb0uMTHRREZGmuPHj3u0HTBggAkODnbXP336dCPJLFy40N0mNzfXNG7cuMR6LjZ37lwjyXz77bdm5syZJigoyP3cDzzwgOncubP7ONx7773ufkuXLjWSzJ///GeP5+vfv79xOBxm9+7dxhhjtmzZYiSZJ5980qPdoEGDjCQzYcIEr1/z3r17jSQzd+7cy742VGx8BAe/yMnJkSQFBQVdtt2F7Rfae6NatWrun10ul44fP66OHTvqxx9/lMvl8mjbvHlzjzOx0NBQNW3aVD/++KPX+73U9u3b9eijj6p3794aP368pPNnF4sXL1bPnj1ljNHx48fdS7du3eRyufTdd99JklasWKHIyEj179/f/ZzVq1fX448/7nUtv//973X69Gl98sknysnJ0SeffFLix28rVqxQQECARo0a5bH+mWeekTFGn376qbudpCLtLj2b8eY1AxIfwcFPrjZYcnJy5HA4FBIS4vU+vvrqK02YMEHp6enKy8vz2OZyuRQcHOx+XL9+/SL9a9eurZ9//tnr/V4sOztb999/v2688UbNmzfPfZ3q2LFjysrK0pw5czRnzpxi+x49elSStH//fjVu3LjINa6mTZt6XU9oaKi6dOmiBQsWKC8vTwUFBR7BdrH9+/crKiqqyB8JFz523L9/v/vfSpUqqVGjRpetz5vXDEgEEPwkODhYUVFR2rp162Xbbd26VfXq1XOPMrv0l/AFBQUFHo/37Nmj+Ph4NWvWTFOnTlV0dLQCAwO1YsUKTZs2TYWFhR7tAwICin1e8yvvSD906FAdPnxY33zzjWrWrOlef2H/gwcP1pAhQ4rt26pVq1+175IMGjRIw4cPV0ZGhhISElSrVi2/7OdSNl8zyicCCH7Ts2dP/fWvf9WGDRt01113Fdn+5Zdfat++fRo7dqx7Xe3atZWVlVWk7YW/xi9Yvny58vPz9fe//93j7Gbt2rU+11tS+JVk8uTJWrp0qT7++GM1a9bMY9uF0WIFBQXq0qXLZZ+nQYMG2rZtm4wxHjXs3LnTq3ou6Nu3r0aMGKGNGzfqww8/vOx+V61apZycHI+zoB07dri3X/i3sLBQe/bs8TjrubQ+b14zIDEMG3707LPPqnr16hoxYoROnDjhse3kyZMaOXKkatasqeTkZPf6Ro0ayeVyeZw5HTlyREuWLPHof+GM5uIzGJfLpblz5/pcb40aNSSp2AC81KpVqzR+/Hi9+OKL6tOnT5HtAQEB6tevnxYvXqxt27YV2X7s2DH3zz169NDhw4f10UcfudddGL7uixtuuEGzZ8/WxIkT1bNnzxLb9ejRQwUFBZo5c6bH+mnTpsnhcLhH0l3499JRdNOnT/d47M1rBiTOgOBHjRs31rx58zRw4EC1bNlSiYmJiomJ0b59+/S3v/1NP//8sz744AOPL6EOGDBAzz//vPr27atRo0YpLy9Ps2fP1k033eRxAbtr164KDAxUz549NWLECJ06dUpvvfWWwsLCdOTIEZ/qveWWWxQQEKBXX31VLpdLTqfT/T2jSw0cOFChoaFq0qSJ3nvvPY9tv/vd7xQeHq7Jkydr7dq1ateunYYPH67mzZvr5MmT+u6777Rq1SqdPHlSkjR8+HDNnDlTjzzyiDZv3qzIyEi9++67ql69uk+vQ1KJH4FdrGfPnurcubNefPFF7du3T61bt9bnn3+uZcuWafTo0e5rPrfccosGDhyoN954Qy6XS3fccYdWr16t3bt3F3nOq33NgCSGYcP//vnPf5pBgwaZiIgIU6lSJSPJVK1a1fzrX/8qtv3nn39uWrRoYQIDA03Tpk3Ne++9V+ww7L///e+mVatWpmrVqqZhw4bm1VdfNW+//baRZPbu3etud+nw4wuKG1r91ltvmdjYWBMQEOAxBPrStpJKXC4eNp2ZmWmSkpJMdHS0qVKliomIiDDx8fFmzpw5Hvvdv3+/6dWrl6levboJCQkxTz/9tFm5cqXXw7Avp7jjkJOTY8aMGWOioqJMlSpVTJMmTcyUKVNMYWGhR7vTp0+bUaNGmbp165oaNWqYnj17moMHDxYZhn21r5lh2DDGGIcxv/IqLOClefPmaejQoRo8eLDHbAQAri98BIdr7pFHHtGRI0f0b//2b6pXr55eeeUV2yUBsIAzIACAFYyCAwBYQQABAKwggAAAVhBAAAArCCCgAkhLSytyM7ySbu5nS3E14vpGAKFUdOrUqcQ7hF68TJw40WqdJ06c0JQpU9ShQweFhoaqVq1auv322y87Z9rVuPT116lTR7fddpvefvvtIhOjlnWvvPKKli5dam3/eXl5mjVrlrp27arIyEgFBQXpN7/5jWbPnl1kUlqUb3wPCKXixRdf1GOPPeZ+/O233xa5u6hkfzbk9PR0vfjii+rRo4fGjx+vypUra/HixRowYIC2b9+uSZMm+fzc9erVU0pKiqTz857NmzdPiYmJ+p//+R9Nnjy5tF7CVXvrrbd8Cr9XXnlF/fv3L3aOu2vhxx9/1FNPPaX4+HiNHTtWNWvW1GeffaYnn3xSGzdu1DvvvGOlLviB3YkYUFFd7u6iFzt16tS1Keh//fjjj2bfvn0e6woLC80999xjnE6nz/V07NjRxMXFeazLzc019erVMzVq1DBnzpwptl9BQYE5ffq0T/u82Nq1a6/qeF+NGjVqmCFDhvzq57nU1dZ47Ngxs23btiLrhw0bZiSZXbt2lXptsIOP4HDNTJw4UQ6HQ9u3b9egQYNUu3Zt920aOnXqpE6dOhXpU9x1jMLCQk2fPl1xcXGqWrWqwsPDNWLEiCI3l3O5XNqxY4fH3VFjYmLctxm4wOFwqE+fPsrPzy+VO6ReUL16dd1+++3Kzc11zwTtcDiUnJys+fPnKy4uTk6nUytXrpQk/fTTT3r00UcVHh4up9OpuLg4vf3220We99ChQ+rTp49q1KihsLAwjRkzRvn5+UXalXTs/vKXv6hly5aqWrWqQkND1b17d23atMldX25urt555x33x4lDhw519y/tGvPy8rRjxw4dP37cvS4kJERxcXFF2vbt21eS9MMPPxTZhvKJj+BwzT3wwANq0qSJXnnlFZ9uCDdixAilpqZq2LBhGjVqlPbu3auZM2fq+++/11dffaUqVapIkpYsWaJhw4Zp7ty5Hr9Ei5ORkSFJPt2Z9XJ+/PFHBQQEeNwUbs2aNVq4cKGSk5MVEhKihg0bKjMzU7fffrs7oEJDQ/Xpp58qMTFR2dnZ7ttfnz59WvHx8Tpw4IBGjRqlqKgovfvuu1qzZs1V1ZOYmKjU1FQlJCToscce07lz5/Tll19q48aN+u1vf6t3331Xjz32mNq2beu+JfiFWbH9UeM333yjzp07a8KECVe8Puiv/0awyPYpGCqm4j6CuzCj9cCBA4u0L25mamOMGTJkiGnQoIH78Zdffmkkmfnz53u0uzBz9MXrL8wSfaUZl0+cOGHCwsLM3XfffVWvrTgdO3Y0zZo1M8eOHTPHjh0zP/zwgxk1apSRZHr27OluJ8lUqlSpyEzgiYmJJjIy0hw/ftxj/YABA0xwcLDJy8szxhgzffp0I8ksXLjQ3SY3N9c0bty4yPG+9NitWbPGSDKjRo0qUv/Fs1+X9BGcP2q88LHcpTNqXyo/P980b97cxMTEmLNnz162LcoPPoLDNTdy5Eif+y5atEjBwcH63e9+p+PHj7uXNm3a6IYbbvC4I+rQoUNljLns2U9hYaEeeughZWVl6fXXX/e5Lun8nURDQ0MVGhqqm2++Wa+//rruvffeIh9RdezYUc2bN3c/NsZo8eLF6tmzp4wxHq+rW7ducrlc7nshrVixQpGRkerfv7+7f/Xq1d1nK5ezePFiORwOTZgwoci2K90N1l81durUScaYK579JCcna/v27Zo5c6YqV+aDm4qC/5K45i6+AZ23du3aJZfLVexN4iTp6NGjXj3fU089pZUrV2revHlq3bq1z3VJUsOGDfXWW2/J4XCoatWqatKkSbF1Xvr6jx07pqysLM2ZM6fEu6BeeF379+9X48aNiwTGxbfKLsmePXsUFRWlOnXqXO1LuuY1FmfKlCl666239PLLL6tHjx4+PQfKJgII11y1atWKrHM4HMVeD7r0ex+FhYUKCwvT/Pnzi33u0NDQq65j0qRJeuONNzR58mQ9/PDDV92vJDVq1FCXLl2u2O7S139hqPTgwYNLvJOp7eHrtmpMTU3V888/r5EjR2r8+PGl/vywiwBCmVC7du1iR6Dt37/f43GjRo20atUq3XnnncUG2dWaNWuWJk6cqNGjR+v555/3+XlKQ2hoqIKCglRQUHDFAGvQoIG2bdsmY4zHGcbOnTuvuJ9GjRrps88+08mTJy97FlTcx3HXqsaLLVu2TI899pjuv/9+zZo1y6u+KB+4BoQyoVGjRtqxY4d7uLIk/eMf/9BXX33l0e73v/+9CgoK9PLLLxd5jnPnzikrK8v9uLhh2JL04YcfatSoUXrooYc0derU0n0hPggICFC/fv20ePFibdu2rcj2i49Jjx49dPjwYX300UfudXl5eSV+LHaxfv36yRhT7JdtLz77rFGjhsdx9GeNxQ3DlqT169drwIAB6tChg+bPn69KlfhVVRFxBoQy4dFHH9XUqVPVrVs3JSYm6ujRo3rzzTcVFxen7Oxsd7uOHTtqxIgRSklJ0ZYtW9S1a1dVqVJFu3bt0qJFi/SXv/zFffG7uGHY33zzjR555BHVrVtX8fHxRT7Ku+OOOxQbG+t+7HA41LFjR7/PXzZ58mStXbtW7dq10/Dhw9W8eXOdPHlS3333nVatWqWTJ09KkoYPH66ZM2fqkUce0ebNmxUZGal3331X1atXv+I+OnfurIcfflgzZszQrl271L17dxUWFurLL79U586dlZycLElq06aNVq1apalTpyoqKkoxMTFq166dX2osbhj2/v371atXLzkcDvXv31+LFi3y6NOqVSvrH0milFgafYcK7nLDsI8dO1Zsn/fee8/ExsaawMBAc8stt5jPPvusyFDiC+bMmWPatGljqlWrZoKCgkzLli3Nc889Zw4fPuxuU9ww7AvrSloubpuTk2MkmQEDBlzx9RY3E0JxJJmkpKRit2VmZpqkpCQTHR1tqlSpYiIiIkx8fLyZM2eOR7v9+/ebXr16merVq5uQkBDz9NNPu4ehX24YtjHGnDt3zkyZMsU0a9bMBAYGmtDQUJOQkGA2b97sbrNjxw7ToUMHU61aNSPJY0h2addY3DDsC+tKWq40ZBvlB7fkBkqwYsUK3XffffrHP/6hli1b2i4HqHD4YBUowdq1azVgwADCB/ATzoAAAFZwBgQAsIIAAgBYQQABAKwggAAAVpS5L6IWFhbq8OHDCgoKuuIMvQCAsscYo5ycHEVFRV12FosyF0CHDx9WdHS07TIAAL/SwYMHVa9evRK3l7kACgoKkiTdpR6qrCqWqwEAeOuczmqDVrh/n5fEbwE0a9YsTZkyRRkZGWrdurVef/11tW3b9or9LnzsVllVVNlBAAFAufO/3y690mUUvwxC+PDDDzV27FhNmDBB3333nVq3bq1u3bp5fbMwAEDF5ZcAmjp1qoYPH65hw4apefPmevPNN1W9evUityYGAFy/Sj2Azpw5o82bN3vctKpSpUrq0qWL0tPTi7TPz89Xdna2xwIAqPhKPYCOHz+ugoIChYeHe6wPDw9XRkZGkfYpKSkKDg52L4yAA4Drg/Uvoo4bN04ul8u9HDx40HZJAIBroNRHwYWEhCggIECZmZke6zMzMxUREVGkvdPplNPpLO0yAABlXKmfAQUGBqpNmzZavXq1e11hYaFWr16t9u3bl/buAADllF++BzR27FgNGTJEv/3tb9W2bVtNnz5dubm5GjZsmD92BwAoh/wSQA8++KCOHTuml156SRkZGbrlllu0cuXKIgMTAADXrzJ3R9Ts7GwFBwerk3ozEwIAlEPnzFmlaZlcLpdq1qxZYjvro+AAANcnAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYEVl2wUAV7Lvz+297lNQ1fi0r9C4Y173SW+92Kd9eavRmmFe9wn6pppP+wqf8d8+9QO8wRkQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFjBZKS4pn7+ryZe99l2y0w/VFJ6zvo276nXdnT+f173mf/bSJ/2tfCLjl73Kfhhl0/7wvWLMyAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsILJSOEzXyYW/eqWD/xQSel5MyvW6z5T03/ndZ+GDY553efz5h973eehoCNe95Gkfx8a4nWf2OeZjBTe4QwIAGAFAQQAsKLUA2jixIlyOBweS7NmzUp7NwCAcs4v14Di4uK0atWq/9tJZS41AQA8+SUZKleurIiICH88NQCggvDLNaBdu3YpKipKsbGxeuihh3TgwIES2+bn5ys7O9tjAQBUfKUeQO3atVNqaqpWrlyp2bNna+/evbr77ruVk5NTbPuUlBQFBwe7l+jo6NIuCQBQBpV6ACUkJOiBBx5Qq1at1K1bN61YsUJZWVlauHBhse3HjRsnl8vlXg4ePFjaJQEAyiC/jw6oVauWbrrpJu3evbvY7U6nU06n099lAADKGL9/D+jUqVPas2ePIiMj/b0rAEA5UuoB9Oyzz2rdunXat2+f/vu//1t9+/ZVQECABg4cWNq7AgCUY6X+EdyhQ4c0cOBAnThxQqGhobrrrru0ceNGhYaGlvauAADlWKkH0AcflO3JJlHUufg2PvVb03qWD72qeN1j+s83ed1n7YO/9bqPJOnwUa+73PTzJq/7VKpa1es+r3zd0us+L4T80+s+knSu9jmf+gHeYC44AIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALDC7zekQ9l36sZAn/pV8uHvF18mFk3r5f0knAU/7vS6z7W0e9JvvO6zoM5/+rAn3272WG8lf5vC/3iXAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwApmw4ZqzUv3qV//TYO97uP4OdvrPueO7PO6T1n3WI9VXve5oZJvM1sDZRVnQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBZORwmcF2//Hdgllwr5/b+91n8Ra/+HDnqp63eOZI7f7sB8paNUPXvcp8GlPuJ5xBgQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVjAZKXCRrIe9n1j0q0e8n1g0uJL3E4um5wd43WfLn3/jdR9Jqpb9jU/9AG9wBgQAsIIAAgBY4XUArV+/Xj179lRUVJQcDoeWLl3qsd0Yo5deekmRkZGqVq2aunTpol27dpVWvQCACsLrAMrNzVXr1q01a9asYre/9tprmjFjht588019/fXXqlGjhrp166ZffvnlVxcLAKg4vB6EkJCQoISEhGK3GWM0ffp0jR8/Xr1795YkzZs3T+Hh4Vq6dKkGDBjw66oFAFQYpXoNaO/evcrIyFCXLl3c64KDg9WuXTulp6cX2yc/P1/Z2dkeCwCg4ivVAMrIyJAkhYeHe6wPDw93b7tUSkqKgoOD3Ut0dHRplgQAKKOsj4IbN26cXC6Xezl48KDtkgAA10CpBlBERIQkKTMz02N9Zmame9ulnE6natas6bEAACq+Ug2gmJgYRUREaPXq1e512dnZ+vrrr9W+vfffMAcAVFxej4I7deqUdu/e7X68d+9ebdmyRXXq1FH9+vU1evRo/fnPf1aTJk0UExOjP/7xj4qKilKfPn1Ks24AQDnndQBt2rRJnTt3dj8eO3asJGnIkCFKTU3Vc889p9zcXD3++OPKysrSXXfdpZUrV6pqVe/nvgIAVFwOY4yxXcTFsrOzFRwcrE7qrcqOKrbLwXVm97Tbve6z4/fFfym7tN302Qjv+zy6yQ+VAJd3zpxVmpbJ5XJd9rq+9VFwAIDrEwEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFZ4fTsGoDw480UDn/qlN/tPH3p5f6uR1ulDvO5z8zN7vO5T4HUP4NrhDAgAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArGAyUpR5lWMbet3n5caLfNpX7UreTyy6Od/7/TR42ftpQgt+/tn7HQFlGGdAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFk5GizGu08Cev+/wm8Nr9bTVw9Uiv+9z0j2/9UAlQvnAGBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWMBkprqmfh7T3us+k8P/0YU9OH/pIQ/Z18brPzc/t9rpPgdc9gIqHMyAAgBUEEADACq8DaP369erZs6eioqLkcDi0dOlSj+1Dhw6Vw+HwWLp3715a9QIAKgivAyg3N1etW7fWrFmzSmzTvXt3HTlyxL28//77v6pIAEDF4/UghISEBCUkJFy2jdPpVEREhM9FAQAqPr9cA0pLS1NYWJiaNm2qJ554QidOnCixbX5+vrKzsz0WAEDFV+oB1L17d82bN0+rV6/Wq6++qnXr1ikhIUEFBcUPPE1JSVFwcLB7iY6OLu2SAABlUKl/D2jAgAHun1u2bKlWrVqpUaNGSktLU3x8fJH248aN09ixY92Ps7OzCSEAuA74fRh2bGysQkJCtHt38V/WczqdqlmzpscCAKj4/B5Ahw4d0okTJxQZGenvXQEAyhGvP4I7deqUx9nM3r17tWXLFtWpU0d16tTRpEmT1K9fP0VERGjPnj167rnn1LhxY3Xr1q1UCwcAlG9eB9CmTZvUuXNn9+ML12+GDBmi2bNna+vWrXrnnXeUlZWlqKgode3aVS+//LKcTt/m5gIAVExeB1CnTp1kjClx+2efffarCkL5UfnGKK/73D3qa6/73FDp2v3xkr69sdd9bvr5Wz9UAlR8zAUHALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAK0r9lty4fvzwgve3Tl8asdwPlRTV+Z8P+NTv5ueKv3Pv5RT4tCcAnAEBAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBVMRgqfbe41zYdezlKvozjBTxb61O/czz+XciUASsIZEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYwWSkqJDOhgf71K/KmRtLuRK7Co4d96mfyc/3uo/D6f1EswGhIV738UVBaC2f+u16JrB0CylFpsDhU79mT+32uk9BdrZP+7oSzoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAomI0WF9F8fvW27hDLhju8H+tTveGZNr/vUDs3xus/XbRZ43Qe/TvPxyV73iX0u3Q+VcAYEALCEAAIAWOFVAKWkpOi2225TUFCQwsLC1KdPH+3cudOjzS+//KKkpCTVrVtXN9xwg/r166fMzMxSLRoAUP55FUDr1q1TUlKSNm7cqC+++EJnz55V165dlZub624zZswYLV++XIsWLdK6det0+PBh3X///aVeOACgfPNqEMLKlSs9HqempiosLEybN29Whw4d5HK59Le//U0LFizQPffcI0maO3eubr75Zm3cuFG333576VUOACjXftU1IJfLJUmqU6eOJGnz5s06e/asunTp4m7TrFkz1a9fX+npxY+iyM/PV3Z2tscCAKj4fA6gwsJCjR49WnfeeadatGghScrIyFBgYKBq1arl0TY8PFwZGRnFPk9KSoqCg4PdS3R0tK8lAQDKEZ8DKCkpSdu2bdMHH3zwqwoYN26cXC6Xezl48OCvej4AQPng0xdRk5OT9cknn2j9+vWqV6+ee31ERITOnDmjrKwsj7OgzMxMRUREFPtcTqdTTqfTlzIAAOWYV2dAxhglJydryZIlWrNmjWJiYjy2t2nTRlWqVNHq1avd63bu3KkDBw6offv2pVMxAKBC8OoMKCkpSQsWLNCyZcsUFBTkvq4THBysatWqKTg4WImJiRo7dqzq1KmjmjVr6qmnnlL79u0ZAQcA8OBVAM2ePVuS1KlTJ4/1c+fO1dChQyVJ06ZNU6VKldSvXz/l5+erW7dueuONN0qlWABAxeEwxhjbRVwsOztbwcHB6qTequyoYrscXMbpz2Ku3OgSq1t85IdKcD3JM2e87nPWFPqhkuL12DrU6z6uLSGlX0gJIjec87qP89NvvWp/zpxVmpbJ5XKpZs2SJ7ZlLjgAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBY4dMdUQFJqtZtr9d94l5J9rqPKePv0qBmJ73u83WbBX6opPTEfTnM6z7mQA0/VFJU7EenvO/0zT9Lv5AS1Naua9KnIuAMCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsKOPTPKKiiXkh3XYJZcJ9amO7hMuK0VbbJeA6wBkQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFjhVQClpKTotttuU1BQkMLCwtSnTx/t3LnTo02nTp3kcDg8lpEjR5Zq0QCA8s+rAFq3bp2SkpK0ceNGffHFFzp79qy6du2q3Nxcj3bDhw/XkSNH3Mtrr71WqkUDAMq/yt40Xrlypcfj1NRUhYWFafPmzerQoYN7ffXq1RUREVE6FQIAKqRfdQ3I5XJJkurUqeOxfv78+QoJCVGLFi00btw45eXllfgc+fn5ys7O9lgAABWfV2dAFyssLNTo0aN15513qkWLFu71gwYNUoMGDRQVFaWtW7fq+eef186dO/Xxxx8X+zwpKSmaNGmSr2UAAMophzHG+NLxiSee0KeffqoNGzaoXr16JbZbs2aN4uPjtXv3bjVq1KjI9vz8fOXn57sfZ2dnKzo6Wp3UW5UdVXwpDQBg0TlzVmlaJpfLpZo1a5bYzqczoOTkZH3yySdav379ZcNHktq1aydJJQaQ0+mU0+n0pQwAQDnmVQAZY/TUU09pyZIlSktLU0xMzBX7bNmyRZIUGRnpU4EAgIrJqwBKSkrSggULtGzZMgUFBSkjI0OSFBwcrGrVqmnPnj1asGCBevToobp162rr1q0aM2aMOnTooFatWvnlBQAAyievrgE5HI5i18+dO1dDhw7VwYMHNXjwYG3btk25ubmKjo5W3759NX78+Mt+Dnix7OxsBQcHcw0IAMopv1wDulJWRUdHa926dd48JQDgOsVccAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKyrbLuBSxhhJ0jmdlYzlYgAAXjuns5L+7/d5ScpcAOXk5EiSNmiF5UoAAL9GTk6OgoODS9zuMFeKqGussLBQhw8fVlBQkBwOh8e27OxsRUdH6+DBg6pZs6alCu3jOJzHcTiP43Aex+G8snAcjDHKyclRVFSUKlUq+UpPmTsDqlSpkurVq3fZNjVr1ryu32AXcBzO4zicx3E4j+Nwnu3jcLkznwsYhAAAsIIAAgBYUa4CyOl0asKECXI6nbZLsYrjcB7H4TyOw3kch/PK03Eoc4MQAADXh3J1BgQAqDgIIACAFQQQAMAKAggAYEW5CaBZs2apYcOGqlq1qtq1a6dvvvnGdknX3MSJE+VwODyWZs2a2S7L79avX6+ePXsqKipKDodDS5cu9dhujNFLL72kyMhIVatWTV26dNGuXbvsFOtHVzoOQ4cOLfL+6N69u51i/SQlJUW33XabgoKCFBYWpj59+mjnzp0ebX755RclJSWpbt26uuGGG9SvXz9lZmZaqtg/ruY4dOrUqcj7YeTIkZYqLl65CKAPP/xQY8eO1YQJE/Tdd9+pdevW6tatm44ePWq7tGsuLi5OR44ccS8bNmywXZLf5ebmqnXr1po1a1ax21977TXNmDFDb775pr7++mvVqFFD3bp10y+//HKNK/WvKx0HSerevbvH++P999+/hhX637p165SUlKSNGzfqiy++0NmzZ9W1a1fl5ua624wZM0bLly/XokWLtG7dOh0+fFj333+/xapL39UcB0kaPny4x/vhtddes1RxCUw50LZtW5OUlOR+XFBQYKKiokxKSorFqq69CRMmmNatW9suwypJZsmSJe7HhYWFJiIiwkyZMsW9LisryzidTvP+++9bqPDauPQ4GGPMkCFDTO/eva3UY8vRo0eNJLNu3TpjzPn/9lWqVDGLFi1yt/nhhx+MJJOenm6rTL+79DgYY0zHjh3N008/ba+oq1Dmz4DOnDmjzZs3q0uXLu51lSpVUpcuXZSenm6xMjt27dqlqKgoxcbG6qGHHtKBAwdsl2TV3r17lZGR4fH+CA4OVrt27a7L90daWprCwsLUtGlTPfHEEzpx4oTtkvzK5XJJkurUqSNJ2rx5s86ePevxfmjWrJnq169fod8Plx6HC+bPn6+QkBC1aNFC48aNU15eno3ySlTmJiO91PHjx1VQUKDw8HCP9eHh4dqxY4elquxo166dUlNT1bRpUx05ckSTJk3S3XffrW3btikoKMh2eVZkZGRIUrHvjwvbrhfdu3fX/fffr5iYGO3Zs0cvvPCCEhISlJ6eroCAANvllbrCwkKNHj1ad955p1q0aCHp/PshMDBQtWrV8mhbkd8PxR0HSRo0aJAaNGigqKgobd26Vc8//7x27typjz/+2GK1nsp8AOH/JCQkuH9u1aqV2rVrpwYNGmjhwoVKTEy0WBnKggEDBrh/btmypVq1aqVGjRopLS1N8fHxFivzj6SkJG3btu26uA56OSUdh8cff9z9c8uWLRUZGan4+Hjt2bNHjRo1utZlFqvMfwQXEhKigICAIqNYMjMzFRERYamqsqFWrVq66aabtHv3btulWHPhPcD7o6jY2FiFhIRUyPdHcnKyPvnkE61du9bj9i0RERE6c+aMsrKyPNpX1PdDScehOO3atZOkMvV+KPMBFBgYqDZt2mj16tXudYWFhVq9erXat29vsTL7Tp06pT179igyMtJ2KdbExMQoIiLC4/2RnZ2tr7/++rp/fxw6dEgnTpyoUO8PY4ySk5O1ZMkSrVmzRjExMR7b27RpoypVqni8H3bu3KkDBw5UqPfDlY5DcbZs2SJJZev9YHsUxNX44IMPjNPpNKmpqWb79u3m8ccfN7Vq1TIZGRm2S7umnnnmGZOWlmb27t1rvvrqK9OlSxcTEhJijh49ars0v8rJyTHff/+9+f77740kM3XqVPP999+b/fv3G2OMmTx5sqlVq5ZZtmyZ2bp1q+ndu7eJiYkxp0+ftlx56brcccjJyTHPPvusSU9PN3v37jWrVq0yt956q2nSpIn55ZdfbJdeap544gkTHBxs0tLSzJEjR9xLXl6eu83IkSNN/fr1zZo1a8ymTZtM+/btTfv27S1WXfqudBx2795t/vSnP5lNmzaZvXv3mmXLlpnY2FjToUMHy5V7KhcBZIwxr7/+uqlfv74JDAw0bdu2NRs3brRd0jX34IMPmsjISBMYGGhuvPFG8+CDD5rdu3fbLsvv1q5dayQVWYYMGWKMOT8U+49//KMJDw83TqfTxMfHm507d9ot2g8udxzy8vJM165dTWhoqKlSpYpp0KCBGT58eIX7I6241y/JzJ07193m9OnT5sknnzS1a9c21atXN3379jVHjhyxV7QfXOk4HDhwwHTo0MHUqVPHOJ1O07hxY/OHP/zBuFwuu4VfgtsxAACsKPPXgAAAFRMBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArPj/LGHcB4b2F0YAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LwN7uIdCd8Gw"
      },
      "source": [
        "### Evaluate the models on all images"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RFKOD4DG8XmU"
      },
      "source": [
        "Now let's run both models using all the test images we loaded at the beginning of this tutorial:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "05aeAuWjvjPx"
      },
      "source": [
        "# Helper function to evaluate a TFLite model on all images\n",
        "def evaluate_model(tflite_file, model_type):\n",
        "  global test_images\n",
        "  global test_labels\n",
        "\n",
        "  test_image_indices = range(test_images.shape[0])\n",
        "  predictions = run_tflite_model(tflite_file, test_image_indices)\n",
        "\n",
        "  accuracy = (np.sum(test_labels== predictions) * 100) / len(test_images)\n",
        "\n",
        "  print('%s model accuracy is %.4f%% (Number of test samples=%d)' % (\n",
        "      model_type, accuracy, len(test_images)))"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xnFilQpBuMh5"
      },
      "source": [
        "Evaluate the float model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T5mWkSbMcU5z",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f38de0dc-7e1b-4fbe-81c5-4b7dbaa31e28"
      },
      "source": [
        "evaluate_model(tflite_model_file, model_type=\"Float\")"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Float model accuracy is 97.8300% (Number of test samples=10000)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Km3cY9ry8ZlG"
      },
      "source": [
        "Evaluate the quantized model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-9cnwiPp6EGm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c5fbf5f9-6dd1-4fd2-af11-1c137a6e7eb6"
      },
      "source": [
        "evaluate_model(tflite_model_quant_file, model_type=\"Quantized\")"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Quantized model accuracy is 97.7900% (Number of test samples=10000)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L7lfxkor8pgv"
      },
      "source": [
        "So you now have an integer quantized a model with almost no difference in the accuracy, compared to the float model.\n"
      ]
    }
  ]
}