vegas_dice_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Counting dice with Computer Vision\n",
    "\n",
    "\n",
    "![](https://user-images.githubusercontent.com/3716307/70001073-3ec5d000-1511-11ea-9b4f-42e14b6af1b7.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SageMaker Ground-Truth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll shortly step through the process of setting up a SageMaker ground-truth labelling job, but first we need to upload our images to Amazon S3.\n",
    "\n",
    "We have already loaded the images onto this SageMaker Notebook instance, and you can find them at `./data`.\n",
    "\n",
    "Using the Amazon SageMaker SDK we can upload these images to the default bucket. See `session.upload_data` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "session = sagemaker.session.Session()\n",
    "default_s3_bucket = 's3://{}'.format(session.default_bucket())\n",
    "print('default_s3_bucket: {}'.format(default_s3_bucket))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_images = session.upload_data('./data', key_prefix='vegas-dice-images')\n",
    "print(\"Bucket for ground-truth Labeling: {}/images/\".format(training_images))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should now open the [AWS Management Console](https://console.aws.amazon.com/sagemaker/groundtruth?region=us-east-1#/labeling-jobs) and setup your SageMaker ground-truth labeling job."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploritory Data Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we are going to download and explore the data. This is a dataset that has been labeled in ground-truth and contains the bounding boxes for the dice present in the picture from the training dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is recommended at the beginning of any ML project to get well acquainted with the format and type of data that you are working with, on a qualitative and quantitative level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install gluoncv --pre -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import json\n",
    "import math\n",
    "import os\n",
    "import random\n",
    "import time\n",
    "import zipfile\n",
    "\n",
    "import cv2\n",
    "import gluoncv as gcv\n",
    "import matplotlib.pyplot as plt\n",
    "import mxnet as mx\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = 'data'\n",
    "images_dir = os.path.join(data_dir, 'images')\n",
    "train_images = glob.glob(images_dir + \"/*\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"We have {} images\".format(len(train_images)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how they look like. We use matplotlib to plot 36 images from the dataset to get a feel for what they are"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_images = 36\n",
    "cols = (int(math.sqrt(n_images)))\n",
    "fig = plt.figure(figsize=(15,10))\n",
    "for n, (image) in enumerate(train_images[:n_images]):\n",
    "    image = plt.imread(image)\n",
    "    a = fig.add_subplot(np.ceil(n_images/float(cols)), cols, n + 1)\n",
    "    plt.imshow(image)\n",
    "    plt.axis('off')\n",
    "plt.subplots_adjust(wspace=0.06, hspace=0.06)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bounding boxes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've included the `output.manifest` file from our complete SageMaker GroundTruth labeling job on all images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's dig in the info we have in the manifest file! On each image, there is one or more dice. We read this information from the `output.manifest` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_info = []\n",
    "with open(os.path.join(data_dir, 'manifest', 'output.manifest')) as f:\n",
    "    lines = f.readlines()\n",
    "    for line in lines:\n",
    "        image_info.append(json.loads(line[:-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each image, we have the following information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info = image_info[10]\n",
    "task = 'dice-labeling'\n",
    "info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the name of the different classes corresponding to the class index as given by the SageMaker ground-truth labeling job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_map = info[task+'-metadata']['class-map']\n",
    "classes = [class_map[str(i)] for i in range(len(class_map))]\n",
    "classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can read the data from this dictionnary to use it to draw a bounding box around the dice using the OpenCV library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info = image_info[random.randint(0, len(image_info)-1)]\n",
    "image = plt.imread(os.path.join(images_dir, info['source-ref'].split('/')[-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boxes = info['dice-labeling']['annotations']\n",
    "for box in boxes:\n",
    "    cv2.rectangle(image, (int(box['left']), int(box['top'])), (int(box['left']+box['width']), int(box['top']+box['height'])), (0,255,0), 3)\n",
    "    cv2.putText(image, str(box['class_id']+1), (int(box['left']+box['width']), int(box['top']+box['height'])), 1, 3, (255,0,0), 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "o = plt.imshow(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning an object detection model\n",
    "\n",
    "Now that we have explored the dataset, let's run a training job on SageMaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import os\n",
    "import re\n",
    "import subprocess\n",
    "import sys\n",
    "import time\n",
    "from time import gmtime, strftime\n",
    "\n",
    "import boto3\n",
    "import sagemaker\n",
    "import numpy as np\n",
    "from utils import get_execution_role"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Running the job locally"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is always a good practice to run first the training job in local model in order to make sure that the training complete successfully, this allows much faster feedback cycle than waiting for the creation of remote instances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuring the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_output_path = '{}/'.format(default_s3_bucket)\n",
    "print(\"S3 bucket path: {}\".format(s3_output_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first run it locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance_type = 'local' if mx.context.num_gpus() == 0 else 'local_gpu'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure docker is setup properly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!/bin/bash ./setup.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get the SageMaker execution role"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    role = sagemaker.get_execution_role()\n",
    "except:\n",
    "    role = get_execution_role()\n",
    "\n",
    "print(\"Using IAM role arn: {}\".format(role))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Job definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a descriptive job name \n",
    "job_name_prefix = 'hpo-dice-yolo'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Static hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "static_hyperparameters = {\n",
    "    'epochs': 2\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.mxnet.estimator import MXNet\n",
    "estimator = MXNet(entry_point=\"src/train_yolo.py\",\n",
    "                  role=role,\n",
    "                  instance_type=instance_type,\n",
    "                  instance_count=1,\n",
    "                  output_path=s3_output_path,\n",
    "                  framework_version=\"1.4.1\",\n",
    "                  py_version='py3',\n",
    "                  base_job_name=job_name_prefix,\n",
    "                  hyperparameters=static_hyperparameters\n",
    "                 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first run in local mode to make sure it is running properly, then we can run it remotely"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "estimator.fit({\"train\": training_images})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Hyperparameter Tuner job\n",
    "\n",
    "We are going to run an hyper-parameter tuning job, it is using gaussian processes to estimate the best combination of parameters. Try picking some ranges based on what you know of ML and let the system finds the best candidates for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now pick a cloud instance a create a new estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "static_hyperparameters = {\n",
    "    'epochs' : 50\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance_type = \"ml.p3.2xlarge\"\n",
    "estimator = MXNet(entry_point=\"src/train_yolo.py\",\n",
    "                  role=role,\n",
    "                  instance_type=instance_type,\n",
    "                  instance_count=1,\n",
    "                  output_path=s3_output_path,\n",
    "                  framework_version=\"1.4.1\",\n",
    "                  py_version='py3',\n",
    "                  base_job_name=job_name_prefix,\n",
    "                  hyperparameters=static_hyperparameters\n",
    "                 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Metrics \n",
    "We define the metric that we are going to track, we want to track:\n",
    "- the current running validation Mean Average Precision: `run_validation_mAP`\n",
    "- the final best validation Mean Average Precision: `validation_mAP`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.pyimagesearch.com/wp-content/uploads/2016/09/iou_equation.png\"  width=400>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric_definitions = [\n",
    "    {'Name': 'validation_mAP', 'Regex': 'best mAP ([-+]?[0-9]*[.]?[0-9]+([eE][-+]?[0-9]+)?)'},\n",
    "    {'Name': 'run_validation_mAP', 'Regex': 'running mAP ([-+]?[0-9]*[.]?[0-9]+([eE][-+]?[0-9]+)?)'}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HPO job parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import IntegerParameter, CategoricalParameter, ContinuousParameter, HyperparameterTuner\n",
    "\n",
    "# The hyperparameters we're going to tune\n",
    "hyperparameter_ranges = {\n",
    "    'lr': ContinuousParameter(0.0001, 0.002), # learning rate, how much should the model learn from the current iteration ( < 0.01 )\n",
    "    'batch_size': IntegerParameter(2, 8), # batch size, how many pictures in each learning iteration (> 1)\n",
    "    'lr_factor': ContinuousParameter(0.3, 1), # learning rate factor, How much to multiply the learning rate after 2/3 of trainign (0 < x < 1)\n",
    "    'wd': ContinuousParameter(0.00001, 0.00005), # Weight decay: Regularization to force small weights ( < 0.001 )\n",
    "    'class_factor': ContinuousParameter(2, 8), # Class factor: How much to weigh the importance of getting the right class vs finding objects (> 1)\n",
    "    'model': CategoricalParameter([\"yolo3_darknet53_coco\", \"yolo3_mobilenet1.0_coco\"]),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are running a total of 4 jobs, 2 in parallel. This is quite a small number to really appreciate the power of the bayesian sampling of the hyper parameters but given the time and budget constraint, it is an acceptable compromise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_jobs = 4\n",
    "max_parallel_jobs = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create the tuner object which will `Maximize` the `validataion_mAP` metric across training runs by picking candidate parameter sets from the ranges provided"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner = HyperparameterTuner(estimator,\n",
    "                            objective_metric_name='validation_mAP',\n",
    "                            objective_type='Maximize',\n",
    "                            hyperparameter_ranges=hyperparameter_ranges,\n",
    "                            metric_definitions=metric_definitions,\n",
    "                            max_jobs=max_jobs,\n",
    "                            max_parallel_jobs=max_parallel_jobs,\n",
    "                            base_tuning_job_name=job_name_prefix\n",
    "                           )\n",
    "tuner.fit({\"train\":training_images})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_name = tuner.latest_tuning_job.job_name\n",
    "print(\"Tuning job: %s\" % job_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"You can monitor the progress of your jobs here: https://us-east-1.console.aws.amazon.com/sagemaker/home?region=us-east-1#/hyper-tuning-jobs/{}\".format(job_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!pygmentize src/train_yolo.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 style=\"color: red\">Stop Here!</h1> \n",
    "\n",
    "Now go back up to <a href=\"#Estimator\">here</a> and we'll explain what just happened :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Continue after this point when at least one job as completed in the HPO job**\n",
    "\n",
    "Now that you have a good handle on what happened, let's see if we can deploy the best model from our HPO job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_job = tuner.best_training_job()\n",
    "tuner.best_training_job()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deploy the best tuning job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "estimator_best_job = estimator.attach(best_job, session)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We deploy the best tuning job on a cluster of one CPU instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployed_model = estimator_best_job.deploy(1, 'ml.c5.4xlarge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict bounding boxes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, image = gcv.data.transforms.presets.yolo.load_test('test.jpg', short=384)\n",
    "output = deployed_model.predict(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cid = np.array(output['cid'])\n",
    "scores = np.array(output['score'])\n",
    "bbox = np.array(output['bbox'])\n",
    "\n",
    "\n",
    "o = gcv.utils.viz.plot_bbox(image, bbox, scores, cid, class_names=classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running inference with the webcam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to process the images from the browser, to the endpoint and back\n",
    "\n",
    "This **ONLY** works in Jupyter Notebook **NOT** in Jupyter Labs\n",
    "\n",
    "\n",
    "1) START: Capture an image in javascript through the webcam in the browse\n",
    "\n",
    "2) Convert that image to base64 and send it over to the python kernel\n",
    "\n",
    "3) Convert the image back to numpy and send it over to the SageMaker endpoint\n",
    "\n",
    "4) Get the predicted bounding boxes and paint them over the numpy image\n",
    "\n",
    "5) Convert the annoted image back to base64 and send it to the javascript\n",
    "\n",
    "6) Display the annoted frame\n",
    "\n",
    "7) GOTO START"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "from io import BytesIO\n",
    "from PIL import Image\n",
    "from utils import show_webcam\n",
    "def get_annotated_image(input_image_b64):\n",
    "    prefix, input_image_b64 = input_image_b64.split(',')\n",
    "    input_image_binary = BytesIO(base64.b64decode(input_image_b64))\n",
    "    input_image_np = np.asarray(Image.open(input_image_binary))\n",
    "    input_image_np, _ = mx.image.center_crop(mx.nd.array(input_image_np), (512,384))\n",
    "    _, input_image_loaded = gcv.data.transforms.presets.yolo.transform_test(input_image_np, short=384)\n",
    "    output = deployed_model.predict(input_image_loaded)\n",
    "    cid = np.array(output['cid'])\n",
    "    scores = np.array(output['score'])\n",
    "    bbox = np.array(output['bbox'])\n",
    "    output_image_np = gcv.utils.viz.cv_plot_bbox(input_image_loaded, bbox, scores, cid, class_names=classes)\n",
    "    output_image_PIL = Image.fromarray(output_image_np)\n",
    "    output_buffer = BytesIO()\n",
    "    output_image_PIL.save(output_buffer, format=\"JPEG\")\n",
    "    output_image_b64 = 'data:image/jpeg;base64,'+base64.b64encode(output_buffer.getvalue()).decode(\"utf-8\")\n",
    "    print(output_image_b64)\n",
    "\n",
    "show_webcam()\n",
    "#display() #uncomment this command if you want to remove the webcam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compilation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compile the model using SageMaker neo for faster runtime on specific hardware platforms, in the cloud or on the edge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "compiled_model = estimator_best_job.compile_model('ml_c5', {'data' : (1, 3, 384, 512)}, s3_output_path, framework='mxnet', framework_version='1.4.1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can find the compiled model on s3 and refer to the [SageMaker documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/neo.html) for the depoloyment instructions"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_mxnet_p36",
   "language": "python",
   "name": "conda_mxnet_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}