notebooks/208-optical-character-recognition/208-optical-character-recognition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a3c3d7f4",
   "metadata": {},
   "source": [
    "# Optical Character Recognition (OCR) with OpenVINO™\n",
    "\n",
    "This tutorial demonstrates how to perform optical character recognition (OCR) with OpenVINO models. It is a continuation of the [004-hello-detection](../004-hello-detection/004-hello-detection.ipynb) tutorial, which shows only text detection.\n",
    "\n",
    "The [horizontal-text-detection-0001](https://docs.openvino.ai/latest/omz_models_model_horizontal_text_detection_0001.html) and [text-recognition-resnet](https://docs.openvino.ai/latest/omz_models_model_text_recognition_resnet_fc.html) models are used together for text detection and then text recognition.\n",
    "\n",
    "In this tutorial, Open Model Zoo tools including Model Downloader, Model Converter and Info Dumper are used to download and convert the models from [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo). For more information, refer to the [104-model-tools](../104-model-tools/104-model-tools.ipynb) tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "067aa27d-e71d-476d-9d70-02710044c3e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q \"openvino-dev=2022.3\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38a8791e",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fcabe82",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from IPython.display import Markdown, display\n",
    "from PIL import Image\n",
    "from openvino.runtime import Core\n",
    "\n",
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import load_image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48a52fad",
   "metadata": {},
   "source": [
    "## Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92e378bf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ie = Core()\n",
    "\n",
    "model_dir = Path(\"model\")\n",
    "precision = \"FP16\"\n",
    "detection_model = \"horizontal-text-detection-0001\"\n",
    "recognition_model = \"text-recognition-resnet-fc\"\n",
    "\n",
    "model_dir.mkdir(exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7307283",
   "metadata": {},
   "source": [
    "## Download Models\n",
    "\n",
    "The next cells will run Model Downloader to download the detection and recognition models. If the models have been downloaded before, they will not be downloaded again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed249c63-33b9-4def-903e-9f15a53bbd82",
   "metadata": {},
   "outputs": [],
   "source": [
    "download_command = f\"omz_downloader --name {detection_model},{recognition_model} --output_dir {model_dir} --cache_dir {model_dir} --precision {precision}  --num_attempts 5\"\n",
    "display(Markdown(f\"Download command: `{download_command}`\"))\n",
    "display(Markdown(f\"Downloading {detection_model}, {recognition_model}...\"))\n",
    "!$download_command\n",
    "display(Markdown(f\"Finished downloading {detection_model}, {recognition_model}.\"))\n",
    "\n",
    "detection_model_path = (model_dir / \"intel/horizontal-text-detection-0001\" / precision / detection_model).with_suffix(\".xml\")\n",
    "recognition_model_path = (model_dir / \"public/text-recognition-resnet-fc\" / precision / recognition_model).with_suffix(\".xml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5586f3a-2442-42d1-899c-39c0b452dcfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "### The text-recognition-resnet-fc model consists of many files. All filenames are printed in\n",
    "### the output of Model Downloader. Uncomment the next two lines to show this output.\n",
    "\n",
    "# for line in download_result:\n",
    "#    print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec25afa0-55b0-488d-98ad-07b3bdfb1dc9",
   "metadata": {},
   "source": [
    "## Convert Models\n",
    "\n",
    "The downloaded detection model is an Intel model, which is already in OpenVINO Intermediate Representation (OpenVINO IR) format. The text recognition model is a public model which needs to be converted to OpenVINO IR. Since this model was downloaded from Open Model Zoo, use Model Converter to convert the model to OpenVINO IR format.\n",
    "\n",
    "The output of Model Converter will be displayed. When the conversion is successful, the last lines of output will include `[ SUCCESS ] Generated IR version 11 model.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d848c9c3-1baa-4494-94be-afea85c7ea98",
   "metadata": {},
   "outputs": [],
   "source": [
    "convert_command = f\"omz_converter --name {recognition_model} --precisions {precision} --download_dir {model_dir} --output_dir {model_dir}\"\n",
    "display(Markdown(f\"Convert command: `{convert_command}`\"))\n",
    "display(Markdown(f\"Converting {recognition_model}...\"))\n",
    "! $convert_command"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b12579c5",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Object Detection\n",
    "\n",
    "Load a detection model, load an image, do inference and get the detection inference result.\n",
    "\n",
    "### Load a Detection Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bba4170",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "detection_model = ie.read_model(\n",
    "    model=detection_model_path, weights=detection_model_path.with_suffix(\".bin\")\n",
    ")\n",
    "detection_compiled_model = ie.compile_model(model=detection_model, device_name=\"CPU\")\n",
    "\n",
    "detection_input_layer = detection_compiled_model.input(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9729bafe",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Load an Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66495015",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# The `image_file` variable can point to a URL or a local image.\n",
    "image_file = \"../data/image/intel_rnb.jpg\"\n",
    "\n",
    "image = load_image(image_file)\n",
    "\n",
    "# N,C,H,W = batch size, number of channels, height, width.\n",
    "N, C, H, W = detection_input_layer.shape\n",
    "\n",
    "# Resize the image to meet network expected input sizes.\n",
    "resized_image = cv2.resize(image, (W, H))\n",
    "\n",
    "# Reshape to the network input shape.\n",
    "input_image = np.expand_dims(resized_image.transpose(2, 0, 1), 0)\n",
    "\n",
    "plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c2ed960",
   "metadata": {},
   "source": [
    "### Do Inference\n",
    "\n",
    "Text boxes are detected in the images and returned as blobs of data in the shape of `[100, 5]`. Each description of detection has the `[x_min, y_min, x_max, y_max, conf]` format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1f7a391",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "output_key = detection_compiled_model.output(\"boxes\")\n",
    "boxes = detection_compiled_model([input_image])[output_key]\n",
    "\n",
    "# Remove zero only boxes.\n",
    "boxes = boxes[~np.all(boxes == 0, axis=1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6191cad8-35b2-4cdb-a00c-1f3d7a01754d",
   "metadata": {},
   "source": [
    "### Get Detection Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09025d97",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def multiply_by_ratio(ratio_x, ratio_y, box):\n",
    "    return [\n",
    "        max(shape * ratio_y, 10) if idx % 2 else shape * ratio_x\n",
    "        for idx, shape in enumerate(box[:-1])\n",
    "    ]\n",
    "\n",
    "\n",
    "def run_preprocesing_on_crop(crop, net_shape):\n",
    "    temp_img = cv2.resize(crop, net_shape)\n",
    "    temp_img = temp_img.reshape((1,) * 2 + temp_img.shape)\n",
    "    return temp_img\n",
    "\n",
    "\n",
    "def convert_result_to_image(bgr_image, resized_image, boxes, threshold=0.3, conf_labels=True):\n",
    "    # Define colors for boxes and descriptions.\n",
    "    colors = {\"red\": (255, 0, 0), \"green\": (0, 255, 0), \"white\": (255, 255, 255)}\n",
    "\n",
    "    # Fetch image shapes to calculate a ratio.\n",
    "    (real_y, real_x), (resized_y, resized_x) = image.shape[:2], resized_image.shape[:2]\n",
    "    ratio_x, ratio_y = real_x / resized_x, real_y / resized_y\n",
    "\n",
    "    # Convert the base image from BGR to RGB format.\n",
    "    rgb_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "    # Iterate through non-zero boxes.\n",
    "    for box, annotation in boxes:\n",
    "        # Pick a confidence factor from the last place in an array.\n",
    "        conf = box[-1]\n",
    "        if conf > threshold:\n",
    "            # Convert float to int and multiply position of each box by x and y ratio.\n",
    "            (x_min, y_min, x_max, y_max) = map(int, multiply_by_ratio(ratio_x, ratio_y, box))\n",
    "\n",
    "            # Draw a box based on the position. Parameters in the `rectangle` function are: image, start_point, end_point, color, thickness.\n",
    "            cv2.rectangle(rgb_image, (x_min, y_min), (x_max, y_max), colors[\"green\"], 3)\n",
    "\n",
    "            # Add a text to an image based on the position and confidence. Parameters in the `putText` function are: image, text, bottomleft_corner_textfield, font, font_scale, color, thickness, line_type\n",
    "            if conf_labels:\n",
    "                # Create a background box based on annotation length.\n",
    "                (text_w, text_h), _ = cv2.getTextSize(\n",
    "                    f\"{annotation}\", cv2.FONT_HERSHEY_TRIPLEX, 0.8, 1\n",
    "                )\n",
    "                image_copy = rgb_image.copy()\n",
    "                cv2.rectangle(\n",
    "                    image_copy,\n",
    "                    (x_min, y_min - text_h - 10),\n",
    "                    (x_min + text_w, y_min - 10),\n",
    "                    colors[\"white\"],\n",
    "                    -1,\n",
    "                )\n",
    "                # Add weighted image copy with white boxes under a text.\n",
    "                cv2.addWeighted(image_copy, 0.4, rgb_image, 0.6, 0, rgb_image)\n",
    "                cv2.putText(\n",
    "                    rgb_image,\n",
    "                    f\"{annotation}\",\n",
    "                    (x_min, y_min - 10),\n",
    "                    cv2.FONT_HERSHEY_SIMPLEX,\n",
    "                    0.8,\n",
    "                    colors[\"red\"],\n",
    "                    1,\n",
    "                    cv2.LINE_AA,\n",
    "                )\n",
    "\n",
    "    return rgb_image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe18dbe3-fcce-40ca-bff7-57d11885efbb",
   "metadata": {},
   "source": [
    "## Text Recogntion\n",
    "\n",
    "Load the text recognition model and do inference on the detected boxes from the detection model.\n",
    "\n",
    "### Load Text Recognition Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e4653fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "recognition_model = ie.read_model(\n",
    "    model=recognition_model_path, weights=recognition_model_path.with_suffix(\".bin\")\n",
    ")\n",
    "\n",
    "recognition_compiled_model = ie.compile_model(model=recognition_model, device_name=\"CPU\")\n",
    "\n",
    "recognition_output_layer = recognition_compiled_model.output(0)\n",
    "recognition_input_layer = recognition_compiled_model.input(0)\n",
    "\n",
    "# Get the height and width of the input layer.\n",
    "_, _, H, W = recognition_input_layer.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f70b8d2-cd1f-4554-ad66-46961336501f",
   "metadata": {},
   "source": [
    "### Do Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "409b77d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate scale for image resizing.\n",
    "(real_y, real_x), (resized_y, resized_x) = image.shape[:2], resized_image.shape[:2]\n",
    "ratio_x, ratio_y = real_x / resized_x, real_y / resized_y\n",
    "\n",
    "# Convert the image to grayscale for the text recognition model.\n",
    "grayscale_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)\n",
    "\n",
    "# Get a dictionary to encode output, based on the model documentation.\n",
    "letters = \"~0123456789abcdefghijklmnopqrstuvwxyz\"\n",
    "\n",
    "# Prepare an empty list for annotations.\n",
    "annotations = list()\n",
    "cropped_images = list()\n",
    "# fig, ax = plt.subplots(len(boxes), 1, figsize=(5,15), sharex=True, sharey=True)\n",
    "# Get annotations for each crop, based on boxes given by the detection model.\n",
    "for i, crop in enumerate(boxes):\n",
    "    # Get coordinates on corners of a crop.\n",
    "    (x_min, y_min, x_max, y_max) = map(int, multiply_by_ratio(ratio_x, ratio_y, crop))\n",
    "    image_crop = run_preprocesing_on_crop(grayscale_image[y_min:y_max, x_min:x_max], (W, H))\n",
    "\n",
    "    # Run inference with the recognition model.\n",
    "    result = recognition_compiled_model([image_crop])[recognition_output_layer]\n",
    "\n",
    "    # Squeeze the output to remove unnecessary dimension.\n",
    "    recognition_results_test = np.squeeze(result)\n",
    "\n",
    "    # Read an annotation based on probabilities from the output layer.\n",
    "    annotation = list()\n",
    "    for letter in recognition_results_test:\n",
    "        parsed_letter = letters[letter.argmax()]\n",
    "\n",
    "        # Returning 0 index from `argmax` signalizes an end of a string.\n",
    "        if parsed_letter == letters[0]:\n",
    "            break\n",
    "        annotation.append(parsed_letter)\n",
    "    annotations.append(\"\".join(annotation))\n",
    "    cropped_image = Image.fromarray(image[y_min:y_max, x_min:x_max])\n",
    "    cropped_images.append(cropped_image)\n",
    "\n",
    "boxes_with_annotations = list(zip(boxes, annotations))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec75936e-63bf-49b4-9bd9-790f3bc396c9",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Show Results\n",
    "\n",
    "### Show Detected Text Boxes and OCR Results for the Image\n",
    "\n",
    "Visualize the result by drawing boxes around recognized text and showing the OCR result from the text recognition model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "152458bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12, 12))\n",
    "plt.imshow(convert_result_to_image(image, resized_image, boxes_with_annotations, conf_labels=True));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd96468c-30fc-4679-8984-a530dff453c5",
   "metadata": {},
   "source": [
    "### Show the OCR Result per Bounding Box\n",
    "\n",
    "Depending on the image, the OCR result may not be readable in the image with boxes, as displayed in the cell above. Use the code below to display the extracted boxes and the OCR result per box."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b756c02c-51f9-47ab-9808-ee2a1443860d",
   "metadata": {},
   "outputs": [],
   "source": [
    "for cropped_image, annotation in zip(cropped_images, annotations):\n",
    "    display(cropped_image, Markdown(\"\".join(annotation)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18e3dcbe",
   "metadata": {},
   "source": [
    "### Print Annotations in Plain Text Format\n",
    "\n",
    "Print annotations for detected text based on their position in the input image, starting from the upper left corner.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59ea77ce",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "[\n",
    "    annotation\n",
    "    for _, annotation in sorted(zip(boxes, annotations), key=lambda x: x[0][0] ** 2 + x[0][1] ** 2)\n",
    "]"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "8e25c8ed6cc2cfe6c8620be5042bb64fac4c236f57496fb5eb68e9ea1795f1fe"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}