workshops/01_FineTuneSpamClassifier/notebooks/03_ModelInference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "59eae6c3-87ba-442b-a93c-0e6b9d6b1cb8",
   "metadata": {},
   "source": [
    "# AWS Machine Learning Purpose-built Accelerators Tutorial\n",
    "## Learn how to use [AWS Trainium](https://aws.amazon.com/machine-learning/trainium/) and [AWS Inferentia](https://aws.amazon.com/machine-learning/inferentia/) with [Amazon SageMaker](https://aws.amazon.com/sagemaker/), to optimize your ML workload\n",
    "## Part 3/3 - Compiling and deploying a Bert model to AWS Inferentia 2 with SageMaker + [Hugging Face Optimum Neuron](https://huggingface.co/docs/optimum-neuron/index)\n",
    "\n",
    "In this tutorial, you'll learn how to compile a model to AWS Inferentia and then deploy it to a SageMaker real-time endpoint powered by AWS Inferentia2. First we'll kick-off a SageMaker job to compile the model. We need to do this once. After that, we can deploy our model to a SageMaker endpoint and finally get some predictions.\n",
    "\n",
    "In section 02, you extract some metadata from the Optimum Neuron API and render a table with the current tested/supported models (similar models not listed there can also be compatible, but you need to check by yourself). This table is important for you to understand which models can be selected for deployment. However, if you also need to fine-tune your model, check a similar table in the notebook **Part 2** to see which models can be fine-tuned with AWS Trainium using HF Optimum Neuron. That way you can plan your end2end solution and start implementing it right now."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06990e00-6436-429a-a835-47fb4b2012f5",
   "metadata": {},
   "source": [
    "## 1) Install some required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90593a41-85e9-463e-894a-b80f54b8cdd5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d435090-f88c-4241-bfa2-1a5bf6c0e39b",
   "metadata": {},
   "source": [
    "## 2) Supported models/tasks\n",
    "\n",
    "Models with **[TP]** after the name support Tensor Parallelism"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf36f85b-3416-4be5-b092-fdccfb496f5c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from IPython.display import Markdown, display\n",
    "\n",
    "display(Markdown(\"../docs/optimum_neuron_models.md\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2245d70a-bfa2-40a7-82a3-b61bc3577e92",
   "metadata": {},
   "source": [
    "## 3) Compiling a pre-trained model to AWS Inferentia\n",
    "\n",
    "**IMPORTANT:** Copy the **SageMaker training job name** from the previous notebook **02_ModelFineTuning** or from your AWS Console/SageMaker and set the variable **training_job_name**. It is necessary because we'll use the fine-tuned model as the input for the compilation job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9216e034-62a5-40cf-bc00-c63eabe3f0e1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import boto3\n",
    "import sagemaker\n",
    "\n",
    "print(sagemaker.__version__)\n",
    "if not sagemaker.__version__ >= \"2.146.0\": print(\"You need to upgrade or restart the kernel if you already upgraded\")\n",
    "\n",
    "sess = sagemaker.Session()\n",
    "role = sagemaker.get_execution_role()\n",
    "bucket = sess.default_bucket()\n",
    "region = sess.boto_region_name\n",
    "\n",
    "training_job_name=\"\"\n",
    "\n",
    "if os.path.isfile(\"training_job_name.txt\"): training_job_name = open(\"training_job_name.txt\", \"r\").read().strip()\n",
    "assert len(training_job_name)>0, \"Please copy the name of the training_job you ran in the previous notebook and set training_job_name\"\n",
    "checkpoint_s3_uri=f\"s3://{bucket}/output/{training_job_name}/output/model.tar.gz\"\n",
    "\n",
    "if not os.path.isdir('src'): os.makedirs('src', exist_ok=True)\n",
    "\n",
    "print(f\"sagemaker role arn: {role}\")\n",
    "print(f\"sagemaker bucket: {bucket}\")\n",
    "print(f\"sagemaker session region: {region}\")\n",
    "print(f\"Training job name: {training_job_name}\")\n",
    "print(f\"Model S3 URI: {checkpoint_s3_uri}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70a27737-3dbd-46f0-b098-e6ff72627b49",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 3.1) Compilation script that will be invoked by SageMaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f849bb42-1a85-489a-b621-615f9f1d917b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pygmentize src/compile.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7a6f646-8306-4572-b4d1-496e8012bd20",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pygmentize src/requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f815449-7358-414a-ba98-bfa8afc69432",
   "metadata": {},
   "source": [
    "### 3.2) SageMaker Estimator\n",
    "This object will help you to configure the compilation job (SageMaker Training Job).\n",
    "\n",
    "This job will invoke **compile.py** script, which will compile our model to Inferentia2 and than save the artifacts for deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14fffbb8-2bcc-4466-92d6-6b30b2c412a9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "task=\"SequenceClassification\"\n",
    "# Source: https://huggingface.co/docs/optimum-neuron/guides/export_model#exporting-a-model-to-neuron-via-neuronmodel\n",
    "input_shapes={\"batch_size\": 1, \"sequence_length\": 512}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "558c6bd9-aa4a-44b9-86ee-d4a646d17453",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import logging\n",
    "from sagemaker.pytorch import PyTorch\n",
    "\n",
    "estimator = PyTorch(\n",
    "    entry_point=\"compile.py\", # Specify your train script\n",
    "    source_dir=\"src\",\n",
    "    role=role,\n",
    "    sagemaker_session=sess,\n",
    "    container_log_level=logging.DEBUG,\n",
    "    instance_count=1,\n",
    "    instance_type='ml.trn1.2xlarge',\n",
    "    output_path=f\"s3://{bucket}/output\",\n",
    "    disable_profiler=True,\n",
    "    \n",
    "    image_uri=f\"763104351884.dkr.ecr.{region}.amazonaws.com/pytorch-training-neuronx:2.1.2-neuronx-py310-sdk2.20.0-ubuntu20.04\",\n",
    "    \n",
    "    volume_size = 512,\n",
    "    hyperparameters={     \n",
    "        \"task\": task,\n",
    "        \"input_shapes\": f\"'{json.dumps(input_shapes)}'\",\n",
    "        \"dynamic_batch_size\": True\n",
    "    }\n",
    ")\n",
    "estimator.framework_version = '2.1.2' # workround when using image_uri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1dbad0c1-adc0-44e9-8985-27c797924610",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "estimator.fit({\"checkpoint\": checkpoint_s3_uri})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f88afd2-8eb8-4a95-bfdf-2ec416432657",
   "metadata": {},
   "source": [
    "## 4) Deploy a SageMaker real-time endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f13bd976-533e-4172-908e-a869c9ccaf97",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "from sagemaker.utils import name_from_base\n",
    "from sagemaker.pytorch.model import PyTorchModel\n",
    "\n",
    "# depending on the inf2 instance you deploy the model you'll have more or less accelerators\n",
    "# we'll ask SageMaker to launch 1 worker per core\n",
    "\n",
    "model_data=estimator.model_data\n",
    "print(f\"Model data: {model_data}\")\n",
    "\n",
    "instance_type_idx=0 # default ml.inf2.xlarge\n",
    "instance_types=['ml.inf2.xlarge', 'ml.inf2.8xlarge', 'ml.inf2.24xlarge','ml.inf2.48xlarge']\n",
    "num_workers=[2,2,12,24]\n",
    "\n",
    "print(f\"Instance type: {instance_types[instance_type_idx]}. Num SM workers: {num_workers[instance_type_idx]}\")\n",
    "pytorch_model = PyTorchModel(\n",
    "    image_uri=f\"763104351884.dkr.ecr.{region}.amazonaws.com/pytorch-inference-neuronx:2.1.2-neuronx-py310-sdk2.20.0-ubuntu20.04\",\n",
    "    model_data=model_data,\n",
    "    role=role,    \n",
    "    name=name_from_base('bert-spam-classifier'),\n",
    "    sagemaker_session=sess,\n",
    "    container_log_level=logging.DEBUG,\n",
    "    model_server_workers=num_workers[instance_type_idx], # 1 worker per inferentia chip\n",
    "    framework_version=\"2.1.2\",\n",
    "    env = {\n",
    "        'SAGEMAKER_MODEL_SERVER_TIMEOUT': '3600',\n",
    "        'TASK': task\n",
    "    }\n",
    "    # for production it is important to define vpc_config and use a vpc_endpoint\n",
    "    #vpc_config={\n",
    "    #    'Subnets': ['<SUBNET1>', '<SUBNET2>'],\n",
    "    #    'SecurityGroupIds': ['<SECURITYGROUP1>', '<DEFAULTSECURITYGROUP>']\n",
    "    #}\n",
    ")\n",
    "pytorch_model._is_compiled_model = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4936386-2f04-4325-b793-e21fbe8997f6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "predictor = pytorch_model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=instance_types[instance_type_idx],\n",
    "    model_data_download_timeout=3600, # it takes some time to download all the artifacts and load the model\n",
    "    container_startup_health_check_timeout=1800\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "255b8086-a665-4ca8-92c3-8b6c0ebef7d1",
   "metadata": {},
   "source": [
    "## 5) Run a simple test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d34c41b2-4440-4e5a-9349-0ccda8f42e22",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.serializers import JSONSerializer\n",
    "from sagemaker.deserializers import JSONDeserializer\n",
    "predictor.serializer = JSONSerializer()\n",
    "predictor.deserializer = JSONDeserializer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "c20beb52-ef40-4beb-a15e-54cf591cef9e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed time: 30.43961524963379\n",
      "Pred: 0 - not spam / score: 5.735881805419922\n",
      "Elapsed time: 28.235435485839844\n",
      "Pred: 1 - spam / score: 5.159011363983154\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "labels={0: \"not spam\", 1: \"spam\"}\n",
    "not_spam=\" Deezer.com 10,406,168 Artist DB\\n\\nWe have scraped the Deezer Artist DB, right now there are 10,406,168 listings according to Deezer.com\\n\\nPlease note in going through part of the list, it is obvious there are mistakes inside their system.\\n\\nExamples include and Artist with &amp; in its name might also be found with \"and\" but the Albums for each have different totals etc. Have no clue if there are duplicate albums etc do this error in their system. Even a comma in a name could mean the Artist shows up more than once, I saw in 1 instance that 1 Artist had 6 different ArtistIDs due to spelling errors.\\n\\nSo what is this DB, very simple, it gives you the ArtistID and the actual name of the Artist in another column. If you want to see the artist you add the baseurl to the ArtistID\\n\\nAn example is ArtistID 115 is AC/DC\\n\\n[https://www.deezer.com/us/artist/115](https://www.deezer.com/us/artist/115)\\n\\nYou do not have to use [https://www.deezer.com/us/artist/](https://www.deezer.com/us/artist/) if your first language is other than English, just see if Deezer supports your language and use that baseref\\n\\nFrench for example is [https://www.deezer.com/fr/artist/115](https://www.deezer.com/fr/artist/115)\\n\\nI am providing the DB in 3 different formats:\\n\\n \\n\\nI tried posting download links here but it seems Reddit does not like that so get them here:\\n\\n[https://pastebin\\\\[DOT\\\\]com/V3KJbgif](https://pastebin.com/V3KJbgif)\\n\\n&amp;#x200B;\\n\\n**Special thanks go to** [**/user/KoalaBear84**](https://www.reddit.com/user/KoalaBear84) **for writing the scraper.**\\n\\n&amp;#x200B;\\n\\n**Cross Posted to related Reddit Groups**\"\n",
    "spam=\"🚨 ATTENTION ALL USERS! 🚨\\n\\n🆘 Are you looking for a way to GET RICH QUICK? 🆘\\n\\n💰 Don't waste your time with boring old jobs! 💰\\n\\n💸 Join our CRAZY MONEY-MAKING SYSTEM today! 💸\\n\\n🤑 Just sign up and start earning BIG BUCKS right away! 🤑\\n\\n👉 Plus, if you refer your friends, you'll get even MORE CASH! 👈\\n\\n🔥 This is the HOTTEST OFFER of the year! 🔥\\n\\n👍 Don't wait\"\n",
    "\n",
    "for i,text in enumerate([not_spam, spam]):\n",
    "    t=time.time()\n",
    "    pred = predictor.predict({\"prompt\": text})\n",
    "    elapsed = (time.time()-t)*1000\n",
    "    print(f\"Elapsed time: {elapsed}\")\n",
    "    print(f\"Pred: {i} - {labels[pred[0][0]]} / score: {pred[0][1]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc9a1e13-2660-43b1-abc2-4ad6bf665910",
   "metadata": {},
   "source": [
    "## 5) Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bb61e12-8905-49c9-a783-4607e75ec4a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.delete_model()\n",
    "predictor.delete_endpoint()"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}