2019-fall-recurrent-nets-soln.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate new Simpson scripts with LSTM RNN\n",
    "## Link to slides [here](https://docs.google.com/presentation/d/1ztu3_4xsuWH1FAsGqnGJkP_TBKS3Q27FPUMFkraMB34/edit?usp=sharing)\n",
    "In this project, we will be using an LSTM with the help of an Embedding layer to train our network on an episode from the Simpsons, specifically the episode \"Moe's Tavern\". This is taken from [this](https://www.kaggle.com/wcukierski/the-simpsons-by-the-data) dataset on kaggle. This model can be applied to any text. We could use more episodes from the Simpsons, a book, articles, wikipedia, etc. It will learn the semantic word associations and being able to generate text in relation to what it is trained on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, lets import all of our libraries we need.\n",
    "\n",
    "I have also created a helper file to handle tokenizing punctuation and generating our input sequences. There is also a format file that handles formating for Epoch output, using tabulate to make it pretty."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'tensorboardX'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-5c91976dbd1a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[0;31m# tensorboardX\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mtensorboardX\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mSummaryWriter\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'tensorboardX'"
     ]
    }
   ],
   "source": [
    "# general imports\n",
    "import numpy as np\n",
    "import time\n",
    "import os\n",
    "\n",
    "# torch imports\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import torch.backends.cudnn as cudnn\n",
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "# tensorboardX\n",
    "from tensorboardX import SummaryWriter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grab important files and create folders\n",
    "Here we download the dataset and save it to the folder \"data\" that we create. There is also a python script file with helper functions I have written to help with processing the data, and the format file for formating epoch output.\n",
    "\n",
    "To do this, we download a script called gdown that allows downloading files from Google Drive from the command line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget https://raw.githubusercontent.com/circulosmeos/gdown.pl/master/gdown.pl\n",
    "#!chmod +x gdown.pl\n",
    "#!mkdir data\n",
    "\n",
    "#!./gdown.pl https://drive.google.com/file/d/1dgOnAVDDTDAg59SYcuu-aBzva7gk5YFk/view helper.py\n",
    "#!./gdown.pl https://drive.google.com/file/d/1F8Jd_0fhlT5kCwd0m5kUDS2KyiVkOSXy/view data/moes_tavern_lines.txt\n",
    "    \n",
    "import helper\n",
    "from format import print_iter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset statistics\n",
    "Before starting our project, we should take a look at the data we are dealing with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------Dataset Stats-----------\n",
      "Approximate number of unique words: 11492\n",
      "Number of scenes: 262\n",
      "Average number of sentences in each scene: 15\n",
      "Number of lines: 4258\n",
      "Average number of words in each line: 12\n"
     ]
    }
   ],
   "source": [
    "script_text = helper.load_script('data/moes_tavern_lines.txt')\n",
    "\n",
    "print('----------Dataset Stats-----------')\n",
    "print('Approximate number of unique words: {}'.format(len({word: None for word in script_text.split()})))\n",
    "scenes = script_text.split('\\n\\n')\n",
    "print('Number of scenes: {}'.format(len(scenes)))\n",
    "sentence_count_scene = [scene.count('\\n') for scene in scenes]\n",
    "print('Average number of sentences in each scene: {:.0f}'.format(np.average(sentence_count_scene)))\n",
    "\n",
    "sentences = [sentence for scene in scenes for sentence in scene.split('\\n')]\n",
    "print('Number of lines: {}'.format(len(sentences)))\n",
    "word_count_sentence = [len(sentence.split()) for sentence in sentences]\n",
    "print('Average number of words in each line: {:.0f}'.format(np.average(word_count_sentence)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenize Text\n",
    "In order to prepare our data for our network, we need to tokenize the words. That is, we will be converting every unique word and punctuation into an integer. Before we do that, we need to make the punctuation more easier to convert to a number. For example, we will be taking any new lines and converting them to the word \"||return||\". This makes the text easier to tokenize and pass into our model. The functions that do this are in the helper.py file.\n",
    "\n",
    "A note on tokenizing: 0 is a reserved integer, that is its not used to represent any words. So our integers for our words will start at 1. This is needed as when we use the model to generate new text, it needs a starting point, known as a seed. If this seed is smaller than our sequence length, then the function pad_sequences will pad that seed with 0's in order to represent \"nothing\". This help reduces noise in the network. Think of it as whitespace, it doesn't change the meaning to the input phrase.\n",
    "\n",
    "This is the list of punctuation and special characters that are converted, notice that spaces are put before and after to make splitting the text easier:\n",
    "- '!' : ' ||exclaimation_mark|| '\n",
    "- ',' : ' ||comma|| '\n",
    "- '\"' : ' ||quotation_mark|| '\n",
    "- ';' : ' ||semicolon|| '\n",
    "- '.' : ' ||period|| '\n",
    "- '?' : ' ||question_mark|| '\n",
    "- '(' : ' ||left_parentheses|| '\n",
    "- ')' : ' ||right_parentheses|| '\n",
    "- '--' : ' ||dash|| '\n",
    "- '\\n' : ' ||return|| '\n",
    "- ':' : ' ||colon|| '\n",
    "\n",
    "We also convert all of the text to lowercase as this reduces the vocabulary list and trains the network faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "script_text = helper.tokenize_punctuation(script_text) # helper function to convert non-word characters\n",
    "script_text = script_text.lower()\n",
    "\n",
    "script_text = script_text.split() # splits the text based on spaces into a list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Conversion Dictionaries and Input Data\n",
    "Now that the tokens have been generated, we will create some dictionaries to convert our tokenized integers back to words, and words to integers. We will also generate our inputs and targets to pass into our model. \n",
    "\n",
    "To do this, we need to specify the sequence length, which is the amount of words we pass into the model at one time. I choose 12 for the average sentence length seen in Dataset Stats, but feel free to change this. A sequence length of 1 is just one word, so we could get better output depending on our sequence length. We use the helper function gen_sequences to do this for us.\n",
    "\n",
    "The dataset and dataloader is defined using the specified batch_size.\n",
    "\n",
    "The targets are simply just the next word in our text. So if we have a sentence: \"Hi, how are you?\" and we input \"Hi, how are you\" our target for this sentence will be \"?\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of vocabulary: 6691, Dataloader size: 2286\n"
     ]
    }
   ],
   "source": [
    "sequence_length = 12\n",
    "batch_size = 32\n",
    "\n",
    "int_to_word = {i+1: word for i, word in enumerate(set(script_text))}\n",
    "word_to_int = {word: i for i, word in int_to_word.items()} # flip word_to_int dict to get int to word\n",
    "int_script_text = np.array([word_to_int[word] for word in script_text], dtype=np.int32) # convert text to integers\n",
    "int_script_text, targets = helper.gen_sequences(int_script_text, sequence_length) # transform int_script_text to sequences of sequence_length and generate targets\n",
    "\n",
    "vocab_size = len(word_to_int) + 1 # add one since indexes are 1 to length\n",
    "# convert to tensors and define dataset\n",
    "dataset = TensorDataset(torch.from_numpy(int_script_text), torch.from_numpy(targets))\n",
    "# define dataloader for the dataset\n",
    "dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=4)\n",
    "\n",
    "print(\"Number of vocabulary: {}, Dataloader size: {}\".format(vocab_size, len(dataloader)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Model\n",
    "Here is the fun part, building our model. We will use LSTM cells and an Embedding layer, with a fully connected Linear layer at the end for the prediction. Documentation for LSTM cells can be found [here](https://pytorch.org/docs/stable/nn.html#lstm) and for embedding [here](https://pytorch.org/docs/stable/nn.html#embedding).\n",
    "\n",
    "An LSTM network can be defined simply as:    \n",
    "```\n",
    "nn.LSTM(input_size, hidden_size, num_layers, dropout, batch_first=True)\n",
    "```\n",
    "- dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs.\n",
    "           Introduces a Dropout layer on the outputs of each LSTM layer except the last layer, \n",
    "- hidden_size: Number of LSTM cells in each layer\n",
    "- batch_first: Whether the first dimensions is batch_size or sequence_length. Leave this to true for our model, as batch_size is first.\n",
    "\n",
    "Import to note is that the output of the LSTM network here has the shape of (seq_len, batch_size, hidden_size), so in our forward pass you need to use the `transpose` tensor method to swap the batch_size and seq_len axes before input into the Linear layer. Also, for input into the Linear layer from the LSTM layer requires the last step of the output. Remember, the LSTM network returns the output for **every state**, so make sure to get the last one.\n",
    "\n",
    "An embedding layer can be defined as:\n",
    "```\n",
    "Embedding(input_dim, embed_size)\n",
    "```\n",
    "Our input dimension will be the length of our vocabulary, the size can be whatever you want to set it at, my case I used 300.\n",
    "Our model will predict the next word based in the input sequence. We could also predict the next two words, or predict entire sentences. For now though we will just stick with one word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LSTM_Model(nn.Module):\n",
    "    def __init__(self, vocab_size, embed_size, lstm_size=400, num_layers=1, dropout=0.3):\n",
    "        super(LSTM_Model, self).__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size, embed_size)\n",
    "        # batch_size is first\n",
    "        self.hidden_dim = lstm_size\n",
    "        self.vocab_size = vocab_size\n",
    "        self.num_layers = num_layers\n",
    "        \n",
    "        \n",
    "        self.LSTM = nn.LSTM(input_size=embed_size, hidden_size=lstm_size, num_layers=num_layers, dropout=dropout, batch_first=True)\n",
    "        self.classifier = nn.Linear(lstm_size, vocab_size)\n",
    "        \n",
    "    def forward(self, x, prev_hidden):\n",
    "        batch_size = x.size(0)\n",
    "        out = self.embedding(x)\n",
    "        out, hidden = self.LSTM(out, prev_hidden)\n",
    "        # the output from the LSTM needs to be flattened for the classifier, so reshape output to: (batch_size * seq_len, hidden_dim)\n",
    "        out = out.contiguous().view(-1, self.hidden_dim)\n",
    "        \n",
    "        out = self.classifier(out)\n",
    "        \n",
    "        # reshape to split apart the batch_size * seq_len dimension\n",
    "        out = out.view(batch_size, -1, self.vocab_size)\n",
    "        \n",
    "        # only need the output of the layer, so remove the middle seq_len dimension\n",
    "        out = out[:, -1]\n",
    "        \n",
    "        return out, hidden\n",
    "    \n",
    "    def init_hidden(self, batch_size):\n",
    "        weight = next(self.parameters()).data\n",
    "    \n",
    "        hidden = (weight.new(self.num_layers, batch_size, self.hidden_dim).zero_(),\n",
    "                      weight.new(self.num_layers, batch_size, self.hidden_dim).zero_())\n",
    " \n",
    "        \n",
    "        return hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters and Compiling the Model\n",
    "The Adam optimizer is very effective and has built in dynamic reduction of the learning rate, so let's use that. We will also set the learning rate, epochs, and batch size.\n",
    "\n",
    "We use the CrossEntropyLoss, which requires raw logits as input, since softmax is built into the loss function.\n",
    "\n",
    "Dropout should be a bit high as we are training on a small amount of data, so our model is prone to overfit quickly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using device: cuda\n",
      "LSTM_Model(\n",
      "  (embedding): Embedding(6691, 300)\n",
      "  (LSTM): LSTM(300, 400, num_layers=2, batch_first=True, dropout=0.5)\n",
      "  (classifier): Linear(in_features=400, out_features=6691, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "model = LSTM_Model(vocab_size, 300, lstm_size=400, num_layers=2, dropout=0.5)\n",
    "\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "print(\"Using device: {}\".format(device))\n",
    "model.to(device)\n",
    "\n",
    "learn_rate = 0.001\n",
    "\n",
    "optimizer = optim.Adam(model.parameters(), lr=learn_rate)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "# torch summary has a bug where it won't work with embedding layers\n",
    "print(model)\n",
    "\n",
    "if device == 'cuda':\n",
    "    # helps with runtime of model, use if you have a constant batch size\n",
    "    cudnn.benchmark = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load weights if continuing training\n",
    "load = torch.load(\"checkpoints/best.weights.pt\")\n",
    "model.load_state_dict(load[\"net\"])\n",
    "print(\"Loaded model from epoch {}.\".format(load[\"epoch\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "Now it is time to train the model. We will use tensorboardX to graph our loss and we will save the checkpoint everytime our training loss decreases. We do not use validation data because we want the model to be closely related to how our text is constructed.\n",
    "\n",
    "Note that the model overfits easily since we don't have much data, so train for a small number of epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1 / 10], Batch [0 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 8.80771 |\n",
      "Epoch [1 / 10], Batch [114 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 6.32366 |\n",
      "Epoch [1 / 10], Batch [228 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  5.9305 |\n",
      "Epoch [1 / 10], Batch [342 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.73574 |\n",
      "Epoch [1 / 10], Batch [456 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  5.5945 |\n",
      "Epoch [1 / 10], Batch [570 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.50513 |\n",
      "Epoch [1 / 10], Batch [684 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.44215 |\n",
      "Epoch [1 / 10], Batch [798 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.38248 |\n",
      "Epoch [1 / 10], Batch [912 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.35269 |\n",
      "Epoch [1 / 10], Batch [1026 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  5.3164 |\n",
      "Epoch [1 / 10], Batch [1140 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.27902 |\n",
      "Epoch [1 / 10], Batch [1254 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.25009 |\n",
      "Epoch [1 / 10], Batch [1368 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.21508 |\n",
      "Epoch [1 / 10], Batch [1482 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.18793 |\n",
      "Epoch [1 / 10], Batch [1596 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.17103 |\n",
      "Epoch [1 / 10], Batch [1710 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.14389 |\n",
      "Epoch [1 / 10], Batch [1824 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.12583 |\n",
      "Epoch [1 / 10], Batch [1938 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.10951 |\n",
      "Epoch [1 / 10], Batch [2052 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.09676 |\n",
      "Epoch [1 / 10], Batch [2166 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  5.0816 |\n",
      "Epoch [1 / 10], Batch [2280 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 5.06561 |\n",
      "Epoch [2 / 10], Batch [0 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.45647 |\n",
      "Epoch [2 / 10], Batch [114 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.42708 |\n",
      "Epoch [2 / 10], Batch [228 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.43628 |\n",
      "Epoch [2 / 10], Batch [342 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.48712 |\n",
      "Epoch [2 / 10], Batch [456 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.48329 |\n",
      "Epoch [2 / 10], Batch [570 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.49174 |\n",
      "Epoch [2 / 10], Batch [684 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.47612 |\n",
      "Epoch [2 / 10], Batch [798 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.47748 |\n",
      "Epoch [2 / 10], Batch [912 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.47812 |\n",
      "Epoch [2 / 10], Batch [1026 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.47727 |\n",
      "Epoch [2 / 10], Batch [1140 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.46799 |\n",
      "Epoch [2 / 10], Batch [1254 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.46366 |\n",
      "Epoch [2 / 10], Batch [1368 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.45713 |\n",
      "Epoch [2 / 10], Batch [1482 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.45975 |\n",
      "Epoch [2 / 10], Batch [1596 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.46421 |\n",
      "Epoch [2 / 10], Batch [1710 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.45838 |\n",
      "Epoch [2 / 10], Batch [1824 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.45735 |\n",
      "Epoch [2 / 10], Batch [1938 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.45172 |\n",
      "Epoch [2 / 10], Batch [2052 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.45308 |\n",
      "Epoch [2 / 10], Batch [2166 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.44975 |\n",
      "Epoch [2 / 10], Batch [2280 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.45207 |\n",
      "\n",
      "Saving Checkpoint..\n",
      "\n",
      "Epoch [3 / 10], Batch [0 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.14805 |\n",
      "Epoch [3 / 10], Batch [114 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  4.0756 |\n",
      "Epoch [3 / 10], Batch [228 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.07716 |\n",
      "Epoch [3 / 10], Batch [342 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  4.0953 |\n",
      "Epoch [3 / 10], Batch [456 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.12092 |\n",
      "Epoch [3 / 10], Batch [570 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  4.1199 |\n",
      "Epoch [3 / 10], Batch [684 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.12359 |\n",
      "Epoch [3 / 10], Batch [798 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.11787 |\n",
      "Epoch [3 / 10], Batch [912 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.11167 |\n",
      "Epoch [3 / 10], Batch [1026 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.12075 |\n",
      "Epoch [3 / 10], Batch [1140 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |   4.123 |\n",
      "Epoch [3 / 10], Batch [1254 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.12191 |\n",
      "Epoch [3 / 10], Batch [1368 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.12242 |\n",
      "Epoch [3 / 10], Batch [1482 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.12426 |\n",
      "Epoch [3 / 10], Batch [1596 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.12235 |\n",
      "Epoch [3 / 10], Batch [1710 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.12022 |\n",
      "Epoch [3 / 10], Batch [1824 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.11981 |\n",
      "Epoch [3 / 10], Batch [1938 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.12033 |\n",
      "Epoch [3 / 10], Batch [2052 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.12453 |\n",
      "Epoch [3 / 10], Batch [2166 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  4.1302 |\n",
      "Epoch [3 / 10], Batch [2280 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.13278 |\n",
      "\n",
      "Saving Checkpoint..\n",
      "\n",
      "Epoch [4 / 10], Batch [0 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 4.41721 |\n",
      "Epoch [4 / 10], Batch [114 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.89023 |\n",
      "Epoch [4 / 10], Batch [228 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.86545 |\n",
      "Epoch [4 / 10], Batch [342 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.83403 |\n",
      "Epoch [4 / 10], Batch [456 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.83527 |\n",
      "Epoch [4 / 10], Batch [570 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.82529 |\n",
      "Epoch [4 / 10], Batch [684 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.82284 |\n",
      "Epoch [4 / 10], Batch [798 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  3.8244 |\n",
      "Epoch [4 / 10], Batch [912 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  3.8302 |\n",
      "Epoch [4 / 10], Batch [1026 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.82939 |\n",
      "Epoch [4 / 10], Batch [1140 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.83229 |\n",
      "Epoch [4 / 10], Batch [1254 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.82445 |\n",
      "Epoch [4 / 10], Batch [1368 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.83415 |\n",
      "Epoch [4 / 10], Batch [1482 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.83185 |\n",
      "Epoch [4 / 10], Batch [1596 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.82847 |\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [4 / 10], Batch [1710 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.83252 |\n",
      "Epoch [4 / 10], Batch [1824 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.83849 |\n",
      "Epoch [4 / 10], Batch [1938 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.83959 |\n",
      "Epoch [4 / 10], Batch [2052 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.84305 |\n",
      "Epoch [4 / 10], Batch [2166 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.84324 |\n",
      "Epoch [4 / 10], Batch [2280 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.84167 |\n",
      "\n",
      "Saving Checkpoint..\n",
      "\n",
      "Epoch [5 / 10], Batch [0 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.78213 |\n",
      "Epoch [5 / 10], Batch [114 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.55942 |\n",
      "Epoch [5 / 10], Batch [228 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.52406 |\n",
      "Epoch [5 / 10], Batch [342 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.51978 |\n",
      "Epoch [5 / 10], Batch [456 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.52786 |\n",
      "Epoch [5 / 10], Batch [570 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.52809 |\n",
      "Epoch [5 / 10], Batch [684 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.52929 |\n",
      "Epoch [5 / 10], Batch [798 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.53087 |\n",
      "Epoch [5 / 10], Batch [912 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.52931 |\n",
      "Epoch [5 / 10], Batch [1026 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |   3.533 |\n",
      "Epoch [5 / 10], Batch [1140 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  3.5275 |\n",
      "Epoch [5 / 10], Batch [1254 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.52937 |\n",
      "Epoch [5 / 10], Batch [1368 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.52616 |\n",
      "Epoch [5 / 10], Batch [1482 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.53484 |\n",
      "Epoch [5 / 10], Batch [1596 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.53729 |\n",
      "Epoch [5 / 10], Batch [1710 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.54848 |\n",
      "Epoch [5 / 10], Batch [1824 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.55062 |\n",
      "Epoch [5 / 10], Batch [1938 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  3.5523 |\n",
      "Epoch [5 / 10], Batch [2052 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.55405 |\n",
      "Epoch [5 / 10], Batch [2166 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.56276 |\n",
      "Epoch [5 / 10], Batch [2280 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.55977 |\n",
      "\n",
      "Saving Checkpoint..\n",
      "\n",
      "Epoch [6 / 10], Batch [0 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  3.7697 |\n",
      "Epoch [6 / 10], Batch [114 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.16132 |\n",
      "Epoch [6 / 10], Batch [228 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.24233 |\n",
      "Epoch [6 / 10], Batch [342 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.22293 |\n",
      "Epoch [6 / 10], Batch [456 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.22306 |\n",
      "Epoch [6 / 10], Batch [570 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.19851 |\n",
      "Epoch [6 / 10], Batch [684 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.20484 |\n",
      "Epoch [6 / 10], Batch [798 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.21105 |\n",
      "Epoch [6 / 10], Batch [912 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.21513 |\n",
      "Epoch [6 / 10], Batch [1026 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.22322 |\n",
      "Epoch [6 / 10], Batch [1140 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  3.2367 |\n",
      "Epoch [6 / 10], Batch [1254 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.24332 |\n",
      "Epoch [6 / 10], Batch [1368 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  3.2494 |\n",
      "Epoch [6 / 10], Batch [1482 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.25445 |\n",
      "Epoch [6 / 10], Batch [1596 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.26403 |\n",
      "Epoch [6 / 10], Batch [1710 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.26439 |\n",
      "Epoch [6 / 10], Batch [1824 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.26875 |\n",
      "Epoch [6 / 10], Batch [1938 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.27675 |\n",
      "Epoch [6 / 10], Batch [2052 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  3.2778 |\n",
      "Epoch [6 / 10], Batch [2166 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.28341 |\n",
      "Epoch [6 / 10], Batch [2280 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.28806 |\n",
      "\n",
      "Saving Checkpoint..\n",
      "\n",
      "Epoch [7 / 10], Batch [0 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.83057 |\n",
      "Epoch [7 / 10], Batch [114 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.89767 |\n",
      "Epoch [7 / 10], Batch [228 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.90917 |\n",
      "Epoch [7 / 10], Batch [342 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.91235 |\n",
      "Epoch [7 / 10], Batch [456 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.92916 |\n",
      "Epoch [7 / 10], Batch [570 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.94535 |\n",
      "Epoch [7 / 10], Batch [684 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.95951 |\n",
      "Epoch [7 / 10], Batch [798 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.96665 |\n",
      "Epoch [7 / 10], Batch [912 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.97541 |\n",
      "Epoch [7 / 10], Batch [1026 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.97924 |\n",
      "Epoch [7 / 10], Batch [1140 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.98209 |\n",
      "Epoch [7 / 10], Batch [1254 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.99174 |\n",
      "Epoch [7 / 10], Batch [1368 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.99707 |\n",
      "Epoch [7 / 10], Batch [1482 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.00293 |\n",
      "Epoch [7 / 10], Batch [1596 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.01421 |\n",
      "Epoch [7 / 10], Batch [1710 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.02164 |\n",
      "Epoch [7 / 10], Batch [1824 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.02546 |\n",
      "Epoch [7 / 10], Batch [1938 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.03418 |\n",
      "Epoch [7 / 10], Batch [2052 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.03703 |\n",
      "Epoch [7 / 10], Batch [2166 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.04002 |\n",
      "Epoch [7 / 10], Batch [2280 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.04694 |\n",
      "\n",
      "Saving Checkpoint..\n",
      "\n",
      "Epoch [8 / 10], Batch [0 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 3.02145 |\n",
      "Epoch [8 / 10], Batch [114 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.75764 |\n",
      "Epoch [8 / 10], Batch [228 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.72111 |\n",
      "Epoch [8 / 10], Batch [342 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.72415 |\n",
      "Epoch [8 / 10], Batch [456 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.72416 |\n",
      "Epoch [8 / 10], Batch [570 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.73703 |\n",
      "Epoch [8 / 10], Batch [684 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.73901 |\n",
      "Epoch [8 / 10], Batch [798 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.74546 |\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [8 / 10], Batch [912 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  2.7572 |\n",
      "Epoch [8 / 10], Batch [1026 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  2.7615 |\n",
      "Epoch [8 / 10], Batch [1140 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.76832 |\n",
      "Epoch [8 / 10], Batch [1254 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.78105 |\n",
      "Epoch [8 / 10], Batch [1368 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.79039 |\n",
      "Epoch [8 / 10], Batch [1482 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.79753 |\n",
      "Epoch [8 / 10], Batch [1596 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.79884 |\n",
      "Epoch [8 / 10], Batch [1710 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.80528 |\n",
      "Epoch [8 / 10], Batch [1824 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.80711 |\n",
      "Epoch [8 / 10], Batch [1938 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.81355 |\n",
      "Epoch [8 / 10], Batch [2052 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.81688 |\n",
      "Epoch [8 / 10], Batch [2166 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.82467 |\n",
      "Epoch [8 / 10], Batch [2280 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.82998 |\n",
      "\n",
      "Saving Checkpoint..\n",
      "\n",
      "Epoch [9 / 10], Batch [0 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.94308 |\n",
      "Epoch [9 / 10], Batch [114 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.50747 |\n",
      "Epoch [9 / 10], Batch [228 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.52431 |\n",
      "Epoch [9 / 10], Batch [342 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.53393 |\n",
      "Epoch [9 / 10], Batch [456 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.53499 |\n",
      "Epoch [9 / 10], Batch [570 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.53152 |\n",
      "Epoch [9 / 10], Batch [684 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.53116 |\n",
      "Epoch [9 / 10], Batch [798 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.53514 |\n",
      "Epoch [9 / 10], Batch [912 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.54097 |\n",
      "Epoch [9 / 10], Batch [1026 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.55143 |\n",
      "Epoch [9 / 10], Batch [1140 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.56604 |\n",
      "Epoch [9 / 10], Batch [1254 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.57522 |\n",
      "Epoch [9 / 10], Batch [1368 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.58514 |\n",
      "Epoch [9 / 10], Batch [1482 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.58982 |\n",
      "Epoch [9 / 10], Batch [1596 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.59374 |\n",
      "Epoch [9 / 10], Batch [1710 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.60513 |\n",
      "Epoch [9 / 10], Batch [1824 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.61213 |\n",
      "Epoch [9 / 10], Batch [1938 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.62212 |\n",
      "Epoch [9 / 10], Batch [2052 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.62876 |\n",
      "Epoch [9 / 10], Batch [2166 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  2.6363 |\n",
      "Epoch [9 / 10], Batch [2280 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.64409 |\n",
      "\n",
      "Saving Checkpoint..\n",
      "\n",
      "Epoch [10 / 10], Batch [0 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.39811 |\n",
      "Epoch [10 / 10], Batch [114 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.34522 |\n",
      "Epoch [10 / 10], Batch [228 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.31622 |\n",
      "Epoch [10 / 10], Batch [342 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.32983 |\n",
      "Epoch [10 / 10], Batch [456 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.35035 |\n",
      "Epoch [10 / 10], Batch [570 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.36239 |\n",
      "Epoch [10 / 10], Batch [684 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  2.3773 |\n",
      "Epoch [10 / 10], Batch [798 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  2.3889 |\n",
      "Epoch [10 / 10], Batch [912 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.40718 |\n",
      "Epoch [10 / 10], Batch [1026 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.41462 |\n",
      "Epoch [10 / 10], Batch [1140 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.41637 |\n",
      "Epoch [10 / 10], Batch [1254 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.42204 |\n",
      "Epoch [10 / 10], Batch [1368 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.43752 |\n",
      "Epoch [10 / 10], Batch [1482 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     |  2.4435 |\n",
      "Epoch [10 / 10], Batch [1596 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.44797 |\n",
      "Epoch [10 / 10], Batch [1710 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.45144 |\n",
      "Epoch [10 / 10], Batch [1824 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.46079 |\n",
      "Epoch [10 / 10], Batch [1938 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.46642 |\n",
      "Epoch [10 / 10], Batch [2052 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.47262 |\n",
      "Epoch [10 / 10], Batch [2166 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.47817 |\n",
      "Epoch [10 / 10], Batch [2280 / 2286]:\n",
      "| Metric   |   Value |\n",
      "|----------|---------|\n",
      "| loss     | 2.48702 |\n",
      "\n",
      "Saving Checkpoint..\n",
      "\n"
     ]
    }
   ],
   "source": [
    "epochs = 10\n",
    "\n",
    "start_time = time.strftime(\"%a_%b_%d_%Y_%H:%M\", time.localtime())\n",
    "\n",
    "# view tensorboard with command: tensorboard --logdir=tensorboard_logs\n",
    "os.makedirs(\"tensorboard_logs\", exist_ok=True)\n",
    "os.makedirs(\"checkpoints\", exist_ok=True)\n",
    "\n",
    "ten_board = SummaryWriter('tensorboard_logs/run_{}'.format(start_time))\n",
    "weight_save_path = 'checkpoints/{}.best.weights.pt'.format(start_time)\n",
    "print_step = len(dataloader) // 20\n",
    "model.train()\n",
    "best_loss = 0\n",
    "\n",
    "for e in range(epochs):\n",
    "    train_loss = 0\n",
    "    \n",
    "    # get inital hidden state\n",
    "    hidden = model.init_hidden(batch_size)\n",
    "    hidden = (hidden[0].to(device), hidden[1].to(device))\n",
    "    \n",
    "    for i, data in enumerate(dataloader):\n",
    "        # make sure you iterate over completely full batches, only\n",
    "        if len(data[0]) < batch_size:\n",
    "            break\n",
    "            \n",
    "        inputs, targets = data\n",
    "        inputs, targets = inputs.type(torch.LongTensor).to(device), targets.type(torch.LongTensor).to(device)\n",
    "        \n",
    "        hidden = tuple([each.data for each in hidden])\n",
    "        optimizer.zero_grad()\n",
    "        outputs, hidden = model(inputs, hidden)\n",
    "        \n",
    "        loss = criterion(outputs, targets)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        train_loss += loss.item()\n",
    "        \n",
    "        \n",
    "        if i % print_step == 0:\n",
    "            print_iter(curr_epoch=e, epochs=epochs, batch_i=i, num_batches=len(dataloader), loss=train_loss/(i+1))\n",
    "    \n",
    "    # print iteration takes the tensorboardX writer and adds the metrics we have to it\n",
    "    print_iter(curr_epoch=e, epochs=epochs, writer=writer, loss=train_loss/len(train_dataloader))\n",
    "    \n",
    "    if e == 0:\n",
    "        best_loss = train_loss\n",
    "    elif train_loss < best_loss:\n",
    "        print('\\nSaving Checkpoint..\\n')\n",
    "        state = {'net': model.state_dict(), 'loss': train_loss, 'epoch': e, 'sequence_length': sequence_length, 'batch_size': batch_size, 'int_to_word': int_to_word, 'word_to_int': word_to_int}\n",
    "        torch.save(state, weight_save_path)\n",
    "        best_loss = train_loss\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing the Model\n",
    "Testing the model simply requires that we convert the output integer back into a word and build our generated text, starting from a seed we define. However, we might get better results by instead of doing an argmax to find the highest probability of what the next word should be, we can take a sample of the top possible words and choose one from there. \n",
    "\n",
    "This is done by taking a \"temperature\" which defines how many predictions we will consider as the next possible word. A lower temperature means the word picked will be closer to the word with the highest probability. Then using a random selection to choose a word. Try it with using both. Setting a temperature of 0 will just use argmax on the entire prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#download weights, model, and dictionaries if using my model:\n",
    "!./gdown.pl https://drive.google.com/file/d/1UPUkyo5D2Q-WK1d1NHUEA4lU3AC1fFAM/view data/dictionaries/int_to_word.pkl\n",
    "!./gdown.pl https://drive.google.com/file/d/1UPUkyo5D2Q-WK1d1NHUEA4lU3AC1fFAM/view data/dictionaries/sequence_length.pkl\n",
    "!./gdown.pl https://drive.google.com/file/d/1fCwa1KnaAMJTriM4hmIpUAbwKF3rkKU1/view data/dictionaries/word_to_int.pkl\n",
    "\n",
    "!./gdown.pl https://drive.google.com/file/d/1v5XzYZ3X3xKlJUl-EyRolq-FbwFwvw3n/view saved_model_data/model.best.weights.hdf5\n",
    "!./gdown.pl https://drive.google.com/file/d/1IJnQA4vKPAesZlF6Sc0hDC_DXa7WKdcY/view saved_model_data/model.Tue_Jul_24_2018_01\\:22.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load model if returning to this notebook for testing, model that I trained:\n",
    "load = torch.load(weight_save_path)\n",
    "model.load_state_dict(load[\"net\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.eval()\n",
    "\n",
    "def sample(prediction, temp=0):\n",
    "    if temp <= 0:\n",
    "        return np.argmax(prediction)\n",
    "    prediction = np.asarray(prediction).astype('float64')\n",
    "    prediction = np.log(prediction) / temp\n",
    "    expo_prediction = np.exp(prediction)\n",
    "    prediction = expo_prediction / np.sum(expo_prediction)\n",
    "    probabilities = np.random.multinomial(1, prediction, 1)\n",
    "    return np.argmax(probabilities)\n",
    "\n",
    "def pad_sequences(sequence, maxlen, value=0):\n",
    "    while len(sequence) < maxlen:\n",
    "        sequence = np.insert(sequence, 0, value)\n",
    "    if len(sequence) > maxlen:\n",
    "        sequence = sequence[len(sequence) - maxlen:]\n",
    "    return sequence\n",
    "\n",
    "#generate new script\n",
    "def generate_text(seed_text, num_words, temp=0):\n",
    "    input_text= seed_text\n",
    "    for _  in range(num_words):\n",
    "        #tokenize text to ints\n",
    "        int_text = helper.tokenize_punctuation(input_text)\n",
    "        int_text = int_text.lower()\n",
    "        int_text = int_text.split()\n",
    "        int_text = np.array([word_to_int[word] for word in int_text], dtype=np.int32)\n",
    "        #pad text if it is too short, pads with zeros at beginning of text, so shouldnt have too much noise added\n",
    "        int_text = pad_sequences(int_text, maxlen=sequence_length)\n",
    "        int_text = np.expand_dims(int_text, axis=0)\n",
    "        # init hiddens state\n",
    "        hidden = model.init_hidden(int_text.shape[0])\n",
    "        hidden = (hidden[0].to(device), hidden[1].to(device))\n",
    "        #predict next word:\n",
    "        prediction, _ = model(torch.from_numpy(int_text).type(torch.LongTensor).to(device), hidden)\n",
    "        prediction = prediction.to(\"cpu\").detach()\n",
    "        prediction = F.softmax(prediction, dim=1).data\n",
    "        prediction = prediction.numpy().squeeze()\n",
    "        output_word = int_to_word[sample(prediction, temp=temp)]\n",
    "        #append to the result\n",
    "        input_text += ' ' + output_word\n",
    "    #convert tokenized punctuation and other characters back\n",
    "    result = helper.untokenize_punctuation(input_text)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting seed is: Homer_Simpson:\n",
      "\n",
      "\n",
      "Homer_Simpson: locked intimacy hops renee's tune.\n",
      " homer_simpson:(singing/pushing) i know, but only only-- 'cause they are hard to make me a favor for the kids.\n",
      " moe_szyslak:(pleased) hey, moe, you seen that rat?\n",
      " homer_simpson:(tentative) oh, wait.\n",
      " homer_simpson:(reading) jane fonda, daniel schorr, jack anderson...\n",
      " moe_szyslak: hey, hey, hey! hey, hey! hey, speaking.\n",
      "\n",
      "\n",
      " moe_szyslak:(lowering binoculars) no, no, no.\n",
      " moe_szyslak: hey, hey, hey! read it off!\n",
      " moe_szyslak:(eyeing phone) ah, bee o'problem. bee o'problem. mmmm, careful.\n",
      " barney_gumble:(singing) hello...\n",
      " homer_simpson:(singing) all right, sir.\n",
      " barney_gumble: hey, moe.\n",
      " moe_szyslak: yeah...(thinking) but i'm not expecting that!\n",
      " moe_szyslak:(to homer) now, you gotta be kiddin' me.\n",
      " homer_simpson:\n"
     ]
    }
   ],
   "source": [
    "#input amount of words to generate, and the seed text, good options are 'Homer_Simpson:', 'Bart_Simpson:', 'Moe_Szyslak:', or other character's names.:\n",
    "seed = 'Homer_Simpson:'\n",
    "num_words = 200\n",
    "temp = 0.5\n",
    "\n",
    "#print amount of characters specified.\n",
    "print(\"Starting seed is: {}\\n\\n\".format(seed))\n",
    "print(generate_text(seed, num_words, temp=temp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Closing Thoughts\n",
    "Remember that this model can be applied to any type of text, even code! So go and try different texts, like the (not) included Harry Potter book. (for time purposes, I would not use the whole book, as training would take a long time.)\n",
    "\n",
    "Try different hyperparameters and model sizes, as you can get some better results."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "ucfai": {
   "authors": [
    {
     "author": "Brandon Silva",
     "github": "brandons209",
     "web": null
    },
    {
     "author": "John Muchovej",
     "github": "ionlights",
     "web": null
    }
   ],
   "date": "2019-02-27",
   "description": "This lecture is all about Recurrent Neural Networks. These are networks with memory, they can learn on data that is sequential in nature, such as speech, text, videos, gestures, traffic, stock prices, and many types of medical data. Different types of RNNs and strategies for building them will also be covered. The project will be building a LSTM-RNN to generate new original scripts for  the TV series “The Simpsons”. Come and find out if our networks can become better writers for the show!",
   "tags": [],
   "title": "Who Needs Show Writers Nowadays?"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}