HapticNetworkTraining.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Haptic Classification Training\n",
    "\n",
    "This notebook creates the Haptic Network Architecture and trains it for grasp testset1. After training the network weights will be stored in the folder `grasp_Freq_50TimePerSec_7channels_testset1_logs`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies\n",
    "\n",
    "`Python 3.5.4` is used for development and following packages are required to run the code provided in the notebook:\n",
    "\n",
    "`pip install googledrivedownloader`<br>\n",
    "`pip install matplotlib`<br>\n",
    "`pip install tensorflow-gpu`<br>\n",
    "`pip install numpy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle, os, csv, time, shutil\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "from google_drive_downloader import GoogleDriveDownloader as gdd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_image(image, title):\n",
    "    \"\"\"Print the image\n",
    "\n",
    "    :param image: image pixels in list\n",
    "    :param title: title as string to be printed on top of the image\n",
    "    \"\"\"\n",
    "    plt.imshow(image)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    plt.show()\n",
    "\n",
    "def time_taken(start, end):\n",
    "    \"\"\"Human readable time between `start` and `end`\n",
    "\n",
    "    :param start: time.time()\n",
    "    :param end: time.time()\n",
    "    :returns: day:hour:minute:second\n",
    "    \"\"\"\n",
    "    time = end-start\n",
    "    day = time // (24 * 3600)\n",
    "    time = time % (24 * 3600)\n",
    "    hour = time // 3600\n",
    "    time %= 3600\n",
    "    minutes = time // 60\n",
    "    time %= 60\n",
    "    seconds = time\n",
    "    day_hour_min_sec = str('%02d' % int(day))+\":\"+str('%02d' % int(hour))+\":\"+str('%02d' % int(minutes))+\":\"+str('%02d' % int(seconds))\n",
    "    \n",
    "    return day_hour_min_sec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Downloading `category_object_labels.bin`.\n",
    "This contains two dictionaries that stores category labels and object labels\n",
    "\"\"\"\n",
    "\n",
    "gdd.download_file_from_google_drive(file_id='15xPAbso4-uLk8PcRi9cb70r1r3FaU6ux',\n",
    "                                    dest_path='./dataset/category_object_labels.bin',\n",
    "                                    unzip=False)\n",
    "\n",
    "bin_file = open(\"./dataset/category_object_labels.bin\", \"rb\")\n",
    "\n",
    "category_labels = pickle.load(bin_file)\n",
    "object_labels = pickle.load(bin_file)\n",
    "\n",
    "bin_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Haptic Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Downloading testset1 for haptic of grasp interaction and reading it.\n",
    "\"\"\"\n",
    "\n",
    "db_file_name = \"grasp_Freq_50TimePerSec_7channels_testset1.bin\"\n",
    "gdd.download_file_from_google_drive(file_id='16g4Cp67SHqpVSUyOryPEH2HCTZhMcCeB',\n",
    "                                    dest_path='./dataset/'+db_file_name,\n",
    "                                    unzip=False)\n",
    "\n",
    "bin_file = open(\"./dataset/\"+db_file_name, \"rb\")\n",
    "\n",
    "haptic_frames_2d_train = pickle.load(bin_file)\n",
    "category_label_train = pickle.load(bin_file)\n",
    "object_label_train = pickle.load(bin_file)\n",
    "category_label_train_one_hot = pickle.load(bin_file)\n",
    "object_label_train_one_hot = pickle.load(bin_file)\n",
    "\n",
    "haptic_frames_2d_test = pickle.load(bin_file)\n",
    "category_label_test = pickle.load(bin_file)\n",
    "object_label_test = pickle.load(bin_file)\n",
    "category_label_test_one_hot = pickle.load(bin_file)\n",
    "object_label_test_one_hot = pickle.load(bin_file)\n",
    "\n",
    "bin_file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch, frames, channel:  (400, 100, 7)\n",
      "Category:  egg\n",
      "Object:  egg_rough_styrofoam\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Printing shape of the haptic dataset\n",
    "\"\"\"\n",
    "\n",
    "batch_no = 188\n",
    "\n",
    "print(\"batch, frames, channel: \", haptic_frames_2d_train.shape)\n",
    "\n",
    "print(\"Category: \", list(category_labels.keys())[list(category_labels.values()).index(category_label_train[batch_no])])\n",
    "print(\"Object: \", list(object_labels.keys())[list(object_labels.values()).index(object_label_train[batch_no])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Haptic Network Architecture\n",
    "\n",
    "![Haptic Network Architecture](pics/Haptic_CNN.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Haptic Network Hyper-parameters\n",
    "\n",
    "To reduce over-fitting during training, we used dropout after the fully connected layer with 0.5 probability.\n",
    "Training was performed with 400 training epochs using Adam optimization with a learning rate of 1 x $10^{-4}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Network hyper-parameters\n",
    "batch = 5\n",
    "training_epochs = 400\n",
    "display_step = 1\n",
    "\n",
    "frames = time_step_size = haptic_frames_2d_train.shape[1]\n",
    "channel = haptic_frames_2d_train.shape[2]\n",
    "\n",
    "num_classes = category_label_train_one_not.shape[1]\n",
    "\n",
    "model_path = db_file_name.split(\".\")[0]+\"_logs/model.ckpt\"\n",
    "logs_path = db_file_name.split(\".\")[0]+\"_logs/\"\n",
    "\n",
    "X = tf.placeholder('float', [None, frames, channel], name='InputData')\n",
    "Y = tf.placeholder('float', [None, num_classes], name='LabelData')\n",
    "keep_prob = tf.placeholder_with_default(1.0, shape=(), name = 'keep')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "skip_2nd_maxpool = [\"grasp\", \"hold\", \"low\"]\n",
    "\n",
    "# Create model\n",
    "def model():\n",
    "    with tf.name_scope(\"Model\"):\n",
    "        data_placeholder = tf.placeholder('float', [None, frames, channel], name='InputData')\n",
    "        \n",
    "        net = tf.reshape(data_placeholder, [-1, frames, channel, 1])\n",
    "        net = tf.layers.conv2d(inputs=net, filters=32, kernel_size=[20, 5], padding=\"same\", activation=tf.nn.relu)\n",
    "        net = tf.layers.max_pooling2d(inputs=net, pool_size=[10, 1], strides=2)\n",
    "        net = tf.layers.conv2d(inputs=net, filters=64, kernel_size=[1, 3], padding=\"same\", activation=tf.nn.relu)\n",
    "        if (db_file_name.split(\"_\")[0]) in skip_2nd_maxpool:\n",
    "            net = tf.layers.max_pooling2d(inputs=net, pool_size=[1, 1], strides=[1, 2])\n",
    "        else:\n",
    "            net = tf.layers.max_pooling2d(inputs=net, pool_size=[10, 1], strides=2)\n",
    "        net = tf.layers.flatten(net)\n",
    "        # Dense Layer\n",
    "        net = tf.layers.dense(inputs=net, units=1024, activation=tf.nn.relu)\n",
    "        # Add dropout operation\n",
    "        net = tf.layers.dropout(inputs=net, rate=keep_prob)\n",
    "        logits = tf.layers.dense(inputs=net, units=num_classes)\n",
    "        \n",
    "    return logits\n",
    "\n",
    "def loss(prediction, label_placeholder):\n",
    "    with tf.name_scope('Loss'):\n",
    "        cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=prediction, labels=label_placeholder))\n",
    "        # Create a summary to monitor cost tensor\n",
    "        cost_scalar = tf.summary.scalar(\"loss\", cost)\n",
    "    return cost, cost_scalar\n",
    "\n",
    "def training(prediction, label_placeholder):\n",
    "    with tf.name_scope('Optimizer'):\n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate=0.0001)\n",
    "        train_op = optimizer.minimize(cost)\n",
    "    return train_op\n",
    "\n",
    "def evaluate(prediction, Y):\n",
    "    with tf.name_scope('Accuracy'):\n",
    "        # Test model\n",
    "        correct_prediction = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))\n",
    "        # Calculate accuracy\n",
    "        accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float'))\n",
    "        # Create a summary to monitor accuracy tensor\n",
    "        accuracy_scalar = tf.summary.scalar(\"accuracy\", accuracy)\n",
    "    return accuracy, accuracy_scalar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model_dict:  {'Model': <tf.Tensor 'Model/dense_1/BiasAdd:0' shape=(?, 20) dtype=float32>, 'Loss': <tf.Tensor 'Loss/Mean:0' shape=() dtype=float32>, 'Loss_scalar': <tf.Tensor 'Loss/loss:0' shape=() dtype=string>, 'Optimizer': <tf.Operation 'Optimizer/Adam' type=NoOp>, 'Accuracy': <tf.Tensor 'Accuracy/Mean:0' shape=() dtype=float32>, 'Accuracy_scalar': <tf.Tensor 'Accuracy/accuracy:0' shape=() dtype=string>}\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Creating the Neural Network\n",
    "\"\"\"\n",
    "\n",
    "model_dict = {}\n",
    "\n",
    "prediction = model()\n",
    "model_dict[\"Model\"] = prediction\n",
    "\n",
    "cost, cost_scalar = loss(prediction, Y)\n",
    "model_dict[\"Loss\"] = cost\n",
    "model_dict[\"Loss_scalar\"] = cost_scalar\n",
    "\n",
    "train_op = training(prediction, Y)\n",
    "model_dict[\"Optimizer\"] = train_op\n",
    "\n",
    "eval_op, accuracy_scalar = evaluate(prediction, Y)\n",
    "model_dict[\"Accuracy\"] = eval_op\n",
    "model_dict[\"Accuracy_scalar\"] = accuracy_scalar\n",
    "\n",
    "print(\"model_dict: \", model_dict)\n",
    "\n",
    "# Initializing the variables\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "# 'Saver' op to save and restore all the variables\n",
    "saver = tf.train.Saver(max_to_keep=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.path.exists(logs_path):\n",
    "    shutil.rmtree(logs_path)\n",
    "    os.makedirs(logs_path)\n",
    "else:\n",
    "    os.makedirs(logs_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Writing 'Time', 'Epoch', 'Cost', 'Accuracy' in CSV file\n",
    "\"\"\"\n",
    "\n",
    "epoch_cost_accuracy = []\n",
    "epoch_cost_accuracy.append(\"Time\")\n",
    "epoch_cost_accuracy.append(\"Epoch\")\n",
    "epoch_cost_accuracy.append(\"Cost\")\n",
    "epoch_cost_accuracy.append(\"Accuracy\")\n",
    "\n",
    "with open(logs_path+db_file_name.split(\".\")[0]+\"_data.csv\",'w') as f:\n",
    "    writer = csv.writer(f, lineterminator=\"\\n\")\n",
    "    writer.writerow(epoch_cost_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 , Time:  00:00:00:04\n",
      "Cost - 2.9825677961111 --> Accuracy - 0.15000000298023225\n",
      "Epoch: 0002 , Time:  00:00:00:06\n",
      "Cost - 2.7820644050836 --> Accuracy - 0.18000000491738322\n",
      "Epoch: 0003 , Time:  00:00:00:08\n",
      "Cost - 2.5925866603851 --> Accuracy - 0.20000000447034838\n",
      "Epoch: 0004 , Time:  00:00:00:10\n",
      "Cost - 2.42690599113702 --> Accuracy - 0.25000000521540644\n",
      "Epoch: 0005 , Time:  00:00:00:12\n",
      "Cost - 2.3041903585195 --> Accuracy - 0.27000000402331353\n",
      "Epoch: 0006 , Time:  00:00:00:14\n",
      "Cost - 2.16228994578123 --> Accuracy - 0.1800000049173832\n",
      "Epoch: 0007 , Time:  00:00:00:16\n",
      "Cost - 2.10475341230630 --> Accuracy - 0.28000000566244126\n",
      "Epoch: 0008 , Time:  00:00:00:18\n",
      "Cost - 2.02683709263801 --> Accuracy - 0.2500000052154064\n",
      "Epoch: 0009 , Time:  00:00:00:20\n",
      "Cost - 1.92295539528131 --> Accuracy - 0.3300000078976154\n",
      "Epoch: 0010 , Time:  00:00:00:22\n",
      "Cost - 1.8511080071330 --> Accuracy - 0.3200000040233135\n",
      "Epoch: 0011 , Time:  00:00:00:24\n",
      "Cost - 1.7919423401355 --> Accuracy - 0.32000000625848773\n",
      "Epoch: 0012 , Time:  00:00:00:26\n",
      "Cost - 1.71226998865604 --> Accuracy - 0.3200000040233135\n",
      "Epoch: 0013 , Time:  00:00:00:28\n",
      "Cost - 1.64241032600402 --> Accuracy - 0.33000000566244125\n",
      "Epoch: 0014 , Time:  00:00:00:30\n",
      "Cost - 1.59479502663016 --> Accuracy - 0.34000000804662706\n",
      "Epoch: 0015 , Time:  00:00:00:32\n",
      "Cost - 1.5369962081313 --> Accuracy - 0.35000001043081286\n",
      "Epoch: 0016 , Time:  00:00:00:34\n",
      "Cost - 1.49814024940133 --> Accuracy - 0.3200000077486038\n",
      "Epoch: 0017 , Time:  00:00:00:36\n",
      "Cost - 1.46568499952554 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0018 , Time:  00:00:00:38\n",
      "Cost - 1.39610694460570 --> Accuracy - 0.34000000730156904\n",
      "Epoch: 0019 , Time:  00:00:00:40\n",
      "Cost - 1.36249305531382 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0020 , Time:  00:00:00:42\n",
      "Cost - 1.30701570920646 --> Accuracy - 0.36000000610947613\n",
      "Epoch: 0021 , Time:  00:00:00:44\n",
      "Cost - 1.26706828363239 --> Accuracy - 0.3700000070035457\n",
      "Epoch: 0022 , Time:  00:00:00:46\n",
      "Cost - 1.22954559177160 --> Accuracy - 0.3600000068545341\n",
      "Epoch: 0023 , Time:  00:00:00:48\n",
      "Cost - 1.17389271110296 --> Accuracy - 0.3600000075995922\n",
      "Epoch: 0024 , Time:  00:00:00:50\n",
      "Cost - 1.18321345001459 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0025 , Time:  00:00:00:52\n",
      "Cost - 1.12794224843382 --> Accuracy - 0.36000000461936\n",
      "Epoch: 0026 , Time:  00:00:00:54\n",
      "Cost - 1.11798537671566 --> Accuracy - 0.39000000730156903\n",
      "Epoch: 0027 , Time:  00:00:00:56\n",
      "Cost - 1.06605545617 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0028 , Time:  00:00:00:58\n",
      "Cost - 1.03473954573273 --> Accuracy - 0.3200000070035458\n",
      "Epoch: 0029 , Time:  00:00:01:00\n",
      "Cost - 1.00378630943596 --> Accuracy - 0.3700000070035458\n",
      "Epoch: 0030 , Time:  00:00:01:02\n",
      "Cost - 0.96250775121152 --> Accuracy - 0.39000000953674313\n",
      "Epoch: 0031 , Time:  00:00:01:04\n",
      "Cost - 0.97066547833383 --> Accuracy - 0.3600000075995922\n",
      "Epoch: 0032 , Time:  00:00:01:06\n",
      "Cost - 0.92023132499307 --> Accuracy - 0.33000000417232517\n",
      "Epoch: 0033 , Time:  00:00:01:08\n",
      "Cost - 0.88206973876804 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0034 , Time:  00:00:01:10\n",
      "Cost - 0.85747941043227 --> Accuracy - 0.3500000067055225\n",
      "Epoch: 0035 , Time:  00:00:01:12\n",
      "Cost - 0.85173158217221 --> Accuracy - 0.4000000081956387\n",
      "Epoch: 0036 , Time:  00:00:01:14\n",
      "Cost - 0.82457216791808 --> Accuracy - 0.36000000685453415\n",
      "Epoch: 0037 , Time:  00:00:01:16\n",
      "Cost - 0.80346254203468 --> Accuracy - 0.39000000804662704\n",
      "Epoch: 0038 , Time:  00:00:01:18\n",
      "Cost - 0.78836998390033 --> Accuracy - 0.3900000087916851\n",
      "Epoch: 0039 , Time:  00:00:01:20\n",
      "Cost - 0.74538779258728 --> Accuracy - 0.3500000081956387\n",
      "Epoch: 0040 , Time:  00:00:01:22\n",
      "Cost - 0.72202814072370 --> Accuracy - 0.38000000640749937\n",
      "Epoch: 0041 , Time:  00:00:01:24\n",
      "Cost - 0.70049701360985 --> Accuracy - 0.3800000064074993\n",
      "Epoch: 0042 , Time:  00:00:01:26\n",
      "Cost - 0.67496424801647 --> Accuracy - 0.38000001013278956\n",
      "Epoch: 0043 , Time:  00:00:01:28\n",
      "Cost - 0.64803737420588 --> Accuracy - 0.41000000983476637\n",
      "Epoch: 0044 , Time:  00:00:01:30\n",
      "Cost - 0.66116424314677 --> Accuracy - 0.39000000804662704\n",
      "Epoch: 0045 , Time:  00:00:01:32\n",
      "Cost - 0.60219540465623 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0046 , Time:  00:00:01:34\n",
      "Cost - 0.64146221312694 --> Accuracy - 0.4000000067055226\n",
      "Epoch: 0047 , Time:  00:00:01:36\n",
      "Cost - 0.61965633062645 --> Accuracy - 0.4400000102818013\n",
      "Epoch: 0048 , Time:  00:00:01:38\n",
      "Cost - 0.59688909873366 --> Accuracy - 0.3300000064074994\n",
      "Epoch: 0049 , Time:  00:00:01:40\n",
      "Cost - 0.58673041788861 --> Accuracy - 0.3500000089406967\n",
      "Epoch: 0050 , Time:  00:00:01:42\n",
      "Cost - 0.54139325034338 --> Accuracy - 0.4000000081956387\n",
      "Epoch: 0051 , Time:  00:00:01:44\n",
      "Cost - 0.51934200464747 --> Accuracy - 0.3700000077486038\n",
      "Epoch: 0052 , Time:  00:00:01:46\n",
      "Cost - 0.485112235322594 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0053 , Time:  00:00:01:48\n",
      "Cost - 0.52348232958465 --> Accuracy - 0.3800000064074994\n",
      "Epoch: 0054 , Time:  00:00:01:50\n",
      "Cost - 0.51091674640774 --> Accuracy - 0.37000000700354585\n",
      "Epoch: 0055 , Time:  00:00:01:52\n",
      "Cost - 0.47354361042380 --> Accuracy - 0.36000000685453415\n",
      "Epoch: 0056 , Time:  00:00:01:54\n",
      "Cost - 0.46106303855776 --> Accuracy - 0.3700000070035458\n",
      "Epoch: 0057 , Time:  00:00:01:56\n",
      "Cost - 0.450060619739815 --> Accuracy - 0.3700000062584877\n",
      "Epoch: 0058 , Time:  00:00:01:58\n",
      "Cost - 0.45738455643877 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0059 , Time:  00:00:02:00\n",
      "Cost - 0.422331776400096 --> Accuracy - 0.38000000566244135\n",
      "Epoch: 0060 , Time:  00:00:02:02\n",
      "Cost - 0.389087670017033 --> Accuracy - 0.4200000077486039\n",
      "Epoch: 0061 , Time:  00:00:02:04\n",
      "Cost - 0.390707791643217 --> Accuracy - 0.36000000610947613\n",
      "Epoch: 0062 , Time:  00:00:02:06\n",
      "Cost - 0.3853908397722 --> Accuracy - 0.4000000104308129\n",
      "Epoch: 0063 , Time:  00:00:02:08\n",
      "Cost - 0.40030778106302 --> Accuracy - 0.36000000908970836\n",
      "Epoch: 0064 , Time:  00:00:02:10\n",
      "Cost - 0.35040816422551 --> Accuracy - 0.4200000084936619\n",
      "Epoch: 0065 , Time:  00:00:02:12\n",
      "Cost - 0.34502147254534 --> Accuracy - 0.3300000049173831\n",
      "Epoch: 0066 , Time:  00:00:02:14\n",
      "Cost - 0.35633066301234 --> Accuracy - 0.3800000086426736\n",
      "Epoch: 0067 , Time:  00:00:02:16\n",
      "Cost - 0.32429617601446 --> Accuracy - 0.37000000923872\n",
      "Epoch: 0068 , Time:  00:00:02:18\n",
      "Cost - 0.327407016465440 --> Accuracy - 0.39000000804662704\n",
      "Epoch: 0069 , Time:  00:00:02:20\n",
      "Cost - 0.32027137022232 --> Accuracy - 0.3300000064074994\n",
      "Epoch: 0070 , Time:  00:00:02:22\n",
      "Cost - 0.32553021539933 --> Accuracy - 0.3400000073015691\n",
      "Epoch: 0071 , Time:  00:00:02:24\n",
      "Cost - 0.300660881050862 --> Accuracy - 0.36000000610947613\n",
      "Epoch: 0072 , Time:  00:00:02:26\n",
      "Cost - 0.31401351287495 --> Accuracy - 0.36000000610947613\n",
      "Epoch: 0073 , Time:  00:00:02:28\n",
      "Cost - 0.278453181777149 --> Accuracy - 0.3500000067055226\n",
      "Epoch: 0074 , Time:  00:00:02:30\n",
      "Cost - 0.26887874887324 --> Accuracy - 0.3900000073015691\n",
      "Epoch: 0075 , Time:  00:00:02:32\n",
      "Cost - 0.246652581123635 --> Accuracy - 0.4000000104308129\n",
      "Epoch: 0076 , Time:  00:00:02:34\n",
      "Cost - 0.237797032971866 --> Accuracy - 0.36000000908970836\n",
      "Epoch: 0077 , Time:  00:00:02:36\n",
      "Cost - 0.264891944173723 --> Accuracy - 0.3700000084936619\n",
      "Epoch: 0078 , Time:  00:00:02:38\n",
      "Cost - 0.285992359672673 --> Accuracy - 0.39000000804662704\n",
      "Epoch: 0079 , Time:  00:00:02:40\n",
      "Cost - 0.249491186405066 --> Accuracy - 0.35000000745058063\n",
      "Epoch: 0080 , Time:  00:00:02:42\n",
      "Cost - 0.284999329730635 --> Accuracy - 0.3600000068545342\n",
      "Epoch: 0081 , Time:  00:00:02:44\n",
      "Cost - 0.23938025777461 --> Accuracy - 0.3500000089406967\n",
      "Epoch: 0082 , Time:  00:00:02:46\n",
      "Cost - 0.18133874419145 --> Accuracy - 0.38000000789761557\n",
      "Epoch: 0083 , Time:  00:00:02:48\n",
      "Cost - 0.224506148137152 --> Accuracy - 0.40000000894069676\n",
      "Epoch: 0084 , Time:  00:00:02:50\n",
      "Cost - 0.221840767073444 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0085 , Time:  00:00:02:52\n",
      "Cost - 0.185320714279077 --> Accuracy - 0.4000000074505807\n",
      "Epoch: 0086 , Time:  00:00:02:54\n",
      "Cost - 0.184677635773550 --> Accuracy - 0.3500000081956387\n",
      "Epoch: 0087 , Time:  00:00:02:56\n",
      "Cost - 0.1756022444984 --> Accuracy - 0.38000000640749937\n",
      "Epoch: 0088 , Time:  00:00:02:58\n",
      "Cost - 0.197479147854028 --> Accuracy - 0.34000000655651097\n",
      "Epoch: 0089 , Time:  00:00:03:00\n",
      "Cost - 0.165600852749776 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0090 , Time:  00:00:03:02\n",
      "Cost - 0.170329311850946 --> Accuracy - 0.38000000715255744\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0091 , Time:  00:00:03:04\n",
      "Cost - 0.145208079670555 --> Accuracy - 0.3800000071525575\n",
      "Epoch: 0092 , Time:  00:00:03:06\n",
      "Cost - 0.165558858186705 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0093 , Time:  00:00:03:08\n",
      "Cost - 0.141749881068244 --> Accuracy - 0.35000000670552256\n",
      "Epoch: 0094 , Time:  00:00:03:10\n",
      "Cost - 0.169948584306985 --> Accuracy - 0.3600000083446503\n",
      "Epoch: 0095 , Time:  00:00:03:12\n",
      "Cost - 0.15145254991366 --> Accuracy - 0.3800000086426736\n",
      "Epoch: 0096 , Time:  00:00:03:14\n",
      "Cost - 0.156915412441594 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0097 , Time:  00:00:03:16\n",
      "Cost - 0.13052815099363 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0098 , Time:  00:00:03:18\n",
      "Cost - 0.141537678439635 --> Accuracy - 0.35000000745058063\n",
      "Epoch: 0099 , Time:  00:00:03:20\n",
      "Cost - 0.207786356826545 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0100 , Time:  00:00:03:22\n",
      "Cost - 0.16128903574426 --> Accuracy - 0.39000000804662716\n",
      "Epoch: 0101 , Time:  00:00:03:24\n",
      "Cost - 0.161709482956211 --> Accuracy - 0.4000000089406968\n",
      "Epoch: 0102 , Time:  00:00:03:26\n",
      "Cost - 0.117087022648774 --> Accuracy - 0.3200000077486038\n",
      "Epoch: 0103 , Time:  00:00:03:28\n",
      "Cost - 0.08401488263480 --> Accuracy - 0.4100000098347665\n",
      "Epoch: 0104 , Time:  00:00:03:30\n",
      "Cost - 0.089329956535948 --> Accuracy - 0.3700000070035458\n",
      "Epoch: 0105 , Time:  00:00:03:32\n",
      "Cost - 0.096953380282502 --> Accuracy - 0.3700000070035458\n",
      "Epoch: 0106 , Time:  00:00:03:34\n",
      "Cost - 0.121025809017010 --> Accuracy - 0.35000000596046454\n",
      "Epoch: 0107 , Time:  00:00:03:36\n",
      "Cost - 0.13524135705083 --> Accuracy - 0.37000000923871995\n",
      "Epoch: 0108 , Time:  00:00:03:38\n",
      "Cost - 0.13603478777222 --> Accuracy - 0.41000000908970835\n",
      "Epoch: 0109 , Time:  00:00:03:40\n",
      "Cost - 0.098258641839493 --> Accuracy - 0.36000000685453415\n",
      "Epoch: 0110 , Time:  00:00:03:42\n",
      "Cost - 0.143936470564221 --> Accuracy - 0.37000000923871995\n",
      "Epoch: 0111 , Time:  00:00:03:44\n",
      "Cost - 0.178701743896817 --> Accuracy - 0.38000000789761545\n",
      "Epoch: 0112 , Time:  00:00:03:46\n",
      "Cost - 0.189968457669601 --> Accuracy - 0.3300000064074994\n",
      "Epoch: 0113 , Time:  00:00:03:48\n",
      "Cost - 0.1609632605657 --> Accuracy - 0.36000000685453415\n",
      "Epoch: 0114 , Time:  00:00:03:50\n",
      "Cost - 0.102974848012672 --> Accuracy - 0.36000000834465035\n",
      "Epoch: 0115 , Time:  00:00:03:52\n",
      "Cost - 0.108686377014964 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0116 , Time:  00:00:03:54\n",
      "Cost - 0.072756658139405 --> Accuracy - 0.3500000067055226\n",
      "Epoch: 0117 , Time:  00:00:03:56\n",
      "Cost - 0.057091561000561 --> Accuracy - 0.37000000923872\n",
      "Epoch: 0118 , Time:  00:00:03:58\n",
      "Cost - 0.048609995355946 --> Accuracy - 0.3700000077486038\n",
      "Epoch: 0119 , Time:  00:00:04:00\n",
      "Cost - 0.079252512770472 --> Accuracy - 0.3900000095367432\n",
      "Epoch: 0120 , Time:  00:00:04:02\n",
      "Cost - 0.072370435093762 --> Accuracy - 0.41000000759959226\n",
      "Epoch: 0121 , Time:  00:00:04:04\n",
      "Cost - 0.07899764472676 --> Accuracy - 0.39000000730156903\n",
      "Epoch: 0122 , Time:  00:00:04:06\n",
      "Cost - 0.083858146247803 --> Accuracy - 0.39000000730156903\n",
      "Epoch: 0123 , Time:  00:00:04:08\n",
      "Cost - 0.091668148402823 --> Accuracy - 0.3400000080466271\n",
      "Epoch: 0124 , Time:  00:00:04:10\n",
      "Cost - 0.0457293278333963 --> Accuracy - 0.3400000087916851\n",
      "Epoch: 0125 , Time:  00:00:04:12\n",
      "Cost - 0.050217660891939 --> Accuracy - 0.3500000081956387\n",
      "Epoch: 0126 , Time:  00:00:04:14\n",
      "Cost - 0.060806011603563 --> Accuracy - 0.37000000700354585\n",
      "Epoch: 0127 , Time:  00:00:04:16\n",
      "Cost - 0.075725806152331 --> Accuracy - 0.3500000081956387\n",
      "Epoch: 0128 , Time:  00:00:04:18\n",
      "Cost - 0.073789699844201 --> Accuracy - 0.340000006556511\n",
      "Epoch: 0129 , Time:  00:00:04:20\n",
      "Cost - 0.048622114432509 --> Accuracy - 0.40000000819563875\n",
      "Epoch: 0130 , Time:  00:00:04:22\n",
      "Cost - 0.044968034230987 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0131 , Time:  00:00:04:24\n",
      "Cost - 0.0454403934876609 --> Accuracy - 0.37000000849366194\n",
      "Epoch: 0132 , Time:  00:00:04:26\n",
      "Cost - 0.0462163223215611 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0133 , Time:  00:00:04:28\n",
      "Cost - 0.074074166335049 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0134 , Time:  00:00:04:30\n",
      "Cost - 0.067411023558815 --> Accuracy - 0.34000000730156904\n",
      "Epoch: 0135 , Time:  00:00:04:32\n",
      "Cost - 0.16908735531615 --> Accuracy - 0.34000000730156904\n",
      "Epoch: 0136 , Time:  00:00:04:34\n",
      "Cost - 0.370829785833484 --> Accuracy - 0.3300000086426735\n",
      "Epoch: 0137 , Time:  00:00:04:36\n",
      "Cost - 0.193107039498863 --> Accuracy - 0.30000000521540643\n",
      "Epoch: 0138 , Time:  00:00:04:38\n",
      "Cost - 0.178063524642493 --> Accuracy - 0.3400000058114529\n",
      "Epoch: 0139 , Time:  00:00:04:40\n",
      "Cost - 0.125473163151764 --> Accuracy - 0.3300000064074994\n",
      "Epoch: 0140 , Time:  00:00:04:42\n",
      "Cost - 0.0537539609271334 --> Accuracy - 0.3400000058114529\n",
      "Epoch: 0141 , Time:  00:00:04:44\n",
      "Cost - 0.040325399649736 --> Accuracy - 0.3800000078976155\n",
      "Epoch: 0142 , Time:  00:00:04:46\n",
      "Cost - 0.0233524703748116 --> Accuracy - 0.36000000834465024\n",
      "Epoch: 0143 , Time:  00:00:04:49\n",
      "Cost - 0.040086136627360 --> Accuracy - 0.3700000084936619\n",
      "Epoch: 0144 , Time:  00:00:04:50\n",
      "Cost - 0.034718431708461 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0145 , Time:  00:00:04:52\n",
      "Cost - 0.021584984636865 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0146 , Time:  00:00:04:54\n",
      "Cost - 0.019015571960517 --> Accuracy - 0.37000000849366194\n",
      "Epoch: 0147 , Time:  00:00:04:56\n",
      "Cost - 0.0182544462950318 --> Accuracy - 0.42000000774860385\n",
      "Epoch: 0148 , Time:  00:00:04:58\n",
      "Cost - 0.0160778043915342 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0149 , Time:  00:00:05:00\n",
      "Cost - 0.0145427950032171 --> Accuracy - 0.3900000087916851\n",
      "Epoch: 0150 , Time:  00:00:05:02\n",
      "Cost - 0.0147425771181588 --> Accuracy - 0.38000000864267347\n",
      "Epoch: 0151 , Time:  00:00:05:04\n",
      "Cost - 0.0149775093450443 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0152 , Time:  00:00:05:06\n",
      "Cost - 0.01578002635797 --> Accuracy - 0.39000000730156903\n",
      "Epoch: 0153 , Time:  00:00:05:08\n",
      "Cost - 0.01425897762674 --> Accuracy - 0.3800000071525574\n",
      "Epoch: 0154 , Time:  00:00:05:10\n",
      "Cost - 0.0186655238032926 --> Accuracy - 0.38000000864267347\n",
      "Epoch: 0155 , Time:  00:00:05:12\n",
      "Cost - 0.0153869158933957 --> Accuracy - 0.38000000789761545\n",
      "Epoch: 0156 , Time:  00:00:05:14\n",
      "Cost - 0.0158473099574621 --> Accuracy - 0.37000000700354585\n",
      "Epoch: 0157 , Time:  00:00:05:16\n",
      "Cost - 0.0123626615044486 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0158 , Time:  00:00:05:18\n",
      "Cost - 0.011886170191428 --> Accuracy - 0.3700000070035458\n",
      "Epoch: 0159 , Time:  00:00:05:20\n",
      "Cost - 0.0169783597928471 --> Accuracy - 0.38000000864267347\n",
      "Epoch: 0160 , Time:  00:00:05:23\n",
      "Cost - 0.048373412798537 --> Accuracy - 0.3600000083446503\n",
      "Epoch: 0161 , Time:  00:00:05:25\n",
      "Cost - 0.49605863140022 --> Accuracy - 0.31000000759959223\n",
      "Epoch: 0162 , Time:  00:00:05:27\n",
      "Cost - 0.434673482069047 --> Accuracy - 0.33000000715255745\n",
      "Epoch: 0163 , Time:  00:00:05:29\n",
      "Cost - 0.31449962474871 --> Accuracy - 0.36000000685453426\n",
      "Epoch: 0164 , Time:  00:00:05:31\n",
      "Cost - 0.069432650879025 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0165 , Time:  00:00:05:33\n",
      "Cost - 0.038434822439739 --> Accuracy - 0.33000000789761547\n",
      "Epoch: 0166 , Time:  00:00:05:35\n",
      "Cost - 0.0382946346275275 --> Accuracy - 0.3700000077486038\n",
      "Epoch: 0167 , Time:  00:00:05:37\n",
      "Cost - 0.0491983200612594 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0168 , Time:  00:00:05:39\n",
      "Cost - 0.048309890087693 --> Accuracy - 0.3700000084936619\n",
      "Epoch: 0169 , Time:  00:00:05:41\n",
      "Cost - 0.0179192280105780 --> Accuracy - 0.37000000700354585\n",
      "Epoch: 0170 , Time:  00:00:05:43\n",
      "Cost - 0.0151002936356235 --> Accuracy - 0.3900000087916851\n",
      "Epoch: 0171 , Time:  00:00:05:46\n",
      "Cost - 0.012585348340508 --> Accuracy - 0.3700000077486038\n",
      "Epoch: 0172 , Time:  00:00:05:48\n",
      "Cost - 0.0119655363901983 --> Accuracy - 0.37000000700354585\n",
      "Epoch: 0173 , Time:  00:00:05:50\n",
      "Cost - 0.0109846170023956 --> Accuracy - 0.3700000077486038\n",
      "Epoch: 0174 , Time:  00:00:05:52\n",
      "Cost - 0.0099404448090353 --> Accuracy - 0.37000000700354585\n",
      "Epoch: 0175 , Time:  00:00:05:54\n",
      "Cost - 0.0103763693645305 --> Accuracy - 0.37000000700354585\n",
      "Epoch: 0176 , Time:  00:00:05:56\n",
      "Cost - 0.021034497139044 --> Accuracy - 0.39000000804662716\n",
      "Epoch: 0177 , Time:  00:00:05:59\n",
      "Cost - 0.0120748668079613 --> Accuracy - 0.3600000061094762\n",
      "Epoch: 0178 , Time:  00:00:06:01\n",
      "Cost - 0.0152705737949872 --> Accuracy - 0.3400000080466271\n",
      "Epoch: 0179 , Time:  00:00:06:03\n",
      "Cost - 0.0469292592097190 --> Accuracy - 0.3400000087916851\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0180 , Time:  00:00:06:06\n",
      "Cost - 0.197492799954488 --> Accuracy - 0.33000000715255745\n",
      "Epoch: 0181 , Time:  00:00:06:08\n",
      "Cost - 0.28132471534627 --> Accuracy - 0.32000000476837165\n",
      "Epoch: 0182 , Time:  00:00:06:10\n",
      "Cost - 0.120967188421491 --> Accuracy - 0.3300000071525574\n",
      "Epoch: 0183 , Time:  00:00:06:12\n",
      "Cost - 0.0536994707726989 --> Accuracy - 0.3300000064074993\n",
      "Epoch: 0184 , Time:  00:00:06:14\n",
      "Cost - 0.04529439862235 --> Accuracy - 0.3500000067055226\n",
      "Epoch: 0185 , Time:  00:00:06:16\n",
      "Cost - 0.0118635222708689 --> Accuracy - 0.35000000745058063\n",
      "Epoch: 0186 , Time:  00:00:06:18\n",
      "Cost - 0.0093686056832666 --> Accuracy - 0.35000000745058063\n",
      "Epoch: 0187 , Time:  00:00:06:21\n",
      "Cost - 0.008304375702573 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0188 , Time:  00:00:06:23\n",
      "Cost - 0.0077886630642751 --> Accuracy - 0.3600000075995922\n",
      "Epoch: 0189 , Time:  00:00:06:25\n",
      "Cost - 0.00746612610091688 --> Accuracy - 0.3600000075995922\n",
      "Epoch: 0190 , Time:  00:00:06:27\n",
      "Cost - 0.00699929603761 --> Accuracy - 0.3600000075995922\n",
      "Epoch: 0191 , Time:  00:00:06:30\n",
      "Cost - 0.0070303884262102 --> Accuracy - 0.37000000923871995\n",
      "Epoch: 0192 , Time:  00:00:06:32\n",
      "Cost - 0.0066083642548619 --> Accuracy - 0.3700000070035458\n",
      "Epoch: 0193 , Time:  00:00:06:34\n",
      "Cost - 0.0061616079481609 --> Accuracy - 0.38000000864267347\n",
      "Epoch: 0194 , Time:  00:00:06:36\n",
      "Cost - 0.0061466391125577 --> Accuracy - 0.3700000077486038\n",
      "Epoch: 0195 , Time:  00:00:06:38\n",
      "Cost - 0.005670090804414 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0196 , Time:  00:00:06:41\n",
      "Cost - 0.005568484287505 --> Accuracy - 0.3500000081956387\n",
      "Epoch: 0197 , Time:  00:00:06:43\n",
      "Cost - 0.0053849396947043 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0198 , Time:  00:00:06:45\n",
      "Cost - 0.00486619585572043 --> Accuracy - 0.3900000095367432\n",
      "Epoch: 0199 , Time:  00:00:06:47\n",
      "Cost - 0.0053563657678751 --> Accuracy - 0.3600000075995922\n",
      "Epoch: 0200 , Time:  00:00:06:49\n",
      "Cost - 0.00549523852696438 --> Accuracy - 0.38000000789761545\n",
      "Epoch: 0201 , Time:  00:00:06:51\n",
      "Cost - 0.00577689850160823 --> Accuracy - 0.39000000730156903\n",
      "Epoch: 0202 , Time:  00:00:06:53\n",
      "Cost - 0.0121693831128141 --> Accuracy - 0.38000000789761545\n",
      "Epoch: 0203 , Time:  00:00:06:56\n",
      "Cost - 0.041634870127745 --> Accuracy - 0.34000000655651097\n",
      "Epoch: 0204 , Time:  00:00:06:58\n",
      "Cost - 0.27496136862391 --> Accuracy - 0.31000000759959223\n",
      "Epoch: 0205 , Time:  00:00:07:00\n",
      "Cost - 0.355979105352889 --> Accuracy - 0.3400000058114529\n",
      "Epoch: 0206 , Time:  00:00:07:02\n",
      "Cost - 0.116497553963563 --> Accuracy - 0.3600000083446503\n",
      "Epoch: 0207 , Time:  00:00:07:04\n",
      "Cost - 0.037381884899514 --> Accuracy - 0.36000000834465035\n",
      "Epoch: 0208 , Time:  00:00:07:06\n",
      "Cost - 0.0146220021008048 --> Accuracy - 0.3700000077486038\n",
      "Epoch: 0209 , Time:  00:00:07:08\n",
      "Cost - 0.0097727524314905 --> Accuracy - 0.3700000084936619\n",
      "Epoch: 0210 , Time:  00:00:07:10\n",
      "Cost - 0.007564159404137 --> Accuracy - 0.3700000084936619\n",
      "Epoch: 0211 , Time:  00:00:07:12\n",
      "Cost - 0.0067165869557356 --> Accuracy - 0.3700000084936619\n",
      "Epoch: 0212 , Time:  00:00:07:14\n",
      "Cost - 0.0063711313578096 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0213 , Time:  00:00:07:16\n",
      "Cost - 0.0056894290890340 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0214 , Time:  00:00:07:19\n",
      "Cost - 0.005531094388061 --> Accuracy - 0.3900000087916851\n",
      "Epoch: 0215 , Time:  00:00:07:21\n",
      "Cost - 0.0050510032342572 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0216 , Time:  00:00:07:23\n",
      "Cost - 0.0046587282040491 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0217 , Time:  00:00:07:25\n",
      "Cost - 0.0045136540851672 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0218 , Time:  00:00:07:27\n",
      "Cost - 0.0043009312139474 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0219 , Time:  00:00:07:29\n",
      "Cost - 0.0041077782279899 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0220 , Time:  00:00:07:31\n",
      "Cost - 0.00401685992037528 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0221 , Time:  00:00:07:34\n",
      "Cost - 0.0039105040734284 --> Accuracy - 0.3700000084936619\n",
      "Epoch: 0222 , Time:  00:00:07:36\n",
      "Cost - 0.00379723481964902 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0223 , Time:  00:00:07:38\n",
      "Cost - 0.00360875594778917 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0224 , Time:  00:00:07:40\n",
      "Cost - 0.0034835572489100 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0225 , Time:  00:00:07:42\n",
      "Cost - 0.00334855864166456 --> Accuracy - 0.3700000084936619\n",
      "Epoch: 0226 , Time:  00:00:07:45\n",
      "Cost - 0.00320276337704399 --> Accuracy - 0.39000000804662704\n",
      "Epoch: 0227 , Time:  00:00:07:47\n",
      "Cost - 0.003123818356471 --> Accuracy - 0.3900000087916851\n",
      "Epoch: 0228 , Time:  00:00:07:49\n",
      "Cost - 0.00316223610743691 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0229 , Time:  00:00:07:51\n",
      "Cost - 0.00289758237595378 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0230 , Time:  00:00:07:53\n",
      "Cost - 0.00293617945553705 --> Accuracy - 0.38000000789761545\n",
      "Epoch: 0231 , Time:  00:00:07:55\n",
      "Cost - 0.0028857596207672 --> Accuracy - 0.4000000074505806\n",
      "Epoch: 0232 , Time:  00:00:07:57\n",
      "Cost - 0.00274094731157674 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0233 , Time:  00:00:08:00\n",
      "Cost - 0.0027431440143118 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0234 , Time:  00:00:08:02\n",
      "Cost - 0.00257856415410060 --> Accuracy - 0.4000000081956387\n",
      "Epoch: 0235 , Time:  00:00:08:04\n",
      "Cost - 0.00250016879999748 --> Accuracy - 0.38000000789761545\n",
      "Epoch: 0236 , Time:  00:00:08:06\n",
      "Cost - 0.00252298061714100 --> Accuracy - 0.4000000074505806\n",
      "Epoch: 0237 , Time:  00:00:08:08\n",
      "Cost - 0.0024128264707542 --> Accuracy - 0.3600000083446503\n",
      "Epoch: 0238 , Time:  00:00:08:11\n",
      "Cost - 0.0023918100411719 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0239 , Time:  00:00:08:13\n",
      "Cost - 0.0021489623261004 --> Accuracy - 0.41000000759959226\n",
      "Epoch: 0240 , Time:  00:00:08:15\n",
      "Cost - 0.0022127880791231 --> Accuracy - 0.3600000075995922\n",
      "Epoch: 0241 , Time:  00:00:08:17\n",
      "Cost - 0.0021373214731283 --> Accuracy - 0.38000000789761545\n",
      "Epoch: 0242 , Time:  00:00:08:19\n",
      "Cost - 0.00209793647500191 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0243 , Time:  00:00:08:21\n",
      "Cost - 0.0021011581296079 --> Accuracy - 0.3800000078976155\n",
      "Epoch: 0244 , Time:  00:00:08:23\n",
      "Cost - 0.021743156016600 --> Accuracy - 0.34000000655651097\n",
      "Epoch: 0245 , Time:  00:00:08:26\n",
      "Cost - 0.62295483732705 --> Accuracy - 0.2700000062584877\n",
      "Epoch: 0246 , Time:  00:00:08:28\n",
      "Cost - 0.44547227108851 --> Accuracy - 0.3700000062584877\n",
      "Epoch: 0247 , Time:  00:00:08:30\n",
      "Cost - 0.124456550773175 --> Accuracy - 0.33000000566244125\n",
      "Epoch: 0248 , Time:  00:00:08:32\n",
      "Cost - 0.040776854856085 --> Accuracy - 0.3200000062584878\n",
      "Epoch: 0249 , Time:  00:00:08:34\n",
      "Cost - 0.026518652038066 --> Accuracy - 0.3500000081956387\n",
      "Epoch: 0250 , Time:  00:00:08:36\n",
      "Cost - 0.014114358654842 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0251 , Time:  00:00:08:38\n",
      "Cost - 0.009043881310208 --> Accuracy - 0.3400000073015691\n",
      "Epoch: 0252 , Time:  00:00:08:40\n",
      "Cost - 0.0079001375190273 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0253 , Time:  00:00:08:43\n",
      "Cost - 0.0061481501354137 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0254 , Time:  00:00:08:45\n",
      "Cost - 0.0055819091683360 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0255 , Time:  00:00:08:47\n",
      "Cost - 0.0051779575085674 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0256 , Time:  00:00:08:49\n",
      "Cost - 0.0048626538566168 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0257 , Time:  00:00:08:51\n",
      "Cost - 0.0045072907143548 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0258 , Time:  00:00:08:53\n",
      "Cost - 0.0042205657278827 --> Accuracy - 0.38000000789761545\n",
      "Epoch: 0259 , Time:  00:00:08:56\n",
      "Cost - 0.0040220751525339 --> Accuracy - 0.38000000789761545\n",
      "Epoch: 0260 , Time:  00:00:08:58\n",
      "Cost - 0.0038241741840465 --> Accuracy - 0.3600000083446503\n",
      "Epoch: 0261 , Time:  00:00:09:00\n",
      "Cost - 0.00358131123120983 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0262 , Time:  00:00:09:02\n",
      "Cost - 0.00340116068327915 --> Accuracy - 0.3500000089406967\n",
      "Epoch: 0263 , Time:  00:00:09:04\n",
      "Cost - 0.0032924097939030 --> Accuracy - 0.3600000083446503\n",
      "Epoch: 0264 , Time:  00:00:09:06\n",
      "Cost - 0.00311096695149899 --> Accuracy - 0.36000000908970836\n",
      "Epoch: 0265 , Time:  00:00:09:08\n",
      "Cost - 0.0030386132510102 --> Accuracy - 0.3500000081956387\n",
      "Epoch: 0266 , Time:  00:00:09:11\n",
      "Cost - 0.00290545956158894 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0267 , Time:  00:00:09:13\n",
      "Cost - 0.002759320263066 --> Accuracy - 0.3500000089406967\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0268 , Time:  00:00:09:15\n",
      "Cost - 0.00269423361914 --> Accuracy - 0.34000000804662706\n",
      "Epoch: 0269 , Time:  00:00:09:17\n",
      "Cost - 0.0025450825942243 --> Accuracy - 0.34000000804662706\n",
      "Epoch: 0270 , Time:  00:00:09:19\n",
      "Cost - 0.0024510590938007 --> Accuracy - 0.35000000745058063\n",
      "Epoch: 0271 , Time:  00:00:09:21\n",
      "Cost - 0.0024144575577338 --> Accuracy - 0.3600000083446503\n",
      "Epoch: 0272 , Time:  00:00:09:24\n",
      "Cost - 0.00230415125315630 --> Accuracy - 0.3600000083446503\n",
      "Epoch: 0273 , Time:  00:00:09:26\n",
      "Cost - 0.0022059670522139 --> Accuracy - 0.34000000804662706\n",
      "Epoch: 0274 , Time:  00:00:09:28\n",
      "Cost - 0.0021946346914774 --> Accuracy - 0.37000000849366194\n",
      "Epoch: 0275 , Time:  00:00:09:30\n",
      "Cost - 0.00207166080435854 --> Accuracy - 0.36000000908970836\n",
      "Epoch: 0276 , Time:  00:00:09:32\n",
      "Cost - 0.00197604470176884 --> Accuracy - 0.3500000089406967\n",
      "Epoch: 0277 , Time:  00:00:09:34\n",
      "Cost - 0.00199267351090384 --> Accuracy - 0.36000000908970836\n",
      "Epoch: 0278 , Time:  00:00:09:36\n",
      "Cost - 0.00187951947045803 --> Accuracy - 0.35000000745058063\n",
      "Epoch: 0279 , Time:  00:00:09:38\n",
      "Cost - 0.0018464819549080 --> Accuracy - 0.3600000075995923\n",
      "Epoch: 0280 , Time:  00:00:09:40\n",
      "Cost - 0.0017569552030181 --> Accuracy - 0.3600000083446503\n",
      "Epoch: 0281 , Time:  00:00:09:42\n",
      "Cost - 0.001732125230410 --> Accuracy - 0.3600000083446503\n",
      "Epoch: 0282 , Time:  00:00:09:45\n",
      "Cost - 0.00159775710126268 --> Accuracy - 0.37000000849366194\n",
      "Epoch: 0283 , Time:  00:00:09:47\n",
      "Cost - 0.0016194869380342 --> Accuracy - 0.37000000849366194\n",
      "Epoch: 0284 , Time:  00:00:09:49\n",
      "Cost - 0.00159068817374645 --> Accuracy - 0.3500000089406967\n",
      "Epoch: 0285 , Time:  00:00:09:51\n",
      "Cost - 0.00153547078134579 --> Accuracy - 0.34000000804662706\n",
      "Epoch: 0286 , Time:  00:00:09:53\n",
      "Cost - 0.00145864907226496 --> Accuracy - 0.3600000083446503\n",
      "Epoch: 0287 , Time:  00:00:09:55\n",
      "Cost - 0.00144388091084692 --> Accuracy - 0.35000000745058063\n",
      "Epoch: 0288 , Time:  00:00:09:58\n",
      "Cost - 0.00132892102847108 --> Accuracy - 0.3600000083446503\n",
      "Epoch: 0289 , Time:  00:00:10:00\n",
      "Cost - 0.00132961247763887 --> Accuracy - 0.35000000745058063\n",
      "Epoch: 0290 , Time:  00:00:10:02\n",
      "Cost - 0.00132669884096685 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0291 , Time:  00:00:10:04\n",
      "Cost - 0.00124744189747616 --> Accuracy - 0.3400000080466271\n",
      "Epoch: 0292 , Time:  00:00:10:06\n",
      "Cost - 0.00121532147504694 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0293 , Time:  00:00:10:08\n",
      "Cost - 0.00124084433791722 --> Accuracy - 0.37000000700354585\n",
      "Epoch: 0294 , Time:  00:00:10:10\n",
      "Cost - 0.00117566343451471 --> Accuracy - 0.34000000655651097\n",
      "Epoch: 0295 , Time:  00:00:10:12\n",
      "Cost - 0.00114288121294521 --> Accuracy - 0.3600000068545342\n",
      "Epoch: 0296 , Time:  00:00:10:15\n",
      "Cost - 0.00108548885673371 --> Accuracy - 0.3500000089406967\n",
      "Epoch: 0297 , Time:  00:00:10:17\n",
      "Cost - 0.00107583826484187 --> Accuracy - 0.37000000700354585\n",
      "Epoch: 0298 , Time:  00:00:10:19\n",
      "Cost - 0.00104084005902223 --> Accuracy - 0.3700000084936619\n",
      "Epoch: 0299 , Time:  00:00:10:21\n",
      "Cost - 0.00096199490080834 --> Accuracy - 0.37000000700354585\n",
      "Epoch: 0300 , Time:  00:00:10:23\n",
      "Cost - 0.00095723428958081 --> Accuracy - 0.38000000789761545\n",
      "Epoch: 0301 , Time:  00:00:10:25\n",
      "Cost - 0.00098635459294200 --> Accuracy - 0.34000000655651097\n",
      "Epoch: 0302 , Time:  00:00:10:27\n",
      "Cost - 0.00204440766337938 --> Accuracy - 0.39000000804662716\n",
      "Epoch: 0303 , Time:  00:00:10:30\n",
      "Cost - 0.0029481030032911 --> Accuracy - 0.3200000055134297\n",
      "Epoch: 0304 , Time:  00:00:10:32\n",
      "Cost - 0.84874075933576 --> Accuracy - 0.3400000087916852\n",
      "Epoch: 0305 , Time:  00:00:10:34\n",
      "Cost - 0.411739116243552 --> Accuracy - 0.31000000685453416\n",
      "Epoch: 0306 , Time:  00:00:10:36\n",
      "Cost - 0.133703470231557 --> Accuracy - 0.3400000080466271\n",
      "Epoch: 0307 , Time:  00:00:10:39\n",
      "Cost - 0.057940723342471 --> Accuracy - 0.39000000581145294\n",
      "Epoch: 0308 , Time:  00:00:10:41\n",
      "Cost - 0.019440885160292 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0309 , Time:  00:00:10:43\n",
      "Cost - 0.0096217863057972 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0310 , Time:  00:00:10:45\n",
      "Cost - 0.0061532691569482 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0311 , Time:  00:00:10:47\n",
      "Cost - 0.0053466978039068 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0312 , Time:  00:00:10:49\n",
      "Cost - 0.0048652302786649 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0313 , Time:  00:00:10:52\n",
      "Cost - 0.0044484438065410 --> Accuracy - 0.38000000789761545\n",
      "Epoch: 0314 , Time:  00:00:10:54\n",
      "Cost - 0.004076185112899 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0315 , Time:  00:00:10:56\n",
      "Cost - 0.00378165550209814 --> Accuracy - 0.4000000081956387\n",
      "Epoch: 0316 , Time:  00:00:10:58\n",
      "Cost - 0.00358723156969062 --> Accuracy - 0.4000000081956387\n",
      "Epoch: 0317 , Time:  00:00:11:00\n",
      "Cost - 0.0033317183912004 --> Accuracy - 0.4000000081956387\n",
      "Epoch: 0318 , Time:  00:00:11:02\n",
      "Cost - 0.00312464051567076 --> Accuracy - 0.4000000081956387\n",
      "Epoch: 0319 , Time:  00:00:11:04\n",
      "Cost - 0.00294833645548351 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0320 , Time:  00:00:11:06\n",
      "Cost - 0.00277896926563698 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0321 , Time:  00:00:11:08\n",
      "Cost - 0.00265705333113146 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0322 , Time:  00:00:11:10\n",
      "Cost - 0.00253639958582425 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0323 , Time:  00:00:11:13\n",
      "Cost - 0.0024004716498893 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0324 , Time:  00:00:11:15\n",
      "Cost - 0.00227460130281542 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0325 , Time:  00:00:11:17\n",
      "Cost - 0.00217631686659842 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0326 , Time:  00:00:11:19\n",
      "Cost - 0.0020706129146674 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0327 , Time:  00:00:11:21\n",
      "Cost - 0.00199334395801997 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0328 , Time:  00:00:11:23\n",
      "Cost - 0.00191539807028675 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0329 , Time:  00:00:11:25\n",
      "Cost - 0.00182412400718021 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0330 , Time:  00:00:11:27\n",
      "Cost - 0.00175472412447561 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0331 , Time:  00:00:11:29\n",
      "Cost - 0.00165797177887725 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0332 , Time:  00:00:11:31\n",
      "Cost - 0.00159137217269744 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0333 , Time:  00:00:11:33\n",
      "Cost - 0.00154216922765044 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0334 , Time:  00:00:11:35\n",
      "Cost - 0.00147461751642765 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0335 , Time:  00:00:11:37\n",
      "Cost - 0.0014183906130710 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0336 , Time:  00:00:11:39\n",
      "Cost - 0.0013747270191743 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0337 , Time:  00:00:11:41\n",
      "Cost - 0.0013247697233964 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0338 , Time:  00:00:11:43\n",
      "Cost - 0.00126799424106138 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0339 , Time:  00:00:11:46\n",
      "Cost - 0.0012276428492441 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0340 , Time:  00:00:11:48\n",
      "Cost - 0.0011938013330109 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0341 , Time:  00:00:11:50\n",
      "Cost - 0.00113862852076636 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0342 , Time:  00:00:11:52\n",
      "Cost - 0.00112711816327646 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0343 , Time:  00:00:11:54\n",
      "Cost - 0.0010737727443938 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0344 , Time:  00:00:11:56\n",
      "Cost - 0.00103319700970132 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0345 , Time:  00:00:11:58\n",
      "Cost - 0.00099807288893316 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0346 , Time:  00:00:12:00\n",
      "Cost - 0.00097977743976116 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0347 , Time:  00:00:12:02\n",
      "Cost - 0.00093620900511268 --> Accuracy - 0.3800000086426735\n",
      "Epoch: 0348 , Time:  00:00:12:04\n",
      "Cost - 0.00090480129565548 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0349 , Time:  00:00:12:07\n",
      "Cost - 0.00088240868299180 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0350 , Time:  00:00:12:09\n",
      "Cost - 0.00084684787325386 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0351 , Time:  00:00:12:11\n",
      "Cost - 0.00081534539813219 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0352 , Time:  00:00:12:13\n",
      "Cost - 0.00079833284326014 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0353 , Time:  00:00:12:15\n",
      "Cost - 0.00075707748865170 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0354 , Time:  00:00:12:17\n",
      "Cost - 0.00075895427453360 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0355 , Time:  00:00:12:19\n",
      "Cost - 0.00072601162551109 --> Accuracy - 0.38000000715255744\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0356 , Time:  00:00:12:22\n",
      "Cost - 0.00070816836246194 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0357 , Time:  00:00:12:24\n",
      "Cost - 0.00069148248753663 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0358 , Time:  00:00:12:26\n",
      "Cost - 0.00065936980713559 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0359 , Time:  00:00:12:28\n",
      "Cost - 0.00064602962984281 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0360 , Time:  00:00:12:30\n",
      "Cost - 0.00062830074778759 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0361 , Time:  00:00:12:32\n",
      "Cost - 0.00061117545360502 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0362 , Time:  00:00:12:35\n",
      "Cost - 0.00058500331174400 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0363 , Time:  00:00:12:37\n",
      "Cost - 0.00056326555636587 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0364 , Time:  00:00:12:39\n",
      "Cost - 0.00055161616710392 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0365 , Time:  00:00:12:41\n",
      "Cost - 0.00055538244705530 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0366 , Time:  00:00:12:43\n",
      "Cost - 0.00052344092919156 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0367 , Time:  00:00:12:45\n",
      "Cost - 0.0004977753619641 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0368 , Time:  00:00:12:47\n",
      "Cost - 0.00049104041333976 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0369 , Time:  00:00:12:50\n",
      "Cost - 0.00049279564955213 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0370 , Time:  00:00:12:52\n",
      "Cost - 0.00047569770540576 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0371 , Time:  00:00:12:54\n",
      "Cost - 0.000452891125996757 --> Accuracy - 0.37000000774860387\n",
      "Epoch: 0372 , Time:  00:00:12:56\n",
      "Cost - 0.000449991474079070 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0373 , Time:  00:00:12:58\n",
      "Cost - 0.000422677642927737 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0374 , Time:  00:00:13:00\n",
      "Cost - 0.000419045021635611 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0375 , Time:  00:00:13:02\n",
      "Cost - 0.000396440645135953 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0376 , Time:  00:00:13:05\n",
      "Cost - 0.00040407387969025 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0377 , Time:  00:00:13:07\n",
      "Cost - 0.00037676165852644 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0378 , Time:  00:00:13:09\n",
      "Cost - 0.00036213962507645 --> Accuracy - 0.4000000074505807\n",
      "Epoch: 0379 , Time:  00:00:13:11\n",
      "Cost - 0.00038035856314309 --> Accuracy - 0.3900000080466271\n",
      "Epoch: 0380 , Time:  00:00:13:13\n",
      "Cost - 0.0003579816534056 --> Accuracy - 0.3700000070035458\n",
      "Epoch: 0381 , Time:  00:00:13:15\n",
      "Cost - 0.00033470482389930 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0382 , Time:  00:00:13:17\n",
      "Cost - 0.00032853553732365 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0383 , Time:  00:00:13:19\n",
      "Cost - 0.00030884895654139 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0384 , Time:  00:00:13:22\n",
      "Cost - 0.00031376789715977 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0385 , Time:  00:00:13:24\n",
      "Cost - 0.00030170767474260 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0386 , Time:  00:00:13:26\n",
      "Cost - 0.000296360867241674 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0387 , Time:  00:00:13:28\n",
      "Cost - 0.0002905735677700 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0388 , Time:  00:00:13:30\n",
      "Cost - 0.000286473617336469 --> Accuracy - 0.39000000730156903\n",
      "Epoch: 0389 , Time:  00:00:13:32\n",
      "Cost - 0.000267641772506976 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0390 , Time:  00:00:13:34\n",
      "Cost - 0.00027437687376732 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0391 , Time:  00:00:13:36\n",
      "Cost - 0.0002529401053834 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0392 , Time:  00:00:13:38\n",
      "Cost - 0.000291755832404305 --> Accuracy - 0.38000000715255744\n",
      "Epoch: 0393 , Time:  00:00:13:40\n",
      "Cost - 0.95796937347126 --> Accuracy - 0.3200000062584878\n",
      "Epoch: 0394 , Time:  00:00:13:42\n",
      "Cost - 0.22866472342757 --> Accuracy - 0.3100000061094762\n",
      "Epoch: 0395 , Time:  00:00:13:45\n",
      "Cost - 0.09037166157904 --> Accuracy - 0.3300000071525574\n",
      "Epoch: 0396 , Time:  00:00:13:47\n",
      "Cost - 0.017971205997673 --> Accuracy - 0.3100000061094761\n",
      "Epoch: 0397 , Time:  00:00:13:49\n",
      "Cost - 0.0069526499332823 --> Accuracy - 0.36000000685453415\n",
      "Epoch: 0398 , Time:  00:00:13:51\n",
      "Cost - 0.0045952886139275 --> Accuracy - 0.3400000065565109\n",
      "Epoch: 0399 , Time:  00:00:13:53\n",
      "Cost - 0.00370523916742513 --> Accuracy - 0.3300000064074993\n",
      "Epoch: 0400 , Time:  00:00:13:55\n",
      "Cost - 0.003276076340262 --> Accuracy - 0.3300000064074993\n",
      "Optimization Finished!\n",
      "Time taken: day, hour, minutes, seconds-> 00:00:13:55\n"
     ]
    }
   ],
   "source": [
    "# Start Training\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    # Initialize variables\n",
    "    sess.run(init)\n",
    "    \n",
    "    # op to write logs to Tensorboard\n",
    "    summary_writer = tf.summary.FileWriter(logs_path, graph=tf.get_default_graph())\n",
    "    \n",
    "    # Training cycle\n",
    "    for epoch in range(training_epochs):\n",
    "        avg_cost_list = 0.0\n",
    "        total_batch = int(len(haptic_frames_2d_train)/batch)\n",
    "        \n",
    "        # Shuffle data\n",
    "        shuffle_indices = np.arange(len(haptic_frames_2d_train))\n",
    "        np.random.shuffle(shuffle_indices)\n",
    "        \n",
    "        i = 0\n",
    "        # Loop over all batches\n",
    "        for start, end in zip(range(0, len(haptic_frames_2d_train), batch), range(batch, len(haptic_frames_2d_train)+1, batch)):\n",
    "            input_data, label_data = haptic_frames_2d_train[shuffle_indices][start:end], category_label_train_one_not[shuffle_indices][start:end]\n",
    "            X = tf.get_default_graph().get_tensor_by_name(\"Model/InputData:0\")\n",
    "            _, new_cost, cost_scalar = sess.run([model_dict[\"Optimizer\"], model_dict[\"Loss\"], model_dict[\"Loss_scalar\"]], feed_dict={X: input_data, Y: label_data})\n",
    "            # Compute average loss\n",
    "            avg_cost_list += new_cost/total_batch\n",
    "            summary_writer.add_summary(cost_scalar, epoch * total_batch + i)\n",
    "\n",
    "            i += 1\n",
    "            \n",
    "        save_path = saver.save(sess, model_path, epoch)\n",
    "         \n",
    "        # Calculate Accuracy\n",
    "        avg_accuracy_list = 0.0\n",
    "        total_batch = int(len(haptic_frames_2d_test)/batch)\n",
    "        i = 0\n",
    "        for start, end in zip(range(0, len(haptic_frames_2d_test), batch), range(batch, len(haptic_frames_2d_test)+1, batch)):\n",
    "            input_data, label_data = haptic_frames_2d_test[start:end], category_label_test_one_not[start:end]\n",
    "            X = tf.get_default_graph().get_tensor_by_name(\"Model/InputData:0\")\n",
    "            accuracy, accuracy_scalar = sess.run([model_dict[\"Accuracy\"], model_dict[\"Accuracy_scalar\"]], feed_dict={X: input_data, Y: label_data})\n",
    "            # Compute average accuracy\n",
    "            avg_accuracy_list += accuracy/total_batch\n",
    "            summary_writer.add_summary(accuracy_scalar, epoch * total_batch + i)\n",
    "            \n",
    "            i += 1\n",
    "        \n",
    "        # Printing current epoch accuracy\n",
    "        epoch_cost_accuracy = []\n",
    "        epoch_cost_accuracy.append(time_taken(start_time, time.time()))\n",
    "        # Display logs per epoch step\n",
    "        if epoch % display_step == 0:\n",
    "            print(\"Epoch:\", '%04d' % (epoch+1), \", Time: \", time_taken(start_time, time.time()))\n",
    "            a_string = \"Cost - \"\n",
    "            epoch_cost_accuracy.append(epoch+1)\n",
    "            \n",
    "            a_string += str(avg_cost_list)\n",
    "            epoch_cost_accuracy.append(str(avg_cost_list))\n",
    "            \n",
    "            a_string = a_string[0:-2]+\" --> Accuracy - \"\n",
    "            a_string += str(avg_accuracy_list)\n",
    "            epoch_cost_accuracy.append(str(avg_accuracy_list))\n",
    "            \n",
    "            print(a_string)\n",
    "        \n",
    "        # Writing current epoch data\n",
    "        with open(logs_path+db_file_name.split(\".\")[0]+\"_data.csv\", 'a') as f: # append to the file created\n",
    "            writer = csv.writer(f, lineterminator=\"\\n\")\n",
    "            writer.writerow(epoch_cost_accuracy)\n",
    "    \n",
    "    print(\"Optimization Finished!\")\n",
    "    end_time = time.time()\n",
    "    print(\"Time taken: day, hour, minutes, seconds->\", time_taken(start_time, end_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}