2_CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CSE6250BDH Deep Learning Labs\n",
    "## 2. Convolutional Neural Network\n",
    "\n",
    "In this chapter, we will learn how to implement a convolutional neural network by using PyTorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 0. Prepare Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it is said in the lab web page, we will use [Epileptic Seizure Recognition Data Set](http://archive.ics.uci.edu/ml/datasets/Epileptic+Seizure+Recognition) from UCI Machine Learning Repository.\n",
    "\n",
    "As the dataset is formatted in CSV form, we will use Pandas for convenience. Pandas is useful for preprocess such a structured data and easy to connect to most machine learning pipelines, e.g. Scikit-learn and PyTorch we are using here, since it uses Numpy as its backend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>X1</th>\n",
       "      <th>X2</th>\n",
       "      <th>X3</th>\n",
       "      <th>X4</th>\n",
       "      <th>X5</th>\n",
       "      <th>X6</th>\n",
       "      <th>X7</th>\n",
       "      <th>X8</th>\n",
       "      <th>X9</th>\n",
       "      <th>...</th>\n",
       "      <th>X170</th>\n",
       "      <th>X171</th>\n",
       "      <th>X172</th>\n",
       "      <th>X173</th>\n",
       "      <th>X174</th>\n",
       "      <th>X175</th>\n",
       "      <th>X176</th>\n",
       "      <th>X177</th>\n",
       "      <th>X178</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>X21.V1.791</td>\n",
       "      <td>135</td>\n",
       "      <td>190</td>\n",
       "      <td>229</td>\n",
       "      <td>223</td>\n",
       "      <td>192</td>\n",
       "      <td>125</td>\n",
       "      <td>55</td>\n",
       "      <td>-9</td>\n",
       "      <td>-33</td>\n",
       "      <td>...</td>\n",
       "      <td>-17</td>\n",
       "      <td>-15</td>\n",
       "      <td>-31</td>\n",
       "      <td>-77</td>\n",
       "      <td>-103</td>\n",
       "      <td>-127</td>\n",
       "      <td>-116</td>\n",
       "      <td>-83</td>\n",
       "      <td>-51</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>X15.V1.924</td>\n",
       "      <td>386</td>\n",
       "      <td>382</td>\n",
       "      <td>356</td>\n",
       "      <td>331</td>\n",
       "      <td>320</td>\n",
       "      <td>315</td>\n",
       "      <td>307</td>\n",
       "      <td>272</td>\n",
       "      <td>244</td>\n",
       "      <td>...</td>\n",
       "      <td>164</td>\n",
       "      <td>150</td>\n",
       "      <td>146</td>\n",
       "      <td>152</td>\n",
       "      <td>157</td>\n",
       "      <td>156</td>\n",
       "      <td>154</td>\n",
       "      <td>143</td>\n",
       "      <td>129</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>X8.V1.1</td>\n",
       "      <td>-32</td>\n",
       "      <td>-39</td>\n",
       "      <td>-47</td>\n",
       "      <td>-37</td>\n",
       "      <td>-32</td>\n",
       "      <td>-36</td>\n",
       "      <td>-57</td>\n",
       "      <td>-73</td>\n",
       "      <td>-85</td>\n",
       "      <td>...</td>\n",
       "      <td>57</td>\n",
       "      <td>64</td>\n",
       "      <td>48</td>\n",
       "      <td>19</td>\n",
       "      <td>-12</td>\n",
       "      <td>-30</td>\n",
       "      <td>-35</td>\n",
       "      <td>-35</td>\n",
       "      <td>-36</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>X16.V1.60</td>\n",
       "      <td>-105</td>\n",
       "      <td>-101</td>\n",
       "      <td>-96</td>\n",
       "      <td>-92</td>\n",
       "      <td>-89</td>\n",
       "      <td>-95</td>\n",
       "      <td>-102</td>\n",
       "      <td>-100</td>\n",
       "      <td>-87</td>\n",
       "      <td>...</td>\n",
       "      <td>-82</td>\n",
       "      <td>-81</td>\n",
       "      <td>-80</td>\n",
       "      <td>-77</td>\n",
       "      <td>-85</td>\n",
       "      <td>-77</td>\n",
       "      <td>-72</td>\n",
       "      <td>-69</td>\n",
       "      <td>-65</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>X20.V1.54</td>\n",
       "      <td>-9</td>\n",
       "      <td>-65</td>\n",
       "      <td>-98</td>\n",
       "      <td>-102</td>\n",
       "      <td>-78</td>\n",
       "      <td>-48</td>\n",
       "      <td>-16</td>\n",
       "      <td>0</td>\n",
       "      <td>-21</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>-12</td>\n",
       "      <td>-32</td>\n",
       "      <td>-41</td>\n",
       "      <td>-65</td>\n",
       "      <td>-83</td>\n",
       "      <td>-89</td>\n",
       "      <td>-73</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>X14.V1.56</td>\n",
       "      <td>55</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>19</td>\n",
       "      <td>25</td>\n",
       "      <td>40</td>\n",
       "      <td>52</td>\n",
       "      <td>...</td>\n",
       "      <td>-12</td>\n",
       "      <td>-31</td>\n",
       "      <td>-42</td>\n",
       "      <td>-54</td>\n",
       "      <td>-60</td>\n",
       "      <td>-64</td>\n",
       "      <td>-60</td>\n",
       "      <td>-56</td>\n",
       "      <td>-55</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>X3.V1.191</td>\n",
       "      <td>-55</td>\n",
       "      <td>-9</td>\n",
       "      <td>52</td>\n",
       "      <td>111</td>\n",
       "      <td>135</td>\n",
       "      <td>129</td>\n",
       "      <td>103</td>\n",
       "      <td>72</td>\n",
       "      <td>37</td>\n",
       "      <td>...</td>\n",
       "      <td>-125</td>\n",
       "      <td>-99</td>\n",
       "      <td>-79</td>\n",
       "      <td>-62</td>\n",
       "      <td>-41</td>\n",
       "      <td>-26</td>\n",
       "      <td>11</td>\n",
       "      <td>67</td>\n",
       "      <td>128</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>X11.V1.273</td>\n",
       "      <td>1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-8</td>\n",
       "      <td>-11</td>\n",
       "      <td>-12</td>\n",
       "      <td>-17</td>\n",
       "      <td>-15</td>\n",
       "      <td>-16</td>\n",
       "      <td>-18</td>\n",
       "      <td>...</td>\n",
       "      <td>-79</td>\n",
       "      <td>-91</td>\n",
       "      <td>-97</td>\n",
       "      <td>-88</td>\n",
       "      <td>-76</td>\n",
       "      <td>-72</td>\n",
       "      <td>-66</td>\n",
       "      <td>-57</td>\n",
       "      <td>-39</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>X19.V1.874</td>\n",
       "      <td>-278</td>\n",
       "      <td>-246</td>\n",
       "      <td>-215</td>\n",
       "      <td>-191</td>\n",
       "      <td>-177</td>\n",
       "      <td>-167</td>\n",
       "      <td>-157</td>\n",
       "      <td>-139</td>\n",
       "      <td>-118</td>\n",
       "      <td>...</td>\n",
       "      <td>-400</td>\n",
       "      <td>-379</td>\n",
       "      <td>-336</td>\n",
       "      <td>-281</td>\n",
       "      <td>-226</td>\n",
       "      <td>-174</td>\n",
       "      <td>-125</td>\n",
       "      <td>-79</td>\n",
       "      <td>-40</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>X3.V1.491</td>\n",
       "      <td>8</td>\n",
       "      <td>15</td>\n",
       "      <td>13</td>\n",
       "      <td>3</td>\n",
       "      <td>-6</td>\n",
       "      <td>-8</td>\n",
       "      <td>-5</td>\n",
       "      <td>4</td>\n",
       "      <td>25</td>\n",
       "      <td>...</td>\n",
       "      <td>49</td>\n",
       "      <td>31</td>\n",
       "      <td>11</td>\n",
       "      <td>-5</td>\n",
       "      <td>-17</td>\n",
       "      <td>-19</td>\n",
       "      <td>-15</td>\n",
       "      <td>-15</td>\n",
       "      <td>-11</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>X3.V1.6</td>\n",
       "      <td>-5</td>\n",
       "      <td>15</td>\n",
       "      <td>28</td>\n",
       "      <td>28</td>\n",
       "      <td>9</td>\n",
       "      <td>-29</td>\n",
       "      <td>-41</td>\n",
       "      <td>-19</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>-38</td>\n",
       "      <td>-4</td>\n",
       "      <td>25</td>\n",
       "      <td>16</td>\n",
       "      <td>-16</td>\n",
       "      <td>-74</td>\n",
       "      <td>-101</td>\n",
       "      <td>-89</td>\n",
       "      <td>-49</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>X21.V1.724</td>\n",
       "      <td>-167</td>\n",
       "      <td>-230</td>\n",
       "      <td>-280</td>\n",
       "      <td>-315</td>\n",
       "      <td>-338</td>\n",
       "      <td>-369</td>\n",
       "      <td>-405</td>\n",
       "      <td>-392</td>\n",
       "      <td>-298</td>\n",
       "      <td>...</td>\n",
       "      <td>423</td>\n",
       "      <td>434</td>\n",
       "      <td>416</td>\n",
       "      <td>374</td>\n",
       "      <td>319</td>\n",
       "      <td>268</td>\n",
       "      <td>215</td>\n",
       "      <td>165</td>\n",
       "      <td>103</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>X7.V1.162</td>\n",
       "      <td>92</td>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>-32</td>\n",
       "      <td>-51</td>\n",
       "      <td>-65</td>\n",
       "      <td>-37</td>\n",
       "      <td>-19</td>\n",
       "      <td>-25</td>\n",
       "      <td>...</td>\n",
       "      <td>-56</td>\n",
       "      <td>-41</td>\n",
       "      <td>-40</td>\n",
       "      <td>-43</td>\n",
       "      <td>-32</td>\n",
       "      <td>-13</td>\n",
       "      <td>-1</td>\n",
       "      <td>-7</td>\n",
       "      <td>-44</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>X1.V1.211</td>\n",
       "      <td>15</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>-17</td>\n",
       "      <td>-28</td>\n",
       "      <td>-31</td>\n",
       "      <td>-39</td>\n",
       "      <td>-51</td>\n",
       "      <td>-44</td>\n",
       "      <td>...</td>\n",
       "      <td>-88</td>\n",
       "      <td>-102</td>\n",
       "      <td>-97</td>\n",
       "      <td>-77</td>\n",
       "      <td>-45</td>\n",
       "      <td>-19</td>\n",
       "      <td>13</td>\n",
       "      <td>44</td>\n",
       "      <td>68</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>X1.V1.615</td>\n",
       "      <td>-24</td>\n",
       "      <td>-15</td>\n",
       "      <td>-5</td>\n",
       "      <td>-1</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>10</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>32</td>\n",
       "      <td>35</td>\n",
       "      <td>36</td>\n",
       "      <td>34</td>\n",
       "      <td>32</td>\n",
       "      <td>26</td>\n",
       "      <td>23</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>X22.V1.242</td>\n",
       "      <td>-135</td>\n",
       "      <td>-133</td>\n",
       "      <td>-125</td>\n",
       "      <td>-118</td>\n",
       "      <td>-111</td>\n",
       "      <td>-105</td>\n",
       "      <td>-102</td>\n",
       "      <td>-93</td>\n",
       "      <td>-94</td>\n",
       "      <td>...</td>\n",
       "      <td>-49</td>\n",
       "      <td>-39</td>\n",
       "      <td>-35</td>\n",
       "      <td>-29</td>\n",
       "      <td>-10</td>\n",
       "      <td>4</td>\n",
       "      <td>21</td>\n",
       "      <td>31</td>\n",
       "      <td>37</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>X1.V1.863</td>\n",
       "      <td>39</td>\n",
       "      <td>41</td>\n",
       "      <td>41</td>\n",
       "      <td>42</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>46</td>\n",
       "      <td>47</td>\n",
       "      <td>49</td>\n",
       "      <td>...</td>\n",
       "      <td>43</td>\n",
       "      <td>41</td>\n",
       "      <td>41</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>40</td>\n",
       "      <td>41</td>\n",
       "      <td>41</td>\n",
       "      <td>49</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>X9.V1.302</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>-5</td>\n",
       "      <td>-10</td>\n",
       "      <td>-22</td>\n",
       "      <td>-30</td>\n",
       "      <td>-33</td>\n",
       "      <td>-43</td>\n",
       "      <td>-41</td>\n",
       "      <td>...</td>\n",
       "      <td>34</td>\n",
       "      <td>27</td>\n",
       "      <td>22</td>\n",
       "      <td>18</td>\n",
       "      <td>15</td>\n",
       "      <td>13</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>X7.V1.541</td>\n",
       "      <td>-21</td>\n",
       "      <td>-5</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>19</td>\n",
       "      <td>20</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>...</td>\n",
       "      <td>43</td>\n",
       "      <td>28</td>\n",
       "      <td>25</td>\n",
       "      <td>19</td>\n",
       "      <td>30</td>\n",
       "      <td>35</td>\n",
       "      <td>26</td>\n",
       "      <td>5</td>\n",
       "      <td>-13</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>X9.V1.915</td>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>51</td>\n",
       "      <td>76</td>\n",
       "      <td>92</td>\n",
       "      <td>102</td>\n",
       "      <td>104</td>\n",
       "      <td>101</td>\n",
       "      <td>90</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>10</td>\n",
       "      <td>19</td>\n",
       "      <td>31</td>\n",
       "      <td>36</td>\n",
       "      <td>40</td>\n",
       "      <td>43</td>\n",
       "      <td>36</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>X23.V1.964</td>\n",
       "      <td>410</td>\n",
       "      <td>451</td>\n",
       "      <td>491</td>\n",
       "      <td>541</td>\n",
       "      <td>581</td>\n",
       "      <td>641</td>\n",
       "      <td>736</td>\n",
       "      <td>757</td>\n",
       "      <td>692</td>\n",
       "      <td>...</td>\n",
       "      <td>415</td>\n",
       "      <td>428</td>\n",
       "      <td>463</td>\n",
       "      <td>510</td>\n",
       "      <td>562</td>\n",
       "      <td>607</td>\n",
       "      <td>667</td>\n",
       "      <td>748</td>\n",
       "      <td>763</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>X1.V1.614</td>\n",
       "      <td>-24</td>\n",
       "      <td>-27</td>\n",
       "      <td>-23</td>\n",
       "      <td>-28</td>\n",
       "      <td>-34</td>\n",
       "      <td>-40</td>\n",
       "      <td>-47</td>\n",
       "      <td>-43</td>\n",
       "      <td>-38</td>\n",
       "      <td>...</td>\n",
       "      <td>-81</td>\n",
       "      <td>-51</td>\n",
       "      <td>-38</td>\n",
       "      <td>-11</td>\n",
       "      <td>-12</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>-4</td>\n",
       "      <td>-3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>X11.V1.134</td>\n",
       "      <td>-264</td>\n",
       "      <td>-189</td>\n",
       "      <td>-117</td>\n",
       "      <td>-45</td>\n",
       "      <td>20</td>\n",
       "      <td>70</td>\n",
       "      <td>111</td>\n",
       "      <td>143</td>\n",
       "      <td>161</td>\n",
       "      <td>...</td>\n",
       "      <td>-231</td>\n",
       "      <td>-221</td>\n",
       "      <td>-248</td>\n",
       "      <td>-321</td>\n",
       "      <td>-444</td>\n",
       "      <td>-530</td>\n",
       "      <td>-548</td>\n",
       "      <td>-536</td>\n",
       "      <td>-486</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>X18.V1.544</td>\n",
       "      <td>-4</td>\n",
       "      <td>40</td>\n",
       "      <td>78</td>\n",
       "      <td>123</td>\n",
       "      <td>149</td>\n",
       "      <td>185</td>\n",
       "      <td>197</td>\n",
       "      <td>189</td>\n",
       "      <td>167</td>\n",
       "      <td>...</td>\n",
       "      <td>-189</td>\n",
       "      <td>-214</td>\n",
       "      <td>-226</td>\n",
       "      <td>-224</td>\n",
       "      <td>-203</td>\n",
       "      <td>-171</td>\n",
       "      <td>-129</td>\n",
       "      <td>-85</td>\n",
       "      <td>-40</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>X19.V1.294</td>\n",
       "      <td>593</td>\n",
       "      <td>328</td>\n",
       "      <td>88</td>\n",
       "      <td>-106</td>\n",
       "      <td>-456</td>\n",
       "      <td>-732</td>\n",
       "      <td>-921</td>\n",
       "      <td>-782</td>\n",
       "      <td>-522</td>\n",
       "      <td>...</td>\n",
       "      <td>312</td>\n",
       "      <td>360</td>\n",
       "      <td>421</td>\n",
       "      <td>445</td>\n",
       "      <td>413</td>\n",
       "      <td>310</td>\n",
       "      <td>177</td>\n",
       "      <td>41</td>\n",
       "      <td>-71</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>X21.V1.802</td>\n",
       "      <td>-16</td>\n",
       "      <td>-15</td>\n",
       "      <td>-19</td>\n",
       "      <td>-16</td>\n",
       "      <td>-14</td>\n",
       "      <td>-5</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-3</td>\n",
       "      <td>...</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>62</td>\n",
       "      <td>54</td>\n",
       "      <td>53</td>\n",
       "      <td>53</td>\n",
       "      <td>50</td>\n",
       "      <td>57</td>\n",
       "      <td>54</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>X2.V1.72</td>\n",
       "      <td>-20</td>\n",
       "      <td>-38</td>\n",
       "      <td>-53</td>\n",
       "      <td>-58</td>\n",
       "      <td>-66</td>\n",
       "      <td>-66</td>\n",
       "      <td>-69</td>\n",
       "      <td>-77</td>\n",
       "      <td>-87</td>\n",
       "      <td>...</td>\n",
       "      <td>-26</td>\n",
       "      <td>-45</td>\n",
       "      <td>-55</td>\n",
       "      <td>-55</td>\n",
       "      <td>-46</td>\n",
       "      <td>-45</td>\n",
       "      <td>-43</td>\n",
       "      <td>-42</td>\n",
       "      <td>-42</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>X3.V1.744</td>\n",
       "      <td>-340</td>\n",
       "      <td>-381</td>\n",
       "      <td>-376</td>\n",
       "      <td>-336</td>\n",
       "      <td>-275</td>\n",
       "      <td>-204</td>\n",
       "      <td>-131</td>\n",
       "      <td>-70</td>\n",
       "      <td>-16</td>\n",
       "      <td>...</td>\n",
       "      <td>114</td>\n",
       "      <td>-39</td>\n",
       "      <td>-185</td>\n",
       "      <td>-293</td>\n",
       "      <td>-351</td>\n",
       "      <td>-379</td>\n",
       "      <td>-380</td>\n",
       "      <td>-350</td>\n",
       "      <td>-308</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>X12.V1.733</td>\n",
       "      <td>-30</td>\n",
       "      <td>15</td>\n",
       "      <td>61</td>\n",
       "      <td>80</td>\n",
       "      <td>72</td>\n",
       "      <td>41</td>\n",
       "      <td>-11</td>\n",
       "      <td>-31</td>\n",
       "      <td>-47</td>\n",
       "      <td>...</td>\n",
       "      <td>-9</td>\n",
       "      <td>-21</td>\n",
       "      <td>-17</td>\n",
       "      <td>-4</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>-6</td>\n",
       "      <td>-17</td>\n",
       "      <td>-16</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>X8.V1.614</td>\n",
       "      <td>-1</td>\n",
       "      <td>18</td>\n",
       "      <td>35</td>\n",
       "      <td>36</td>\n",
       "      <td>29</td>\n",
       "      <td>17</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>-3</td>\n",
       "      <td>...</td>\n",
       "      <td>-6</td>\n",
       "      <td>-18</td>\n",
       "      <td>-13</td>\n",
       "      <td>-18</td>\n",
       "      <td>-22</td>\n",
       "      <td>-19</td>\n",
       "      <td>-18</td>\n",
       "      <td>-6</td>\n",
       "      <td>-4</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11470</th>\n",
       "      <td>X17.V1.232</td>\n",
       "      <td>-62</td>\n",
       "      <td>-33</td>\n",
       "      <td>13</td>\n",
       "      <td>51</td>\n",
       "      <td>89</td>\n",
       "      <td>111</td>\n",
       "      <td>114</td>\n",
       "      <td>124</td>\n",
       "      <td>120</td>\n",
       "      <td>...</td>\n",
       "      <td>34</td>\n",
       "      <td>28</td>\n",
       "      <td>39</td>\n",
       "      <td>58</td>\n",
       "      <td>74</td>\n",
       "      <td>77</td>\n",
       "      <td>76</td>\n",
       "      <td>46</td>\n",
       "      <td>25</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11471</th>\n",
       "      <td>X21.V1.334</td>\n",
       "      <td>-214</td>\n",
       "      <td>-320</td>\n",
       "      <td>-418</td>\n",
       "      <td>-534</td>\n",
       "      <td>-748</td>\n",
       "      <td>-638</td>\n",
       "      <td>-470</td>\n",
       "      <td>42</td>\n",
       "      <td>454</td>\n",
       "      <td>...</td>\n",
       "      <td>1033</td>\n",
       "      <td>858</td>\n",
       "      <td>597</td>\n",
       "      <td>370</td>\n",
       "      <td>206</td>\n",
       "      <td>-122</td>\n",
       "      <td>-71</td>\n",
       "      <td>-89</td>\n",
       "      <td>172</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11472</th>\n",
       "      <td>X5.V1.332</td>\n",
       "      <td>-61</td>\n",
       "      <td>-71</td>\n",
       "      <td>-77</td>\n",
       "      <td>-74</td>\n",
       "      <td>-74</td>\n",
       "      <td>-72</td>\n",
       "      <td>-72</td>\n",
       "      <td>-76</td>\n",
       "      <td>-77</td>\n",
       "      <td>...</td>\n",
       "      <td>-51</td>\n",
       "      <td>-58</td>\n",
       "      <td>-68</td>\n",
       "      <td>-88</td>\n",
       "      <td>-98</td>\n",
       "      <td>-101</td>\n",
       "      <td>-102</td>\n",
       "      <td>-99</td>\n",
       "      <td>-96</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11473</th>\n",
       "      <td>X4.V1.173</td>\n",
       "      <td>-106</td>\n",
       "      <td>-133</td>\n",
       "      <td>-130</td>\n",
       "      <td>-111</td>\n",
       "      <td>-91</td>\n",
       "      <td>-62</td>\n",
       "      <td>-39</td>\n",
       "      <td>4</td>\n",
       "      <td>42</td>\n",
       "      <td>...</td>\n",
       "      <td>-80</td>\n",
       "      <td>-132</td>\n",
       "      <td>-151</td>\n",
       "      <td>-166</td>\n",
       "      <td>-136</td>\n",
       "      <td>-110</td>\n",
       "      <td>-90</td>\n",
       "      <td>-63</td>\n",
       "      <td>-52</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11474</th>\n",
       "      <td>X8.V1.88</td>\n",
       "      <td>-50</td>\n",
       "      <td>-48</td>\n",
       "      <td>-63</td>\n",
       "      <td>-62</td>\n",
       "      <td>-46</td>\n",
       "      <td>-24</td>\n",
       "      <td>-1</td>\n",
       "      <td>13</td>\n",
       "      <td>27</td>\n",
       "      <td>...</td>\n",
       "      <td>44</td>\n",
       "      <td>43</td>\n",
       "      <td>46</td>\n",
       "      <td>47</td>\n",
       "      <td>46</td>\n",
       "      <td>32</td>\n",
       "      <td>13</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11475</th>\n",
       "      <td>X7.V1.462</td>\n",
       "      <td>-68</td>\n",
       "      <td>-43</td>\n",
       "      <td>-24</td>\n",
       "      <td>-3</td>\n",
       "      <td>-15</td>\n",
       "      <td>-16</td>\n",
       "      <td>-15</td>\n",
       "      <td>-17</td>\n",
       "      <td>-19</td>\n",
       "      <td>...</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>90</td>\n",
       "      <td>91</td>\n",
       "      <td>96</td>\n",
       "      <td>98</td>\n",
       "      <td>102</td>\n",
       "      <td>100</td>\n",
       "      <td>91</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11476</th>\n",
       "      <td>X22.V1.984</td>\n",
       "      <td>-293</td>\n",
       "      <td>-297</td>\n",
       "      <td>-302</td>\n",
       "      <td>-307</td>\n",
       "      <td>-318</td>\n",
       "      <td>-331</td>\n",
       "      <td>-347</td>\n",
       "      <td>-361</td>\n",
       "      <td>-369</td>\n",
       "      <td>...</td>\n",
       "      <td>12</td>\n",
       "      <td>-6</td>\n",
       "      <td>314</td>\n",
       "      <td>497</td>\n",
       "      <td>826</td>\n",
       "      <td>877</td>\n",
       "      <td>778</td>\n",
       "      <td>571</td>\n",
       "      <td>274</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11477</th>\n",
       "      <td>X10.V1.421</td>\n",
       "      <td>28</td>\n",
       "      <td>36</td>\n",
       "      <td>26</td>\n",
       "      <td>4</td>\n",
       "      <td>-35</td>\n",
       "      <td>-52</td>\n",
       "      <td>-51</td>\n",
       "      <td>-39</td>\n",
       "      <td>-12</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>23</td>\n",
       "      <td>29</td>\n",
       "      <td>18</td>\n",
       "      <td>-8</td>\n",
       "      <td>-32</td>\n",
       "      <td>-42</td>\n",
       "      <td>-42</td>\n",
       "      <td>-37</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11478</th>\n",
       "      <td>X3.V1.554</td>\n",
       "      <td>452</td>\n",
       "      <td>470</td>\n",
       "      <td>401</td>\n",
       "      <td>291</td>\n",
       "      <td>163</td>\n",
       "      <td>83</td>\n",
       "      <td>20</td>\n",
       "      <td>-32</td>\n",
       "      <td>-61</td>\n",
       "      <td>...</td>\n",
       "      <td>182</td>\n",
       "      <td>237</td>\n",
       "      <td>263</td>\n",
       "      <td>211</td>\n",
       "      <td>114</td>\n",
       "      <td>48</td>\n",
       "      <td>8</td>\n",
       "      <td>-15</td>\n",
       "      <td>-48</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11479</th>\n",
       "      <td>X16.V1.353</td>\n",
       "      <td>62</td>\n",
       "      <td>68</td>\n",
       "      <td>75</td>\n",
       "      <td>75</td>\n",
       "      <td>75</td>\n",
       "      <td>77</td>\n",
       "      <td>81</td>\n",
       "      <td>85</td>\n",
       "      <td>82</td>\n",
       "      <td>...</td>\n",
       "      <td>-1</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>9</td>\n",
       "      <td>18</td>\n",
       "      <td>22</td>\n",
       "      <td>30</td>\n",
       "      <td>36</td>\n",
       "      <td>41</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11480</th>\n",
       "      <td>X13.V1.26</td>\n",
       "      <td>26</td>\n",
       "      <td>33</td>\n",
       "      <td>31</td>\n",
       "      <td>26</td>\n",
       "      <td>31</td>\n",
       "      <td>40</td>\n",
       "      <td>49</td>\n",
       "      <td>61</td>\n",
       "      <td>61</td>\n",
       "      <td>...</td>\n",
       "      <td>6</td>\n",
       "      <td>40</td>\n",
       "      <td>65</td>\n",
       "      <td>69</td>\n",
       "      <td>55</td>\n",
       "      <td>37</td>\n",
       "      <td>25</td>\n",
       "      <td>22</td>\n",
       "      <td>30</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11481</th>\n",
       "      <td>X8.V1.444</td>\n",
       "      <td>-86</td>\n",
       "      <td>-101</td>\n",
       "      <td>-115</td>\n",
       "      <td>-129</td>\n",
       "      <td>-139</td>\n",
       "      <td>-147</td>\n",
       "      <td>-154</td>\n",
       "      <td>-159</td>\n",
       "      <td>-165</td>\n",
       "      <td>...</td>\n",
       "      <td>-39</td>\n",
       "      <td>-43</td>\n",
       "      <td>-44</td>\n",
       "      <td>-45</td>\n",
       "      <td>-39</td>\n",
       "      <td>-38</td>\n",
       "      <td>-35</td>\n",
       "      <td>-34</td>\n",
       "      <td>-36</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11482</th>\n",
       "      <td>X8.V1.904</td>\n",
       "      <td>-121</td>\n",
       "      <td>4</td>\n",
       "      <td>104</td>\n",
       "      <td>185</td>\n",
       "      <td>269</td>\n",
       "      <td>352</td>\n",
       "      <td>405</td>\n",
       "      <td>421</td>\n",
       "      <td>392</td>\n",
       "      <td>...</td>\n",
       "      <td>339</td>\n",
       "      <td>331</td>\n",
       "      <td>315</td>\n",
       "      <td>298</td>\n",
       "      <td>282</td>\n",
       "      <td>253</td>\n",
       "      <td>203</td>\n",
       "      <td>145</td>\n",
       "      <td>108</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11483</th>\n",
       "      <td>X1.V1.902</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>16</td>\n",
       "      <td>10</td>\n",
       "      <td>17</td>\n",
       "      <td>23</td>\n",
       "      <td>42</td>\n",
       "      <td>31</td>\n",
       "      <td>34</td>\n",
       "      <td>...</td>\n",
       "      <td>190</td>\n",
       "      <td>152</td>\n",
       "      <td>132</td>\n",
       "      <td>104</td>\n",
       "      <td>56</td>\n",
       "      <td>41</td>\n",
       "      <td>63</td>\n",
       "      <td>126</td>\n",
       "      <td>193</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11484</th>\n",
       "      <td>X13.V1.583</td>\n",
       "      <td>-18</td>\n",
       "      <td>-12</td>\n",
       "      <td>-5</td>\n",
       "      <td>-4</td>\n",
       "      <td>-3</td>\n",
       "      <td>-9</td>\n",
       "      <td>-16</td>\n",
       "      <td>-27</td>\n",
       "      <td>-35</td>\n",
       "      <td>...</td>\n",
       "      <td>-63</td>\n",
       "      <td>-58</td>\n",
       "      <td>-51</td>\n",
       "      <td>-48</td>\n",
       "      <td>-41</td>\n",
       "      <td>-30</td>\n",
       "      <td>-26</td>\n",
       "      <td>-17</td>\n",
       "      <td>-17</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11485</th>\n",
       "      <td>X7.V1.81</td>\n",
       "      <td>-27</td>\n",
       "      <td>-8</td>\n",
       "      <td>-22</td>\n",
       "      <td>-48</td>\n",
       "      <td>-75</td>\n",
       "      <td>-99</td>\n",
       "      <td>-109</td>\n",
       "      <td>-105</td>\n",
       "      <td>-99</td>\n",
       "      <td>...</td>\n",
       "      <td>-43</td>\n",
       "      <td>-75</td>\n",
       "      <td>-70</td>\n",
       "      <td>-25</td>\n",
       "      <td>19</td>\n",
       "      <td>34</td>\n",
       "      <td>5</td>\n",
       "      <td>-27</td>\n",
       "      <td>-32</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11486</th>\n",
       "      <td>X15.V1.601</td>\n",
       "      <td>7</td>\n",
       "      <td>-28</td>\n",
       "      <td>-46</td>\n",
       "      <td>-45</td>\n",
       "      <td>-31</td>\n",
       "      <td>2</td>\n",
       "      <td>17</td>\n",
       "      <td>12</td>\n",
       "      <td>-2</td>\n",
       "      <td>...</td>\n",
       "      <td>39</td>\n",
       "      <td>72</td>\n",
       "      <td>97</td>\n",
       "      <td>111</td>\n",
       "      <td>77</td>\n",
       "      <td>1</td>\n",
       "      <td>-53</td>\n",
       "      <td>-80</td>\n",
       "      <td>-108</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11487</th>\n",
       "      <td>X17.V1.874</td>\n",
       "      <td>98</td>\n",
       "      <td>107</td>\n",
       "      <td>121</td>\n",
       "      <td>143</td>\n",
       "      <td>166</td>\n",
       "      <td>184</td>\n",
       "      <td>185</td>\n",
       "      <td>174</td>\n",
       "      <td>153</td>\n",
       "      <td>...</td>\n",
       "      <td>-31</td>\n",
       "      <td>11</td>\n",
       "      <td>48</td>\n",
       "      <td>77</td>\n",
       "      <td>95</td>\n",
       "      <td>107</td>\n",
       "      <td>111</td>\n",
       "      <td>110</td>\n",
       "      <td>119</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11488</th>\n",
       "      <td>X20.V1.494</td>\n",
       "      <td>267</td>\n",
       "      <td>409</td>\n",
       "      <td>450</td>\n",
       "      <td>416</td>\n",
       "      <td>334</td>\n",
       "      <td>248</td>\n",
       "      <td>173</td>\n",
       "      <td>113</td>\n",
       "      <td>67</td>\n",
       "      <td>...</td>\n",
       "      <td>413</td>\n",
       "      <td>288</td>\n",
       "      <td>173</td>\n",
       "      <td>76</td>\n",
       "      <td>7</td>\n",
       "      <td>-27</td>\n",
       "      <td>-46</td>\n",
       "      <td>-58</td>\n",
       "      <td>-67</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11489</th>\n",
       "      <td>X7.V1.51</td>\n",
       "      <td>90</td>\n",
       "      <td>83</td>\n",
       "      <td>69</td>\n",
       "      <td>41</td>\n",
       "      <td>31</td>\n",
       "      <td>40</td>\n",
       "      <td>58</td>\n",
       "      <td>75</td>\n",
       "      <td>82</td>\n",
       "      <td>...</td>\n",
       "      <td>83</td>\n",
       "      <td>77</td>\n",
       "      <td>52</td>\n",
       "      <td>19</td>\n",
       "      <td>-20</td>\n",
       "      <td>-33</td>\n",
       "      <td>-18</td>\n",
       "      <td>9</td>\n",
       "      <td>30</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11490</th>\n",
       "      <td>X14.V1.404</td>\n",
       "      <td>-22</td>\n",
       "      <td>-64</td>\n",
       "      <td>-121</td>\n",
       "      <td>-201</td>\n",
       "      <td>-292</td>\n",
       "      <td>-336</td>\n",
       "      <td>-398</td>\n",
       "      <td>-527</td>\n",
       "      <td>-773</td>\n",
       "      <td>...</td>\n",
       "      <td>183</td>\n",
       "      <td>-152</td>\n",
       "      <td>-499</td>\n",
       "      <td>-577</td>\n",
       "      <td>-415</td>\n",
       "      <td>-223</td>\n",
       "      <td>-88</td>\n",
       "      <td>-18</td>\n",
       "      <td>-61</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11491</th>\n",
       "      <td>X20.V1.883</td>\n",
       "      <td>102</td>\n",
       "      <td>137</td>\n",
       "      <td>158</td>\n",
       "      <td>158</td>\n",
       "      <td>128</td>\n",
       "      <td>92</td>\n",
       "      <td>60</td>\n",
       "      <td>56</td>\n",
       "      <td>63</td>\n",
       "      <td>...</td>\n",
       "      <td>76</td>\n",
       "      <td>64</td>\n",
       "      <td>59</td>\n",
       "      <td>44</td>\n",
       "      <td>39</td>\n",
       "      <td>23</td>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11492</th>\n",
       "      <td>X13.V1.624</td>\n",
       "      <td>-27</td>\n",
       "      <td>-53</td>\n",
       "      <td>-84</td>\n",
       "      <td>-104</td>\n",
       "      <td>-131</td>\n",
       "      <td>-171</td>\n",
       "      <td>-199</td>\n",
       "      <td>-203</td>\n",
       "      <td>-177</td>\n",
       "      <td>...</td>\n",
       "      <td>-127</td>\n",
       "      <td>-114</td>\n",
       "      <td>-99</td>\n",
       "      <td>-79</td>\n",
       "      <td>-56</td>\n",
       "      <td>-28</td>\n",
       "      <td>-14</td>\n",
       "      <td>-8</td>\n",
       "      <td>-12</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11493</th>\n",
       "      <td>X23.V1.714</td>\n",
       "      <td>-136</td>\n",
       "      <td>-137</td>\n",
       "      <td>-138</td>\n",
       "      <td>-135</td>\n",
       "      <td>-130</td>\n",
       "      <td>-123</td>\n",
       "      <td>-116</td>\n",
       "      <td>-118</td>\n",
       "      <td>-119</td>\n",
       "      <td>...</td>\n",
       "      <td>-1</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "      <td>13</td>\n",
       "      <td>16</td>\n",
       "      <td>24</td>\n",
       "      <td>34</td>\n",
       "      <td>42</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11494</th>\n",
       "      <td>X21.V1.762</td>\n",
       "      <td>-39</td>\n",
       "      <td>-16</td>\n",
       "      <td>-13</td>\n",
       "      <td>-14</td>\n",
       "      <td>-68</td>\n",
       "      <td>-84</td>\n",
       "      <td>-92</td>\n",
       "      <td>-61</td>\n",
       "      <td>-89</td>\n",
       "      <td>...</td>\n",
       "      <td>-102</td>\n",
       "      <td>-75</td>\n",
       "      <td>-63</td>\n",
       "      <td>-44</td>\n",
       "      <td>-66</td>\n",
       "      <td>-49</td>\n",
       "      <td>-34</td>\n",
       "      <td>-31</td>\n",
       "      <td>-52</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11495</th>\n",
       "      <td>X22.V1.114</td>\n",
       "      <td>-22</td>\n",
       "      <td>-22</td>\n",
       "      <td>-23</td>\n",
       "      <td>-26</td>\n",
       "      <td>-36</td>\n",
       "      <td>-42</td>\n",
       "      <td>-45</td>\n",
       "      <td>-42</td>\n",
       "      <td>-45</td>\n",
       "      <td>...</td>\n",
       "      <td>15</td>\n",
       "      <td>16</td>\n",
       "      <td>12</td>\n",
       "      <td>5</td>\n",
       "      <td>-1</td>\n",
       "      <td>-18</td>\n",
       "      <td>-37</td>\n",
       "      <td>-47</td>\n",
       "      <td>-48</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11496</th>\n",
       "      <td>X19.V1.354</td>\n",
       "      <td>-47</td>\n",
       "      <td>-11</td>\n",
       "      <td>28</td>\n",
       "      <td>77</td>\n",
       "      <td>141</td>\n",
       "      <td>211</td>\n",
       "      <td>246</td>\n",
       "      <td>240</td>\n",
       "      <td>193</td>\n",
       "      <td>...</td>\n",
       "      <td>-65</td>\n",
       "      <td>-33</td>\n",
       "      <td>-7</td>\n",
       "      <td>14</td>\n",
       "      <td>27</td>\n",
       "      <td>48</td>\n",
       "      <td>77</td>\n",
       "      <td>117</td>\n",
       "      <td>170</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11497</th>\n",
       "      <td>X8.V1.28</td>\n",
       "      <td>14</td>\n",
       "      <td>6</td>\n",
       "      <td>-13</td>\n",
       "      <td>-16</td>\n",
       "      <td>10</td>\n",
       "      <td>26</td>\n",
       "      <td>27</td>\n",
       "      <td>-9</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>-65</td>\n",
       "      <td>-48</td>\n",
       "      <td>-61</td>\n",
       "      <td>-62</td>\n",
       "      <td>-67</td>\n",
       "      <td>-30</td>\n",
       "      <td>-2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-8</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11498</th>\n",
       "      <td>X10.V1.932</td>\n",
       "      <td>-40</td>\n",
       "      <td>-25</td>\n",
       "      <td>-9</td>\n",
       "      <td>-12</td>\n",
       "      <td>-2</td>\n",
       "      <td>12</td>\n",
       "      <td>7</td>\n",
       "      <td>19</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>121</td>\n",
       "      <td>135</td>\n",
       "      <td>148</td>\n",
       "      <td>143</td>\n",
       "      <td>116</td>\n",
       "      <td>86</td>\n",
       "      <td>68</td>\n",
       "      <td>59</td>\n",
       "      <td>55</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11499</th>\n",
       "      <td>X16.V1.210</td>\n",
       "      <td>29</td>\n",
       "      <td>41</td>\n",
       "      <td>57</td>\n",
       "      <td>72</td>\n",
       "      <td>74</td>\n",
       "      <td>62</td>\n",
       "      <td>54</td>\n",
       "      <td>43</td>\n",
       "      <td>31</td>\n",
       "      <td>...</td>\n",
       "      <td>-59</td>\n",
       "      <td>-25</td>\n",
       "      <td>-4</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>-2</td>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11500 rows × 180 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0   X1   X2   X3   X4   X5   X6   X7   X8   X9 ...  X170  \\\n",
       "0      X21.V1.791  135  190  229  223  192  125   55   -9  -33 ...   -17   \n",
       "1      X15.V1.924  386  382  356  331  320  315  307  272  244 ...   164   \n",
       "2         X8.V1.1  -32  -39  -47  -37  -32  -36  -57  -73  -85 ...    57   \n",
       "3       X16.V1.60 -105 -101  -96  -92  -89  -95 -102 -100  -87 ...   -82   \n",
       "4       X20.V1.54   -9  -65  -98 -102  -78  -48  -16    0  -21 ...     4   \n",
       "5       X14.V1.56   55   28   18   16   16   19   25   40   52 ...   -12   \n",
       "6       X3.V1.191  -55   -9   52  111  135  129  103   72   37 ...  -125   \n",
       "7      X11.V1.273    1   -2   -8  -11  -12  -17  -15  -16  -18 ...   -79   \n",
       "8      X19.V1.874 -278 -246 -215 -191 -177 -167 -157 -139 -118 ...  -400   \n",
       "9       X3.V1.491    8   15   13    3   -6   -8   -5    4   25 ...    49   \n",
       "10        X3.V1.6   -5   15   28   28    9  -29  -41  -19   14 ...   -38   \n",
       "11     X21.V1.724 -167 -230 -280 -315 -338 -369 -405 -392 -298 ...   423   \n",
       "12      X7.V1.162   92   49    0  -32  -51  -65  -37  -19  -25 ...   -56   \n",
       "13      X1.V1.211   15   12    0  -17  -28  -31  -39  -51  -44 ...   -88   \n",
       "14      X1.V1.615  -24  -15   -5   -1    4    3    6   10   11 ...    32   \n",
       "15     X22.V1.242 -135 -133 -125 -118 -111 -105 -102  -93  -94 ...   -49   \n",
       "16      X1.V1.863   39   41   41   42   43   43   46   47   49 ...    43   \n",
       "17      X9.V1.302    9    4   -5  -10  -22  -30  -33  -43  -41 ...    34   \n",
       "18      X7.V1.541  -21   -5    1    7   19   20   13    2   -1 ...    43   \n",
       "19      X9.V1.915    4   24   51   76   92  102  104  101   90 ...     3   \n",
       "20     X23.V1.964  410  451  491  541  581  641  736  757  692 ...   415   \n",
       "21      X1.V1.614  -24  -27  -23  -28  -34  -40  -47  -43  -38 ...   -81   \n",
       "22     X11.V1.134 -264 -189 -117  -45   20   70  111  143  161 ...  -231   \n",
       "23     X18.V1.544   -4   40   78  123  149  185  197  189  167 ...  -189   \n",
       "24     X19.V1.294  593  328   88 -106 -456 -732 -921 -782 -522 ...   312   \n",
       "25     X21.V1.802  -16  -15  -19  -16  -14   -5    0   -1   -3 ...    56   \n",
       "26       X2.V1.72  -20  -38  -53  -58  -66  -66  -69  -77  -87 ...   -26   \n",
       "27      X3.V1.744 -340 -381 -376 -336 -275 -204 -131  -70  -16 ...   114   \n",
       "28     X12.V1.733  -30   15   61   80   72   41  -11  -31  -47 ...    -9   \n",
       "29      X8.V1.614   -1   18   35   36   29   17   10    1   -3 ...    -6   \n",
       "...           ...  ...  ...  ...  ...  ...  ...  ...  ...  ... ...   ...   \n",
       "11470  X17.V1.232  -62  -33   13   51   89  111  114  124  120 ...    34   \n",
       "11471  X21.V1.334 -214 -320 -418 -534 -748 -638 -470   42  454 ...  1033   \n",
       "11472   X5.V1.332  -61  -71  -77  -74  -74  -72  -72  -76  -77 ...   -51   \n",
       "11473   X4.V1.173 -106 -133 -130 -111  -91  -62  -39    4   42 ...   -80   \n",
       "11474    X8.V1.88  -50  -48  -63  -62  -46  -24   -1   13   27 ...    44   \n",
       "11475   X7.V1.462  -68  -43  -24   -3  -15  -16  -15  -17  -19 ...    87   \n",
       "11476  X22.V1.984 -293 -297 -302 -307 -318 -331 -347 -361 -369 ...    12   \n",
       "11477  X10.V1.421   28   36   26    4  -35  -52  -51  -39  -12 ...     4   \n",
       "11478   X3.V1.554  452  470  401  291  163   83   20  -32  -61 ...   182   \n",
       "11479  X16.V1.353   62   68   75   75   75   77   81   85   82 ...    -1   \n",
       "11480   X13.V1.26   26   33   31   26   31   40   49   61   61 ...     6   \n",
       "11481   X8.V1.444  -86 -101 -115 -129 -139 -147 -154 -159 -165 ...   -39   \n",
       "11482   X8.V1.904 -121    4  104  185  269  352  405  421  392 ...   339   \n",
       "11483   X1.V1.902    7   30   16   10   17   23   42   31   34 ...   190   \n",
       "11484  X13.V1.583  -18  -12   -5   -4   -3   -9  -16  -27  -35 ...   -63   \n",
       "11485    X7.V1.81  -27   -8  -22  -48  -75  -99 -109 -105  -99 ...   -43   \n",
       "11486  X15.V1.601    7  -28  -46  -45  -31    2   17   12   -2 ...    39   \n",
       "11487  X17.V1.874   98  107  121  143  166  184  185  174  153 ...   -31   \n",
       "11488  X20.V1.494  267  409  450  416  334  248  173  113   67 ...   413   \n",
       "11489    X7.V1.51   90   83   69   41   31   40   58   75   82 ...    83   \n",
       "11490  X14.V1.404  -22  -64 -121 -201 -292 -336 -398 -527 -773 ...   183   \n",
       "11491  X20.V1.883  102  137  158  158  128   92   60   56   63 ...    76   \n",
       "11492  X13.V1.624  -27  -53  -84 -104 -131 -171 -199 -203 -177 ...  -127   \n",
       "11493  X23.V1.714 -136 -137 -138 -135 -130 -123 -116 -118 -119 ...    -1   \n",
       "11494  X21.V1.762  -39  -16  -13  -14  -68  -84  -92  -61  -89 ...  -102   \n",
       "11495  X22.V1.114  -22  -22  -23  -26  -36  -42  -45  -42  -45 ...    15   \n",
       "11496  X19.V1.354  -47  -11   28   77  141  211  246  240  193 ...   -65   \n",
       "11497    X8.V1.28   14    6  -13  -16   10   26   27   -9    4 ...   -65   \n",
       "11498  X10.V1.932  -40  -25   -9  -12   -2   12    7   19   22 ...   121   \n",
       "11499  X16.V1.210   29   41   57   72   74   62   54   43   31 ...   -59   \n",
       "\n",
       "       X171  X172  X173  X174  X175  X176  X177  X178  y  \n",
       "0       -15   -31   -77  -103  -127  -116   -83   -51  4  \n",
       "1       150   146   152   157   156   154   143   129  1  \n",
       "2        64    48    19   -12   -30   -35   -35   -36  5  \n",
       "3       -81   -80   -77   -85   -77   -72   -69   -65  5  \n",
       "4         2   -12   -32   -41   -65   -83   -89   -73  5  \n",
       "5       -31   -42   -54   -60   -64   -60   -56   -55  5  \n",
       "6       -99   -79   -62   -41   -26    11    67   128  4  \n",
       "7       -91   -97   -88   -76   -72   -66   -57   -39  2  \n",
       "8      -379  -336  -281  -226  -174  -125   -79   -40  1  \n",
       "9        31    11    -5   -17   -19   -15   -15   -11  4  \n",
       "10       -4    25    16   -16   -74  -101   -89   -49  5  \n",
       "11      434   416   374   319   268   215   165   103  1  \n",
       "12      -41   -40   -43   -32   -13    -1    -7   -44  3  \n",
       "13     -102   -97   -77   -45   -19    13    44    68  4  \n",
       "14       35    36    34    32    26    23    18    20  2  \n",
       "15      -39   -35   -29   -10     4    21    31    37  3  \n",
       "16       41    41    43    43    40    41    41    49  2  \n",
       "17       27    22    18    15    13     9     9     3  3  \n",
       "18       28    25    19    30    35    26     5   -13  4  \n",
       "19        5    10    19    31    36    40    43    36  2  \n",
       "20      428   463   510   562   607   667   748   763  1  \n",
       "21      -51   -38   -11   -12     4     5    -4    -3  2  \n",
       "22     -221  -248  -321  -444  -530  -548  -536  -486  1  \n",
       "23     -214  -226  -224  -203  -171  -129   -85   -40  1  \n",
       "24      360   421   445   413   310   177    41   -71  1  \n",
       "25       56    62    54    53    53    50    57    54  3  \n",
       "26      -45   -55   -55   -46   -45   -43   -42   -42  5  \n",
       "27      -39  -185  -293  -351  -379  -380  -350  -308  1  \n",
       "28      -21   -17    -4     5     7    -6   -17   -16  2  \n",
       "29      -18   -13   -18   -22   -19   -18    -6    -4  2  \n",
       "...     ...   ...   ...   ...   ...   ...   ...   ... ..  \n",
       "11470    28    39    58    74    77    76    46    25  3  \n",
       "11471   858   597   370   206  -122   -71   -89   172  1  \n",
       "11472   -58   -68   -88   -98  -101  -102   -99   -96  3  \n",
       "11473  -132  -151  -166  -136  -110   -90   -63   -52  2  \n",
       "11474    43    46    47    46    32    13     6     2  5  \n",
       "11475    88    90    91    96    98   102   100    91  3  \n",
       "11476    -6   314   497   826   877   778   571   274  1  \n",
       "11477    23    29    18    -8   -32   -42   -42   -37  4  \n",
       "11478   237   263   211   114    48     8   -15   -48  1  \n",
       "11479    -2    -1     9    18    22    30    36    41  2  \n",
       "11480    40    65    69    55    37    25    22    30  5  \n",
       "11481   -43   -44   -45   -39   -38   -35   -34   -36  1  \n",
       "11482   331   315   298   282   253   203   145   108  1  \n",
       "11483   152   132   104    56    41    63   126   193  3  \n",
       "11484   -58   -51   -48   -41   -30   -26   -17   -17  2  \n",
       "11485   -75   -70   -25    19    34     5   -27   -32  5  \n",
       "11486    72    97   111    77     1   -53   -80  -108  4  \n",
       "11487    11    48    77    95   107   111   110   119  1  \n",
       "11488   288   173    76     7   -27   -46   -58   -67  1  \n",
       "11489    77    52    19   -20   -33   -18     9    30  5  \n",
       "11490  -152  -499  -577  -415  -223   -88   -18   -61  1  \n",
       "11491    64    59    44    39    23    11     4     2  2  \n",
       "11492  -114   -99   -79   -56   -28   -14    -8   -12  1  \n",
       "11493     3     5     8    13    16    24    34    42  2  \n",
       "11494   -75   -63   -44   -66   -49   -34   -31   -52  3  \n",
       "11495    16    12     5    -1   -18   -37   -47   -48  2  \n",
       "11496   -33    -7    14    27    48    77   117   170  1  \n",
       "11497   -48   -61   -62   -67   -30    -2    -1    -8  5  \n",
       "11498   135   148   143   116    86    68    59    55  3  \n",
       "11499   -25    -4     2     5     4    -2     2    20  4  \n",
       "\n",
       "[11500 rows x 180 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('data.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it is decribed in their web page, each row is a subject or a patient record randomly extracted for 1 min.\n",
    "We will shift each `y` value, because they are ranged from 1 to 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 1, 5, ..., 5, 3, 4])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = df['y'].values\n",
    "labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3, 0, 4, ..., 4, 2, 3])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = labels - 1\n",
    "labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11500,)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract signal data also."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 135,  190,  229, ..., -116,  -83,  -51],\n",
       "       [ 386,  382,  356, ...,  154,  143,  129],\n",
       "       [ -32,  -39,  -47, ...,  -35,  -35,  -36],\n",
       "       ...,\n",
       "       [  14,    6,  -13, ...,   -2,   -1,   -8],\n",
       "       [ -40,  -25,   -9, ...,   68,   59,   55],\n",
       "       [  29,   41,   57, ...,   -2,    2,   20]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = df.loc[:, 'X1':'X178'].values\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11500, 178)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we will split the whole dataset into three subsets: train, validation, and test set. In practice, the amount of test set is chosen within 15~20% and the amount of validation set is chosen within 10~15%, but it is not limited unless they are too non-sense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7360, 178) (1840, 178) (2300, 178)\n",
      "11500\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(data, labels, test_size=0.2, random_state=6250)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.2, random_state=6250)\n",
    "print(X_train.shape, X_valid.shape, X_test.shape)\n",
    "print(X_train.shape[0] + X_valid.shape[0] + X_test.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Loading Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "\n",
    "# lets fix the random seeds for reproducibility.\n",
    "torch.manual_seed(6250)\n",
    "if torch.cuda.is_available():\n",
    "    torch.cuda.manual_seed(6250)\n",
    "\n",
    "trainset = TensorDataset(torch.from_numpy(X_train.astype('float32')).unsqueeze(1), torch.from_numpy(y_train.astype('long')))\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=10, shuffle=True, num_workers=2)\n",
    "\n",
    "validset = TensorDataset(torch.from_numpy(X_valid.astype('float32')).unsqueeze(1), torch.from_numpy(y_valid.astype('long')))\n",
    "validloader = torch.utils.data.DataLoader(validset, batch_size=10, shuffle=False, num_workers=2)\n",
    "\n",
    "testset = TensorDataset(torch.from_numpy(X_test.astype('float32')).unsqueeze(1), torch.from_numpy(y_test.astype('long')))\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=10, shuffle=False, num_workers=2)\n",
    "\n",
    "classes = ('Seizure', 'TumorArea', 'HealthyArea', 'EyesClosed', 'EyesOpen')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, let's check a few data points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[  88.,   79.,   94.,  ...,   21.,   29.,   16.]],\n",
      "\n",
      "        [[  82.,   18., -107.,  ...,  -49., -150., -250.]],\n",
      "\n",
      "        [[  75.,   67.,   58.,  ...,  -29.,  -30.,  -32.]],\n",
      "\n",
      "        ...,\n",
      "\n",
      "        [[   4.,    9.,   23.,  ...,   11.,    6.,    8.]],\n",
      "\n",
      "        [[ 400.,  415.,  433.,  ...,  -73.,  -79.,  -83.]],\n",
      "\n",
      "        [[  27.,   25.,    4.,  ...,   22.,   46.,   54.]]])\n",
      "tensor([1, 0, 2, 0, 1, 0, 4, 2, 0, 4])\n"
     ]
    }
   ],
   "source": [
    "# get some random training samples\n",
    "dataiter = iter(trainloader)\n",
    "X_samples, y_samples = dataiter.next()\n",
    "\n",
    "print(X_samples)\n",
    "print(y_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Define CNN class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will define a model similar to what we did in the previous chapter.\n",
    "\n",
    "A key module to construct a convolutional net is `nn.ConvXd`. Although we use 1-dimensional convolution layer for this tutorial, parameters are almost same for all X-dimensional convolution layers. You need to specify the number of input channels (`in_channels`), the number of output channels (`out_channels`), the kernel (filter) size (`kernel_size`), stride size (`stride`), zero-padding (`padding`), and so forth. If you are not familar with these terms, it is highly recommended to read introductory materials about CNN.\n",
    "\n",
    "We will use 2 convolution layers with max pooling and they are followed by 3 fully-connected layers with ReLU activation. We do not apply any activation at the output layer to use activation-combined loss function for a better numerical stability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.autograd import Variable\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class ConvNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(ConvNet, self).__init__()\n",
    "        self.conv1 = nn.Conv1d(in_channels=1, out_channels=6, kernel_size=5)\n",
    "        self.pool = nn.MaxPool1d(kernel_size=2)\n",
    "        self.conv2 = nn.Conv1d(6, 16, 5)\n",
    "        self.fc1 = nn.Linear(in_features=16 * 41, out_features=128)\n",
    "        self.fc2 = nn.Linear(128, 64)\n",
    "        self.fc3 = nn.Linear(64, 5)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.pool(F.relu(self.conv1(x)))\n",
    "        x = self.pool(F.relu(self.conv2(x)))\n",
    "        x = x.view(-1, 16 * 41)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "model = ConvNet()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, check some sample outputs to see if the model works (here 'work' means whether it gives output without error or not).\n",
    "It outputs 5-dim vector as we expected for 10 input data as we passed, which seem correct. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  1.0865,  -0.5738,   0.0271,   0.3288,   0.5611],\n",
       "        [ -3.7110,  -4.9135,  -1.6561,   5.0656,   8.6459],\n",
       "        [ -0.6619,  -1.2821,  -0.1897,   1.3865,   0.9626],\n",
       "        [ -7.8289, -17.4362,  -1.5873,   5.2889,   7.5970],\n",
       "        [ -1.6765,  -1.7986,  -0.3355,  -0.7259,   0.8651],\n",
       "        [ -7.5507,  -5.3690,   3.5921,  10.2904,  10.5336],\n",
       "        [ -1.1641,  -0.5070,  -0.2023,   0.4206,   0.5371],\n",
       "        [  1.0538,   0.0700,   1.4677,   1.5054,   0.7855],\n",
       "        [ -8.2407,  -7.0444,  -2.4038,   2.8656,   5.4439],\n",
       "        [  0.1938,  -0.8643,   0.3121,  -0.0725,   0.1098]],\n",
       "       grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = model(X_samples)\n",
    "output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Define a Loss function and optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `CrossEntropyLoss` which combines LogSoftmax and Negative Log-likelihood loss function. **This is the reason why we did not apply activation function such as softmax after the output layer.**\n",
    "\n",
    "On the other hand, we will use ADAM optimizer with its default parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Train the network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we train the model, check first if GPU is available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cuda = torch.cuda.is_available()\n",
    "cuda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If GPU is available, we can move the model (all its parameter Tensors) onto GPU by simply calling `MODEL = MODEL.cuda()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "if cuda:\n",
    "    model = model.cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we decided to use GPU and moved the model above, we also need to put all the data Tensors used in training epochs onto GPU. It can be done also in the same manner, simply calling `TENSOR.cuda()`.\n",
    "\n",
    "The following training epochs are very similar to what we have seen in the previous chapter, especially from line 4 to 24. Here, we add a few more code lines to see the model performance on the validation set at each epoch. It is almost same with the training part: getting data from Loader (and move them onto GPU if needed) and passing them through the forward computation. We do not need (MUST NOT) to do backward computations (back-propagations) on the validation set.\n",
    "\n",
    "`MODEL.train()` and `MODEL.eval()` are functions acting as a 'switch' which turns on/off any randomness of layers in the model. For example, if we use dropout in the model, we do not in this tutorial though (it is related with an exercise below), it means any connection between layers will be disconnected according to the keeping probability we set. However, this should be 'turned on' ONLY in the training process and should be 'turned off' on evaluation (inference) mode.\n",
    "\n",
    "We also use a couple of list to track train/validation losses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished Training\n"
     ]
    }
   ],
   "source": [
    "train_losses = []\n",
    "valid_losses = []\n",
    "\n",
    "for epoch in range(10):  # loop over the dataset multiple times\n",
    "\n",
    "    # set the model as train mode\n",
    "    model.train()\n",
    "    train_loss = 0.0\n",
    "    train_counter = 0\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        # get the inputs\n",
    "        inputs, targets = data\n",
    "\n",
    "        if cuda:\n",
    "            inputs, targets = inputs.cuda(), targets.cuda()\n",
    "\n",
    "        # zero the parameter gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # forward + backward + optimize\n",
    "        outputs = model(inputs)\n",
    "        loss = criterion(outputs, targets)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        train_loss += (loss.item() * inputs.size(0))\n",
    "        train_counter += inputs.size(0)\n",
    "\n",
    "    train_losses.append(train_loss/train_counter)\n",
    "    \n",
    "    # switch to evaluation mode\n",
    "    model.eval()\n",
    "    valid_loss = 0.0\n",
    "    valid_counter = 0\n",
    "    with torch.no_grad():\n",
    "        for i, data in enumerate(validloader, 0):\n",
    "            # get the inputs\n",
    "            inputs, targets = data\n",
    "\n",
    "            if cuda:\n",
    "                inputs, targets = inputs.cuda(), targets.cuda()\n",
    "\n",
    "            outputs = model(inputs)\n",
    "            loss = criterion(outputs, targets)\n",
    "\n",
    "            valid_loss += (loss.item() * inputs.size(0))\n",
    "            valid_counter += inputs.size(0)\n",
    "    \n",
    "    valid_losses.append(valid_loss/valid_counter)\n",
    "    \n",
    "print('Finished Training')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot train/validation losses we tracked over the epochs.\n",
    "For your interest, in Jupyter notebook, we can easily make a plot with `matplotlib` by calling a magic function `%matplotlib inline`, or `%matplotlib notebook` for interative plotting (we can also use `matplotlib.pyplot.ion` and `ioff` for Jupyter (IPython) >= 5.0 and matplotlib >= 2.0).\n",
    "\n",
    "### Exercise: online plot?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f7524596208>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd8VvX5//HXlU02kEAgg4Q9QoAQ9hYHLlBEhuBAlIq1aK22aG2t9mvLr1pLrVZFARcyRBBEECuiDBlJENnISEJCgIQdyE4+vz/ObQwQdu6cJPf1fDx4kPvc5z7nyg3J+z7ns8QYg1JKKQXgZncBSimlqg8NBaWUUmU0FJRSSpXRUFBKKVVGQ0EppVQZDQWllFJlNBSUUkqV0VBQSilVRkNBKaVUGQ+7C7hSISEhJjo62u4ylFKqRklOTj5ijAm91H41LhSio6NJSkqyuwyllKpRRCTtcvbT20dKKaXKaCgopZQqo6GglFKqjIaCUkqpMhoKSimlymgoKKWUKqOhoJRSqozTQkFEpotIlohsvcDzo0Vks+PP9yLSwVm1AOzNPs0Ln2+jqKTUmadRSqkazZlXCu8Bgy7yfArQzxgTB/wVmOrEWth/NJcZa1JZsuWgM0+jlFI1mtNCwRizEjh2kee/N8YcdzxcB0Q4qxaAfi1DaRrqx7TVKRhjnHkqpZSqsapLm8I4YOmFnhSR8SKSJCJJ2dnZV3UCNzfhwV4xbM44SVLa8Uu/QCmlXJDtoSAiA7BC4Q8X2scYM9UYk2CMSQgNveR8Thd0V3wEwb6evLtq31UfQymlajNbQ0FE4oB3gSHGmKPOPl8dL3dGd4viq+2H2X8019mnU0qpGse2UBCRKGA+cK8x5qeqOu99PaLxcBNmfJ9SVadUSqkaw5ldUmcBa4FWIpIhIuNE5BERecSxy5+B+sB/RWSTiFTJfNgNA324La4xcxPTOZVfVBWnVEqpGsNp6ykYY0Zd4vmHgIecdf6LGdc7hgU/HGBuYjoP9WlqRwlKKVUt2d7QbIfY8CC6xtRjxppUinUwm1JKlXHJUAB4qHcMB07ksWzbYbtLUUqpasNlQ2Fgm4Y0qe/LtNXaPVUppX7msqHg7iaM7RnNxv0n+GG/DmZTSilw4VAAuDshkgAfD6at1u6pSikFLh4Kft4ejOoaxdKthzhwIs/ucpRSynYuHQoA9/eMBuD971NtrUMppaoDlw+F8OA63BwbxqwN+zlTUGx3OUopZSuXDwWwBrPl5BfzSVK63aUopZStNBSATlF1iY8KZvqaVEpKda0FpZTr0lBweKhPU/Yfy+XrHTqYTSnlujQUHG5s25Dw4DraPVUp5dI0FBw83N0Y2yuaDSnH2HrgpN3lKKWULTQUyhneJRI/L3e9WlBKuSwNhXICfTwZ3iWSz3/M5NDJfLvLUUqpKqehcI6xPWMoNYYP1qbaXYpSSlU5DYVzRNX35ca2YXy8YT95hSV2l6OUUpZj+6DE+QNsNRQqMK5PDCdyi/h0Y4bdpSilFGRugneug6+ec/qpNBQqkNCkLnERQUxfk0KpDmZTStkpIxk+GAxe/tBtvNNPp6FQARFhXO8Y9mWf4dufsuwuRynlqvavhw+GgE8wjF0C9Zy/pryGwgXc0r4RjYJ8tHuqUsoeqWvgwzvBvwGMXQrBUVVyWg2FC/B0d+O+HtGs2XOUHQdP2V2OUsqV7PsWProLgsKtK4Sg8Co7tYbCRdzTNYo6nu5M16sFpVRV2fM1fDwC6sXAA19AQFiVnl5D4SKCfD0Z1jmChZsyyc4psLscpVRtt+tLmDUKQlrA/YutW0dVTEPhEsb2iqaotJQP16XZXYpSqjbb8TnMGQMN2sJ9i8Cvvi1laChcQtNQfwa2bsDMdWnkF+lgNqWUE2ydD3Pvh8Yd4b6F4FvPtlI0FC7Dg71jOHqmkIWbDthdilKqttk8Fz4dB5FdYcx8qBNsazkaCpehR9P6tGkUyLTVKRijg9mUUpXkh5kwfzw06QWj54FPoN0VOS8URGS6iGSJyNYLPC8i8pqI7BGRzSIS76xartXPg9l+OnyaVbuP2F2OUqo2SJoBCx+Fpv3gnrng7W93RYBzrxTeAwZd5PmbgRaOP+OBN51Yi+XQlqt+6e0dGhEa4K2D2ZRS127DO7D4CWh+A4yaA16+dldUxmmhYIxZCRy7yC5DgA+MZR0QLCKNnFUPGz+Et/rArqVX9XJvD3fu696E737KZk9WTiUXp5RyGWvfgCVPQatbYORM8PSxu6Kz2NmmEA6kl3uc4dh2HhEZLyJJIpKUnZ19dWeLHQqNOsC8cXDwx6s6xOjuTfD2cGPa6tSrq0Ep5dpW/wuWPQttBsPd74OHt90VncfOUJAKtlXYimuMmWqMSTDGJISGhl7d2bz84J45Vsv+xyPhVOYVH6KenxdD48OZvzGDY2cKr64OpZRr+u4f8PVfIPYuGDYDPLzsrqhCdoZCBhBZ7nEEcOW/qa9EQJgVDAWnrGHkBaev+BAP9oqhoLiUmTqYTSl1OYyBb/4PVrwEcSNh6Dvg7mF3VRdkZygsAu5z9ELqDpw0xhx0+lnD2sOw6XB4K8x/GEqvbEBai4YB9GsZygfr0igo1sFsSqmLMMa6Olj5MnS6F+74L7i5213VRTmzS+osYC3QSkQyRGSciDwiIo84dlkC7AP2AO8AjzqrlvO0vAkGTYZdS+B/f77il4/rHUN2TgGLf3R+himlaihjrPaDNVMg4UG4/bVqHwgATruGMcaMusTzBvi1s85/Sd1+BUf3wNrXoX4z6x/tMvVpEULLhv5MW53C0PhwRCpqHlFKuazSUlj6NCS+C90esT6E1pDfE649ovmmv1v9hL94CvZ+c9kvExEe7BXD9oOnWLfvYr1ulVIup7TUGoOQ+C70/E2NCgRw9VBw97DaF0JbW5NRZe287Jfe0Smc+n5eTFu9z4kFKqVqlNISWPQYbHwf+vwObvhrjQoEcPVQAGuukXvmgIcPfHw3nL68cRA+nu6M7t6E5TuzSDlyxslFKqWqvZJiWPAr2DQT+j8D1/2pxgUCaChYgiPhntlWIMweBUV5l/Wye7s3wdPNjRlrdOoLpVxaSZE10+mWT6ww6D+pRgYCaCj8IrwzDH0bMhLhs0et+4KXEBrgzeCOjfkkKYOTuUVVUKRSqtopLoRPHoDtn8GN/wd9n7K7omuioVBe2yFw/V9g23z49u+X9ZJxvWPIKyphVuJ+p5amlKqGivKt1dJ2Loab/2E1LNdwGgrn6vWENchk5T9g06xL7t6mUSC9mtfnvTWpFJVc+upCKVVLFOVZt5t3L4NbX7W6udcCGgrnErH+gWP6wqLfQOqaS75kXO8YDp3KZ8kWHcymlEsoPAMfD4e9K2Dw69BlnN0VVRoNhYp4eMHwD6BuNMwZDUf3XnT3/i0b0DTUj+m6MptStV9BDnw0DFJXw51vQ/y9dldUqTQULqROXRg9FxCYeTfkXniQmpubMLZXDD9mnCQ57XjV1aiUqlr5J+HDoZC+3prYrsMIuyuqdBoKF1OvKYz8GE6mw5x7rV4GF3BXfDjBvp68u0q7pypVK+Udhw/ugMyNcPcMaD/M7oqcQkPhUpr0gCFvQNpq+Pxxa5KrCvh6eXBP1yi+2n6I9GO5VVykUsqpco/B+4Ot2ZWHf2j1VKylNBQuR9xw6DcJfvwYVr96wd3u6xGNmwgz1qRWXW1KKec6nQ3v3QbZu6w7B61vsbsip9JQuFz9J0HsMFj+ImxbUOEuYUE+3BbXiLlJ6eTk62A2pWq0gtPw/X/grV5wbJ81HU6LG+yuyuk0FC6XiHUbKbIbLHgEMpIq3G1c76acLihmTmJ6hc8rpaq5vOPw7f+DKbHw1XMQ2goe+AKaDbC7siqhoXAlPH2sy0f/hjBrJBw/f0nO9hFBdI2px4w1qRTrYDalao7TWdaiW/+KhW//Zn0AHPc13P85RHS2u7oqo6FwpfxCYPQnVk+kj0dYXdTOMa53DAdO5PHV9sM2FKiUuiIn0mHJ0zClPax5DVrcCI+stm4XRXaxu7oqp6FwNUJbwYgP4Ohu+GSsNWVuOde3aUhUPV+mrdbuqUpVW0f2wGe/htc6QtJ0q4vpY0lWd9Ow9nZXZxsNhavVtD/c+k/YuxyW/v6srqrubsLYXtEkpx1nU/oJ20pUSlXg0BZrVtPXE2DrPEgYBxM3WW2GIc3trs52GgrXovMD0HMiJE2D9W+d9dTdCZEE+Hjo1YJS1UX6Bpg5HN7qDbu/hl6PwxNb4JZ/WGuqKAA87C6gxrv+BTieAl8+Y82V1OpmAPy9PRjVNYppq1OYdHNrwoPr2FunUq7IGEj5Dla+AqmrrOlrBvwRuj5sfa3Oo1cK18rNDe6cCo07wrxxcHBz2VP394wG4IPvU+2pTSlXVVoKO5fAuwPhgyFwZDfc+BI8sRX6/V4D4SI0FCqDly+Mmg11gq0eSaesKbTDg+swKDaMjzfs50xB8SUOopS6ZqUlsGWeNeBs9ig4cwRu+xc8/iP0fAy8/e2usNrTUKgsAWFWF7aCUzBrhDXfOlb31Jz8YuYlZ9hcoFK1WHEhJL9vNR5/Os4Khzunwm82QsKD1hgjdVk0FCpTWHsYNt3q3fDpQ1BaQnxUXeKjgpmxJoWSUl1rQalKVZgL6960upV+PhG8A60J6x5dZ01r7a7NpldKQ6GytbwJBk2GXUus0ZFYU1+kHs1l+Q4dzKZUpcg/Cav+aQ04+3ISBDeBMZ/C+G+h7WCrrU9dFY1RZ+j2Kzi6B9a+DvWbcVOnBwgPrsO01Snc2C7M7uqUqrnOHIV1/4UN70DBSWh+PfT5HTTpaXdltYaGgrPc9Hc4lgJfPIVH3Wge6BnNS0t2sPXASWLDg+yuTqma5VSmNWNp8ntQlAdtboc+T0LjTnZXVus4NRREZBDwb8AdeNcYM/mc56OA94Fgxz6TjDFLnFlTlXH3sNoXpg+Cufcz6t6lTPFyZ/rqFF4d0dHu6pS6sJzDkPmD9bWbu/VH3MHNo9zXFW13O2cfj3P2c2wT98u/vXNsH6z5N2z62Go8jhsOvX9rTTWjnEKctdC8iLgDPwE3ABlAIjDKGLO93D5TgR+MMW+KSFtgiTEm+mLHTUhIMElJFU9bXS2dSId3rgNPH16O+i9vJ+WwZtJ1NAzU3hCqGjmxH3Yshh2LYP86oAo6RZQFxM9h4XZOcHjAqQxw84ROY6DXRGuAqLoqIpJsjEm41H7OvFLoCuwxxuxzFDQbGAJsL7ePAQIdXwcBmU6sxx7BkXDPbJhxKxOznmea+Q0frE3l6Zta212ZcnVH9lghsGPRL1cGDWOh/zPW3F7untanc1MCpcXlvi45f/tZzxVfeL/y2y9nv8DG0HW81eVbVQlnhkI4UH6lmQyg2zn7/AX4SkR+A/gB1zuxHvuEd4ahb+M99z4+rP8eD37/CN1i6tO3ZajdlSlXYgxkbYftjiDIcnw+C+8M1/8F2gyG+s3srFBVA84MBalg27nXpKOA94wx/xSRHsCHIhJrjDlrdRoRGQ+MB4iKinJKsU7Xdghc/wJdvn6eP3nXY9yMQiZe35ZfD2iOm1tFb5VSlcAYyNz4SxAc2weI1Vtn0GSrwTYowu4qVTXizFDIAMpPPRjB+beHxgGDAIwxa0XEBwgBssrvZIyZCkwFq03BWQU7Xa/H4dg+hm98n+v91vDqN7cwIW0Y/xjZjSBfT7urU7VFaYnVLrDjc+vPqQzr/nxMX2tW39a3gn8Du6tU1ZQzG5o9sBqaBwIHsBqa7zHGbCu3z1JgjjHmPRFpAywHws1FiqpxDc3nMgZ2f4VZ+QqSsYFsE8RczyEMGD2JtjHhdlenaqqSIkhZaYXAzi/gTBa4e0PzgdZtoZY3gW89u6tUNrrchmanhYKjiFuAKVjdTacbY14SkReBJGPMIkePo3cAf6xbS783xnx1sWPW+FD4mTGQuoqcryYTcHANJ4wfGS3vJ/bOp/WHV12eonzYt8K6NbRrCeSfAE8/aHmjdVuoxY3gHWB3laqaqBah4Ay1JhTKObn7e/Z8+iKd89dS4FYH964P4dHrNxDQ0O7SVHVTcBr2/M8Kgt1fQeFp8AmCVrdYQdDsOvDUtTvU+TQUapiSUsPMhV8QvPF1bnVfj5u7JxJ/r9UOEVxDG9dV5cg7Dj8ts4Jg73Iozge/UKttoM3tEN0XPLzsrlJVcxoKNdQ3Ow/z6uylPMhC7nBbhRsG4kZYozhDWthdnqoqp7Nh1xdWEKR8Z/XfDwy3QqDNYIjqbg30UuoyaSjUYOnHcnnko2SOZabwRvQqOh1ZhBQXWN1a+/wOGsXZXaJyhsIz8MNM2L4Q9n8PphTqxlizfrYZDI3jdfZPddU0FGq4/KISnl+4jTlJ6dwS484/I9dQZ9MMKMyBFjdB36cgsqvdZarKUFoKW+fB/56HnEwIbeMIgtutEcai41jUtdNQqCXmJO7nTwu3Ud/Pi7eGNaND5lxr6uC84xDdx7pyaNpff3HUVBlJsPQPcCAJGnW0BpQ16WF3VaoWutxQ0GvRam5ElyjmT+iJh7sw7L3tfOA1HPPEFmsR8iO74cM7rMXJd35hfeJUNcOpTJg/3vq3O5kOQ/4LD6/QQFC20yuFGuJkbhG/nbuJb3ZmcUfHxvxtaHt8pRh+/BhWT4ETadCgrXXl0O5ObYSsrgpzrXUB1kyxRh73fMzqRKDjCZST6e2jWqi01PDmd3v551e7aN7AnzfHdKZZqD+UFFv3pFe9Ckd2Qb2m0OsJ6DBKuypWF8bA1k+tdoNTGdD2DrjhBZ0KWlWZSg0FEWkGZBhjCkSkPxAHfGCMOXHNlV4hVw6Fn63efYSJs3+gsLiUl4fFcXP7RtYTpaWwczGsegUO/mh1Yew5EeLvAy9fe4t2ZQeS4ctnIH09hMVZ7QbRveyuSrmYyg6FTUACEA0sAxYBrYwxt1xjnVdMQ8GSeSKPR2duZFP6CR7uE8PvB7XG093RRGQM7FluhcP+teAbAj0ehS4PWaNfVdU4dRCWvwA/zgK/BjDwz9DxHr21p2xR2aGw0RgTLyJPA/nGmP+IyA/GmCpfIFVD4ReFxaW89MV23l+bRtfoerx+TycanLuiW9r3sPIVaySsdxB0Gw/dJoBffXuKdgVFebD2dVj1Lygtgu6PWm09PoGXfq1STlLZobAea2K7PwK3G2NSRGSrMSb22ku9MhoK5/vshwM8M38L/j4evD6qE92aVvAL/8BGWP2qNYumpy90HgvdJ1grw6nKYQxsW2C1G5zcb40zuOGvUC/G7sqUqvRQaAs8Aqw1xswSkRhghDFm8rWXemU0FCq261AOEz5KJu1YLpMGteahPjFIRWMXsnZa4bBlnrXkYeP4X+bQCWmp4x2uVuYmq91g//fQsD0M+pu1foFS1YTTeh+JSF0g0hiz+WqLuxYaCheWk1/E7+dtZunWQ9wcG8Y/hsUR4HOBxXuOp1q9YXZ+YTWEAtRvbgVE69sgPEGnVLgcOYdh+YuwaSb41oeBf4JO92q7gap2KvtK4VtgMNZKbZuAbOA7Y8yT11jnFdNQuDhjDO+uSmHylztpUs+XN8d0plXYJfrAnzxgzce/8wtIXWVNvubf0JqOufVtENMHPLyr5huoKYryYd0bVjfg4gLrVlzfp7QhX1VblR0KPxhjOonIQ1hXCc+LyGZjTJXPzKahcHnW7zvKY7N+4HR+MZPvas+Qjpe5qlveCdj9P9j5Oez+GorOgHcgtLjBuopofoNrN5gaY611/NWfrAGDrW6FG/+qC96raq+yQ2ELcCPwPvBHY0yihkL1l3Uqn8c+/oENqce4v0cT/nhrW7w8ruCWUFG+NW3zzsWwcwnkHgE3T2jaz7qCaHWLay0EdPBH+PJZSFsNDdpZ7QZN+9tdlVKXpbJD4W7gT8AaY8wEEWkKvGyMuevaS70yGgpXpqiklJeX7WLqyn10jAzmv6PjaRx8FStzlZZA+gZHQCy22iQQiOjySztESPPKLr96OJ0F3/wVNn5oLZV63XPQ6T5w97C7MqUum05zoc6ydMtBnp63GS8PN14b2YneLUKu/mDGQNYOqw1i5+fWJ2iA0NaOgLjV6tVU03syFRfAujetcR7FedDtEej7NNQJtrsypa5YZV8pRAD/AXoBBlgNPG6MybjWQq+UhsLV25t9mgkfJbM76zS/u6Elj/ZvjptbJfziPpHuaKheDKlrrK6uAY2htaOhOro3uF+gF1R1ZIwVeF89B8dToOXNcOP/1d4rIeUSKjsU/gd8DHzo2DQGGG2MueGaqrwKGgrXJrewmGfmb2Hhpkz6tAjhxSGxxIT4VeIJjlnrCe9cbE21UZxn9chpcRO0uQ2aDQRv/8o7X2U7tBW+nGT1wgptAze9BM0H2l2VUtes0uc+MsZ0vNS2qqChcO2MMcxcv5/JS3dSWFzKw31j+PWA5vh6VfI98sJc2Pet9al71xLIOwbu3tBsgHWLqdUt4HcNt7EqizFwJhtWvAQbPwCfYBjwrDXqW9sNVC1R2aHwNfAeMMuxaRQw1hhT5R+hNBQqT1ZOPpOX7mT+xgM0CvLhuVvbckv7sIpHQl+rkmJIX2cFxI7F1jQQ4gaR3a2pvk2J1Zh91t+l5zwurmDbufsWX+T1Jda6x6WO/X7ehuNnwM0DujwM/f8AdepW/nuglI0qOxSigNeBHlg/Qd8DE40x+6+10CuloVD5klKP8eeF29h+8BS9mtfnhcHtaN7AiYu+GAOHt1rhsGsJ5B4FcbdGUIu7NRq47G8365f1edvcL7Dvz489rmBfd6vNo80QCG3pvO9bKRs5vfeRiDxhjJlyVS++BhoKzlFSavh4fRovL9tFbmEJY3tFM3FgiwtPk6GUqlGqYo3mKp/iQjmPu5twb49oVjzVn2GdI3h3dQoD//kdn/1wgJrWbVkpdfWuJRRqeCd0VZH6/t5MviuOBY/2olGQD0/M2cSIt9ex4+Apu0tTSlWBawkF/fhYi3WMDGbBo72YPLQ9u7NyuPW1Vfxl0TZO5hXZXZpSyokuGgoikiMipyr4kwM0vtTBRWSQiOwSkT0iMukC+wwXke0isk1EPr7K70M5gZubMLJrFCue6s+Y7k34YG0q173yLXMT0ykt1c8EStVGTpvmQkTcgZ+AG4AMIBEYZYzZXm6fFsBc4DpjzHERaWCMybrYcbWh2T7bMk/y/MJtJKUdp2NkMC8OaUdchE75oFRNUBUNzZfSFdhjjNlnjCkEZgNDztnnYeANY8xxgEsFgrJXu8ZBfPJID14d3oGM43kMeWMNz8zfwrEzhXaXppSqJM4MhXAgvdzjDMe28loCLUVkjYisE5FBTqxHVQIRYWh8BN881Y9xvWKYm5TOdf/8lo/WpVGit5SUqvGcGQoV9U4697eGB9AC6I81SvpdETnvfoSIjBeRJBFJys7OrvRC1ZUL9PHkudvasvTxPrQJC+S5z7Yy5I3VJKcdt7s0pdQ1cGYoZACR5R5HAJkV7LPQGFNkjEkBdmGFxFmMMVONMQnGmITQ0FCnFayuXMuGAXz8cDf+M6oTR3IKuevN73nqkx/JzimwuzSl1FVwZigkAi1EJEZEvICRwKJz9vkMGAAgIiFYt5P2ObEm5QQiwu0dGrP8d/2Y0L8ZCzcd4LpXvmX66hSKS0rtLk8pdQWcFgrGmGLgMWAZsAOYa4zZJiIvishgx27LgKMish1YATxtjDnqrJqUc/l5e/CHQa1Z9kRfOjWpy4uLt3Pra6tZt0//SZWqKXTlNeUUxhi+2n6YFz/fzoETeQzu0Jhnb2lDWJCP3aUp5ZKqQ5dU5cJEhJvahfH1k/14fGALvtx2iIH//Ja3v9tLYbHeUlKqutJQUE5Vx8ud397Qkq9/248ezUL4+9KdDPr3Slbt1l5kSlVHGgqqSkTV9+Xd+xOY8UAXSkoN907bwISPksk4nmt3aUqpcnStQVWlBrRuQI9m9Zm2OoX/fLObFbuyGNc7hnG9m1LPz8vu8pRyedrQrGxz4EQef1+ygy+2HMTHw50x3aN4uE9TGgRqY7RSlc3pK6/ZRUOh9tmTlcN/V+xl4Y+ZuLsJI7tE8qt+zQgPrmN3aUrVGhoKqsZJO3qGt77by7zkDIyBofHhTOjfnJgQP7tLU6rG01BQNVbmiTymrtzHrA37KSop5fYOjfn1gOa0bBhgd2lK1VgaCqrGy84p4N3V+/hobRpnCku4qV1DHhvQgvYRQXaXplSNo6Ggao3jZwqZ8X0q761J4VR+Mf1ahvKb65qTEF3P7tKUqjE0FFStk5NfxIfr0nh3VQrHzhTSvWk9fnNdC3o2q49IRTO1K6V+pqGgaq3cwmJmbUhn6sq9HD5VQKeoYB4b0JzrWjfQcFDqAjQUVK2XX1TCvOQM3vpuLxnH82jbKJDHrmvOoHZhuLlpOChVnoaCchlFJaUs3JTJf1fsYd+RMzRv4M+j/ZsxuENjPNx1JhelQENBuaCSUsOSLQd5Y8Uedh7KIaqeLxP6N2NofDjeHu52l6eUrTQUlMsqLTUs35nF69/s5seMkzQK8mF836aM7BJFHS8NB+WaNBSUyzPGsGr3EV7/Zg8bUo8R4u/FQ32aMqZ7E/y9dS5I5Vo0FJQqZ/2+o7y+Yg+rdh8hqI4nY3tFM7ZnDEG+nnaXplSV0FBQqgKb0k/w+jd7+HrHYfy9Pbi3RxPG9Y4hxN/b7tKUcioNBaUuYsfBU7yxYg9fbDmIt4cbo7pGMaF/MxoE6LTdqnbSUFDqMuzJOs2b3+7ls00H8HJ34+G+TRnft6m2OahaR0NBqSuQeuQMLy/bxRdbDhLi78Xj17dkZJdIPHWcg6olLjcU9H+8UkB0iB9vjI5nwaM9aRriz58+28pN/1rJl1sPUtM+OCl1LTQUlCqnU1Rd5vyqO+/el4Cbm/DIRxu5683vSUo9ZndpSlUJDQWlziEiXN+2IV8+3oe/D21PxvE8hr21lvEfJLEn67Td5SnlVNqmoNQl5BYWM21VCm+v3EdeUQkju0Ty+PUttKeSqlG0oVmpSnbkdAH/Wb6bmev34+XhxsN9rJ5KftpTSdUAGgpKOUnKkTO8UtZTyZsnrm/BCO2ppKq5atH7SEQGicgSQQ8ZAAAUtklEQVQuEdkjIpMust8wETEicsmClbJbjKOn0vxHe9I0xI/nynoqHdKeSqrGc1ooiIg78AZwM9AWGCUibSvYLwCYCKx3Vi1KOUO8o6fSO2U9lZIZ9tZa7amkajRnXil0BfYYY/YZYwqB2cCQCvb7K/APIN+JtSjlFCLCDeV6Ku0/lsuwt9byqw+T2JutPZVUzePMUAgH0ss9znBsKyMinYBIY8xiJ9ahlNN5uFvzJ333dH+evKElq3cf4cZ/reSPC7aQlaOfd1TN4cxQqGiR3LIbriLiBvwL+N0lDyQyXkSSRCQpOzu7EktUqnL5enkwcWALvvv9AEZ3i2JOYjr9X/6WKV//xJmCYrvLU+qSnBkKGUBkuccRQGa5xwFALPCtiKQC3YFFFTU2G2OmGmMSjDEJoaGhTixZqcoR4u/Ni0Ni+d+T/ejfKpQpX++m38vf8tG6NIpKSu0uT6kLcmYoJAItRCRGRLyAkcCin580xpw0xoQYY6KNMdHAOmCwMUb7m6paIybEj/+O7sz8R3sSE+Jr9VSaspJl27SnkqqenBYKxphi4DFgGbADmGuM2SYiL4rIYGedV6nqKD6qLnN/1YN37ktAgF99aPVUSk7TnkqqetHBa0pVseKSUuYmZfCvr38iO6eAQe3C+P2gVjQN9be7NFWL6Yhmpaq53MJi3l2Vwtvf7SW/uJRRXSN5fGBLQgN0aVBV+TQUlKohsnMKeG35bmZtsOZUGtIxnJFdIomLCEKkok58Sl05DQWlaph92ad5Y8VevtiSSX5RKa3DAhjZJZI7OoUT7Otld3mqhtNQUKqGOpVfxKJNmcxJTGfLgZN4ebhxc2wYI7pE0j2mPm5uevWgrpyGglK1wLbMk8xNTGfBDwc4lV9Mk/q+DE+IZFjnCBoG6noO6vJpKChVi+QXlfDl1kPMTtzPun3HcHcTBrQKZUSXKAa0CsVDp+1Wl6ChoFQtlXLkDHOT0pmXnEF2TgENArwZ1jmC4QmRRIf42V2eqqY0FJSq5YpKSlmxM4s5iems2JVFqYEeTeszsmskN7ULw8fT3e4SVTWioaCUCzl0Mp95yenMSUon/VgeQXU8ubNTOCO6RNKmUaDd5alqQENBKRdUWmpYu+8osxPTWbb1EIUlpXSICGJElyhu79CIAB9Pu0tUNtFQUMrFHT9TyGebDjB7Qzq7DudQx9OdW+MaMbJLJJ2b1NWBcS5GQ0EpBYAxhh8zTjIncT+LNmVyprCEZqF+jOwSxdD4cOr767QarkBDQSl1njMFxXyx+SCzE/ezcf8JPN2t5URHdImid/MQ3HVgXK2loaCUuqifDucwJzGd+RszOJ5bRHhwHe5OiODuhEjCg+vYXZ6qZC4VCkVFRWRkZJCfr2vhViYfHx8iIiLw9NTGydqsoLiEr7dnMTtxP6v3HAGgZ7P69Ghan85N6tEhMghfLw+bq1TXyqVCISUlhYCAAOrXr6+NZ5XEGMPRo0fJyckhJibG7nJUFUk/lssnyRks3XKQ3VmnAXB3E9o1DiQ+qi4J0XXp3KQujYL0SqKmcalQ2LFjB61bt9ZAqGTGGHbu3EmbNm3sLkXZ4GRuERv3Hyc57ThJacfYlH6C/CJrfenw4DrEN6lLQhMrJFqHBehUG9Xc5YZCrbkm1ECofPqeurYgX08GtG7AgNYNAGsE9Y6DpxwhcZzElGN8/mMmAL5e7nSMDCahSV3iHX8CdUxEjVRrQsFOR48eZeDAgQAcOnQId3d3QkNDAdiwYQNeXpeeC3/s2LFMmjSJVq1aObVWpa6Wp7sbcRHBxEUEM7aXdUvxwIk8ktOOk5x6jOT9x3l9xR5KDYhAywYBdI6uS2fHbaeoer76QaMGqDW3j6rLLY6//OUv+Pv789RTT5213RiDMQY3t5p1iV2d3ltV/Z0pKGZT+omyq4kf0o6TU1AMQIi/N52bBJPQpB7xTeoSGx6It4fOz1RVXO72UXW0Z88e7rjjDnr37s369etZvHgxL7zwAhs3biQvL48RI0bw5z//GYDevXvz+uuvExsbS0hICI888ghLly7F19eXhQsX0qBBA5u/G6Uuzc/bg17NQ+jVPASwpt34KSvHcTVxnOT9x1m27TAAXh5uxIUHlV1NdG5SVwfSVQO1LhRe+Hwb2zNPVeox2zYO5Pnb213Va7dv386MGTN46623AJg8eTL16tWjuLiYAQMGMGzYMNq2bXvWa06ePEm/fv2YPHkyTz75JNOnT2fSpEnX/H0oVdXc3ITWYYG0DgtkdLcmAGTl5LMx7QTJacdITjvO9NUpvF2yD4CYED86OxqvE5rUpVmov640V8VqXShUN82aNaNLly5lj2fNmsW0adMoLi4mMzOT7du3nxcKderU4eabbwagc+fOrFq1qkprVsqZGgT4MCg2jEGxYYC1gNDWAydJSrN6On2zM4t5yRmOfb25q3MEI3StiCpT60Lhaj/RO4uf3y//kXfv3s2///1vNmzYQHBwMGPGjKlwwF35hml3d3eKi4urpFal7ODj6U5CdD0SousBVvtb6tFcklKPsWzbId7+bi9vfruX7k3rMbJLFINida0IZ6p1oVCdnTp1ioCAAAIDAzl48CDLli1j0KBBdpelVLUiIsSE+BET4sfdCZEcOpnPpxszmJOYzhNzNhG40IM7HGtFtGscZHe5tY6GQhWKj4+nbdu2xMbG0rRpU3r16mV3SUpVe2FBPvx6QHMm9GvGupSjzElMZ3ZiOh+sTaN9eBDDu0QypGNjHRdRSbRLqroofW9VdXQit5CFmzKZtWE/Ow/l4OPpxi3tGzGySxRdonWtiIpol1SlVK0V7OvF/T2jua9HE7YcOMnsxHQWbcpk/sYDNA3xY3iXSO6KjyA0QLu4XimnjqQSkUEisktE9ojIeX0qReRJEdkuIptFZLmINHFmPUqp2kVEiIsI5m93tmfDHwfyyt0dqO/vxeSlO+nx9+X86sMkvtl5mOKSUrtLrTGcdqUgIu7AG8ANQAaQKCKLjDHby+32A5BgjMkVkQnAP4ARzqpJKVV7+Xp5MKxzBMM6R7An6zSfJKUzLzmDZdsOExbow90JEQxPiCSynq/dpVZrzrxS6ArsMcbsM8YUArOBIeV3MMasMMbkOh6uAyKcWI9SykU0b+DPM7e0Ye0zA3lrTDytGwXwxoo99PnHCka/u45FP2aSX1Rid5nVkjPbFMKB9HKPM4BuF9l/HLDUifUopVyMl4cbg2IbMSi2EZkn8piXbHVtnTjrB4LqeHJnp3BGdo2kdVig3aVWG84MhYqa/yvs6iQiY4AEoN8Fnh8PjAeIioqqrPqUUi6kcXAdJg5swWMDmrNm7xHmJKbz8fr9vPd9Kh0igxmREMntHRoR4OJdW515+ygDiCz3OALIPHcnEbke+CMw2BhTUNGBjDFTjTEJxpiEn6ekrk769+/PsmXLzto2ZcoUHn300Qu+xt/fH4DMzEyGDRt2weNeaj3qKVOmkJubW/b4lltu4cSJE5dbulIux81N6NMilNfviWfdswP5021tySss5tkFW+j60nKe/uRHktOOUdO661cWZ4ZCItBCRGJExAsYCSwqv4OIdALexgqELCfW4lSjRo1i9uzZZ22bPXs2o0aNuuRrGzduzLx586763OeGwpIlSwgODr7q4ynlSur5eTGudwzLnujLgkd7MqRjY77YcpC73lzL9a9+xzsr93H0dIWfVWstp4WCMaYYeAxYBuwA5hpjtonIiyIy2LHby4A/8ImIbBKRRRc4XLU2bNgwFi9eTEGB9Z8nNTWVzMxMOnbsyMCBA4mPj6d9+/YsXLjwvNempqYSGxsLQF5eHiNHjiQuLo4RI0aQl5dXtt+ECRNISEigXbt2PP/88wC89tprZGZmMmDAAAYMGABAdHQ0R45Yi6+/+uqrxMbGEhsby5QpU8rO16ZNGx5++GHatWvHjTfeeNZ5lHJFIkKnqLpMviuODX+8nv93V3uC6njy0pIddP/7ckZOXcsLn29jTuJ+NqWfILew9s5H5tTBa8aYJcCSc7b9udzX11f6SZdOgkNbKveYYe3h5skXfLp+/fp07dqVL7/8kiFDhjB79mxGjBhBnTp1WLBgAYGBgRw5coTu3bszePDgC462fPPNN/H19WXz5s1s3ryZ+Pj4sudeeukl6tWrR0lJCQMHDmTz5s1MnDiRV199lRUrVhASEnLWsZKTk5kxYwbr16/HGEO3bt3o168fdevWZffu3cyaNYt33nmH4cOH8+mnnzJmzJjKea+UquH8vT0Y0SWKEV2i+OlwDnMT00lMO87sDenkOXosiUCTer60CgtwTA0eQKuwAJrU98O9hk/1rSOaK8nPt5B+DoXp06djjOHZZ59l5cqVuLm5ceDAAQ4fPkxYWFiFx1i5ciUTJ04EIC4ujri4uLLn5s6dy9SpUykuLubgwYNs3779rOfPtXr1au68886yWVqHDh3KqlWrGDx4MDExMXTs2BGwpuZOTU2tpHdBqdqlZcMAnrvNmtq+tNSw/1guOw/lsOtQDjsPnWLXoRz+t/0wpY7mBx9PN1o2DKBVQysk2jQKpFVYACE1aPGg2hcKF/lE70x33HEHTz75ZNmqavHx8bz33ntkZ2eTnJyMp6cn0dHRFU6VXV5FVxEpKSm88sorJCYmUrduXR544IFLHudijWTe3r/8B3V3d9fbR0pdBjc3ITrEj+gQv7K1IADyCkvYnZVzVlis2JXFJ441IQBC/L1oHRbouLKwri5aNPSvllOA175QsIm/vz/9+/fnwQcfLGtgPnnyJA0aNMDT05MVK1aQlpZ20WP07duXmTNnMmDAALZu3crmzZsBa8ptPz8/goKCOHz4MEuXLqV///4ABAQEkJOTc97to759+/LAAw8wadIkjDEsWLCADz/8sPK/caVcXB0vd+IigomLOLuDR3ZOwVlXFDsP5fDRujQKiq0pN9wEokP8rFtPDQMdVxYBRNb1tXW1OQ2FSjRq1CiGDh1a1hNp9OjR3H777SQkJNCxY0dat2590ddPmDCBsWPHEhcXR8eOHenatSsAHTp0oFOnTrRr1+68KbfHjx/PzTffTKNGjVixYkXZ9vj4eB544IGyYzz00EN06tRJbxUpVUVCA7wJDfCmd4tfPrCVlBrSjp5h56GfryxOsT3zFEu3HuLni3tfL3daNAygjaOd4ud2i3p+Xhc4U+XSqbPVRel7q5Tz5RYW89Ph0+w6dIodB3+5DXU8t6hsnwYB3jzcpykP9216VefQqbOVUqqG8PXyoGNkMB0jf7kFZYwhO6egrK1ix6FTNAh0foO1hoJSSlVDIkKDQB8aBPrQt2XVzeTg1PUUlFJK1Sy1JhRqWttITaDvqVKup1aEgo+PD0ePHtVfYpXIGMPRo0fx8fGxuxSlVBWqFW0KERERZGRkkJ2dbXcptYqPjw8REbrukVKupFaEgqenJzExMXaXoZRSNV6tuH2klFKqcmgoKKWUKqOhoJRSqkyNm+ZCRLKBi88sd2EhwJFKLKem0/fjbPp+/ELfi7PVhvejiTHmkqPgalwoXAsRSbqcuT9chb4fZ9P34xf6XpzNld4PvX2klFKqjIaCUkqpMq4WClPtLqCa0ffjbPp+/ELfi7O5zPvhUm0KSimlLs7VrhSUUkpdhMuEgogMEpFdIrJHRCbZXY+dRCRSRFaIyA4R2SYij9tdk91ExF1EfhCRxXbXYjcRCRaReSKy0/F/pIfdNdlFRH7r+BnZKiKzRKTWzxDpEqEgIu7AG8DNQFtglIi0tbcqWxUDvzPGtAG6A7928fcD4HFgh91FVBP/Br40xrQGOuCi74uIhAMTgQRjTCzgDoy0tyrnc4lQALoCe4wx+4wxhcBsYIjNNdnGGHPQGLPR8XUO1g99uL1V2UdEIoBbgXftrsVuIhII9AWmARhjCo0xJ+ytylYeQB0R8QB8gUyb63E6VwmFcCC93OMMXPiXYHkiEg10AtbbW4mtpgC/B0rtLqQaaApkAzMct9PeFRE/u4uygzHmAPAKsB84CJw0xnxlb1XO5yqhIBVsc/luVyLiD3wKPGGMOWV3PXYQkduALGNMst21VBMeQDzwpjGmE3AGcMk2OBGpi3VHIQZoDPiJyBh7q3I+VwmFDCCy3OMIXOAy8GJExBMrEGYaY+bbXY+NegGDRSQV67bidSLykb0l2SoDyDDG/HzlOA8rJFzR9UCKMSbbGFMEzAd62lyT07lKKCQCLUQkRkS8sBqLFtlck21ERLDuGe8wxrxqdz12MsY8Y4yJMMZEY/2/+MYYU+s/DV6IMeYQkC4irRybBgLbbSzJTvuB7iLi6/iZGYgLNLrXipXXLsUYUywijwHLsHoQTDfGbLO5LDv1Au4FtojIJse2Z40xS2ysSVUfvwFmOj5A7QPG2lyPLYwx60VkHrARq8feD7jAyGYd0ayUUqqMq9w+UkopdRk0FJRSSpXRUFBKKVVGQ0EppVQZDQWllFJlNBSUqkIi0l9nYlXVmYaCUkqpMhoKSlVARMaIyAYR2SQibzvWWzgtIv8UkY0islxEQh37dhSRdSKyWUQWOObMQUSai8jXIvKj4zXNHIf3L7dewUzHaFmlqgUNBaXOISJtgBFAL2NMR6AEGA34ARuNMfHAd8Dzjpd8APzBGBMHbCm3fSbwhjGmA9acOQcd2zsBT2Ct7dEUa4S5UtWCS0xzodQVGgh0BhIdH+LrAFlYU2vPcezzETBfRIKAYGPMd47t7wOfiEgAEG6MWQBgjMkHcBxvgzEmw/F4ExANrHb+t6XUpWkoKHU+Ad43xjxz1kaRP52z38XmiLnYLaGCcl+XoD+HqhrR20dKnW85MExEGgCISD0RaYL18zLMsc89wGpjzEnguIj0cWy/F/jOsT5Fhojc4TiGt4j4Vul3odRV0E8oSp3DGLNdRJ4DvhIRN6AI+DXWgjPtRCQZOInV7gBwP/CW45d++VlF7wXeFpEXHce4uwq/DaWuis6SqtRlEpHTxhh/u+tQypn09pFSSqkyeqWglFKqjF4pKKWUKqOhoJRSqoyGglJKqTIaCkoppcpoKCillCqjoaCUUqrM/we4SDWQyuVATQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(np.arange(len(train_losses)), train_losses, label='Train')\n",
    "plt.plot(np.arange(len(valid_losses)), valid_losses, label='Validation')\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What can you infer from this plot?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Test the network on the test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to test the model on an unseen set, test set, if we finished training the model.\n",
    "\n",
    "The procedure is same with validation in the training procedure except that we do not need iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of Seizure : 92% out of 438 cases\n",
      "Accuracy of TumorArea : 62% out of 471 cases\n",
      "Accuracy of HealthyArea : 60% out of 462 cases\n",
      "Accuracy of EyesClosed : 71% out of 493 cases\n",
      "Accuracy of EyesOpen : 79% out of 436 cases\n"
     ]
    }
   ],
   "source": [
    "class_correct = list(0. for i in range(len(classes)))\n",
    "class_total = list(0. for i in range(len(classes)))\n",
    "with torch.no_grad():\n",
    "    for data in testloader:\n",
    "        # get the inputs\n",
    "        inputs, targets = data\n",
    "\n",
    "        if cuda:\n",
    "            inputs, targets = inputs.cuda(), targets.cuda()\n",
    "\n",
    "        outputs = model(inputs)\n",
    "        _, predicted = torch.max(outputs.data, 1)\n",
    "        c = (predicted == targets).squeeze()\n",
    "        for i, label in enumerate(targets):\n",
    "            class_correct[label] += c[i].item()\n",
    "            class_total[label] += 1\n",
    "    \n",
    "for i in range(len(classes)):\n",
    "    print('Accuracy of %s : %2d%% out of %d cases' % (classes[i], 100 * class_correct[i] / class_total[i], class_total[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! We have completed the example of convolutional neural network!\n",
    "\n",
    "We will see how we can use recurrent neural network also in the next chapter, and it will be very similar with the one in this chapter just with a differnt model class definition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1. What can we infer from the loss plot above?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2. Try to run on CPU and compare the computation time? Is there a huge difference or not? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3. Modify the network or the training procedure to improve the performance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4. Since the class 1 (Epileptic seizure) is the one we mostly care about, try to convert the problem into binary (class 1 vs rest) classification. Then, train and test the network to see the performce on binary setting."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pytorch1]",
   "language": "python",
   "name": "conda-env-pytorch1-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}