HiggsToBB_H5_part2_machine_learning_examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Guide for the HiggsToBB dataset (Part 2)\n",
    "\n",
    "This notebook is a guide for using the HiggsToBBNTuple_HiggsToBB_QCD_RunII_13TeV_MC dataset. \n",
    "\n",
    "In Part 2, we focus on machine learning tasks on this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset\n",
    "\n",
    "Cite as: Duarte, Javier; (2019). Sample with jet, track and secondary vertex properties for Hbb tagging ML studies HiggsToBBNTuple_HiggsToBB_QCD_RunII_13TeV_MC. CERN Open Data Portal. DOI:[10.7483/OPENDATA.CMS.JGJX.MS7Q](http://doi.org/10.7483/OPENDATA.CMS.JGJX.MS7Q)\n",
    "\n",
    "## Description: \n",
    "\n",
    "The dataset has been produced for developing machine-learning algorithms to differentiate jets originating from a Higgs boson decaying to a bottom quark-antiquark pair (Hbb) from quark or gluon jets originating from quantum chromodynamic (QCD) multijet production.\n",
    "\n",
    "The reconstructed jets are clustered using the anti-kT algorithm with R=0.8 from particle flow (PF) candidates (AK8 jets). The standard L1+L2+L3+residual jet energy corrections are applied to the jets and pileup contamination is mitigated using the charged hadron subtraction (CHS) algorithm. Features of the AK8 jets with transverse momentum pT > 200 GeV and pseudorapidity |η| < 2.4 are provided. Selected features of inclusive (both charged and neutral) PF candidates with pT > 0.95 GeV associated to the AK8 jet are provided. Additional features of charged PF candidates (formed primarily by a charged particle track) with pT > 0.95 GeV associated to the AK8 jet are also provided. Finally, additional features of reconstructed secondary vertices (SVs) associated to the AK8 jet (within ∆R < 0.8) are also provided.\n",
    "\n",
    "## File Information\n",
    "\n",
    "There are two lists of files, one in ROOT format and another in H5. For H5 files, only information for up to 100 particle candidates, up to 60 charged particles or tracks, and up to 5 secondary vertices are stored in zero-padded arrays. \n",
    "\n",
    "List of H5 files: \n",
    "- Test: http://opendata.cern.ch/record/12102/files/HiggsToBBNTuple_HiggsToBB_QCD_RunII_13TeV_MC_test_h5_file_index.txt\n",
    "- Train: http://opendata.cern.ch/record/12102/files/HiggsToBBNTuple_HiggsToBB_QCD_RunII_13TeV_MC_train_h5_file_index.txt\n",
    "\n",
    "List of ROOT files:\n",
    "- Test: http://opendata.cern.ch/record/12102/files/HiggsToBBNTuple_HiggsToBB_QCD_RunII_13TeV_MC_test_root_file_index.txt\n",
    "- Train: http://opendata.cern.ch/record/12102/files/HiggsToBBNTuple_HiggsToBB_QCD_RunII_13TeV_MC_train_root_file_index.txt\n",
    "\n",
    "The size of each H5 file is 1.4 GB, and the size of each ROOT file is 1.1 GB. \n",
    "\n",
    "This notebook help you walk through the **H5** format dataset, and it's derived from this list of notebooks that are developed for the **ROOT** format: https://jmduarte.github.io/capstone-particle-physics-domain/weeks/02-dataset.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tables\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os.path\n",
    "import pandas as pd\n",
    "\n",
    "filename = 'ntuple_merged_11.h5' # train file\n",
    "if not os.path.isfile(filename):\n",
    "    !wget http://opendata.cern.ch/eos/opendata/cms/datascience/HiggsToBBNtupleProducerTool/HiggsToBBNTuple_HiggsToBB_QCD_RunII_13TeV_MC/train/ntuple_merged_11.h5 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2. Machine Learning Examples  \n",
    "## 2.1 Classification: Particle Identification\n",
    "\n",
    "**Features**: dimension (n,27)\n",
    "\n",
    "**Labels**: dimension (n,2) in one-hot. (1,0) means background(QCD) and (0,1) means H(bb).\n",
    "\n",
    "**Classifiers**:\n",
    "\n",
    "1. Decision Tree Classifier: max depth = 5\n",
    "2. SVM Classifier\n",
    "3. Fully Connected Neural Network Classifier: optimizer='adam', loss='categorical_crossentropy', metrics='accuracy'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "import numpy as np\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-05-04 14:03:19--  http://opendata.cern.ch/eos/opendata/cms/datascience/HiggsToBBNtupleProducerTool/HiggsToBBNTuple_HiggsToBB_QCD_RunII_13TeV_MC/train/ntuple_merged_10.h5\n",
      "Resolving opendata.cern.ch (opendata.cern.ch)... 188.185.82.144, 188.184.93.89, 188.184.28.138\n",
      "Connecting to opendata.cern.ch (opendata.cern.ch)|188.185.82.144|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1506845769 (1.4G) [application/octet-stream]\n",
      "Saving to: ‘ntuple_merged_10.h5’\n",
      "\n",
      "ntuple_merged_10.h5   1%[                    ]  14.78M  55.8KB/s    eta 5h 28m ^C\n"
     ]
    },
    {
     "ename": "HDF5ExtError",
     "evalue": "HDF5 error back trace\n\n  File \"H5F.c\", line 509, in H5Fopen\n    unable to open file\n  File \"H5Fint.c\", line 1652, in H5F_open\n    unable to read superblock\n  File \"H5Fsuper.c\", line 623, in H5F__super_read\n    truncated file: eof = 15513498, sblock->base_addr = 0, stored_eof = 1506845769\n\nEnd of HDF5 error back trace\n\nUnable to open/create file 'ntuple_merged_10.h5'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mHDF5ExtError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-9-21422eadc21c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     83\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0misfile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_filename\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     84\u001b[0m      \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msystem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'wget http://opendata.cern.ch/eos/opendata/cms/datascience/HiggsToBBNtupleProducerTool/HiggsToBBNTuple_HiggsToBB_QCD_RunII_13TeV_MC/train/ntuple_merged_10.h5 .'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 85\u001b[0;31m \u001b[0mfeature_array\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabel_array\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_features_labels\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_filename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mremove_mass_pt_window\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     86\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfeature_array\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabel_array\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-9-21422eadc21c>\u001b[0m in \u001b[0;36mget_features_labels\u001b[0;34m(file_name, remove_mass_pt_window)\u001b[0m\n\u001b[1;32m     43\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     44\u001b[0m     \u001b[0;31m# load file\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 45\u001b[0;31m     \u001b[0mh5file\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtables\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen_file\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfile_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     46\u001b[0m     \u001b[0mnjets\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mh5file\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mroot\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mfeatures\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     47\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tfvenv/lib/python3.7/site-packages/tables/file.py\u001b[0m in \u001b[0;36mopen_file\u001b[0;34m(filename, mode, title, root_uep, filters, **kwargs)\u001b[0m\n\u001b[1;32m    313\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    314\u001b[0m     \u001b[0;31m# Finally, create the File instance, and return it\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 315\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mFile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtitle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mroot_uep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfilters\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    316\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    317\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tfvenv/lib/python3.7/site-packages/tables/file.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, filename, mode, title, root_uep, filters, **kwargs)\u001b[0m\n\u001b[1;32m    776\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    777\u001b[0m         \u001b[0;31m# Now, it is time to initialize the File extension\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 778\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_g_new\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    779\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    780\u001b[0m         \u001b[0;31m# Check filters and set PyTables format version for new files.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mtables/hdf5extension.pyx\u001b[0m in \u001b[0;36mtables.hdf5extension.File._g_new\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mHDF5ExtError\u001b[0m: HDF5 error back trace\n\n  File \"H5F.c\", line 509, in H5Fopen\n    unable to open file\n  File \"H5Fint.c\", line 1652, in H5F_open\n    unable to read superblock\n  File \"H5Fsuper.c\", line 623, in H5F__super_read\n    truncated file: eof = 15513498, sblock->base_addr = 0, stored_eof = 1506845769\n\nEnd of HDF5 error back trace\n\nUnable to open/create file 'ntuple_merged_10.h5'"
     ]
    }
   ],
   "source": [
    "# selected features useful for the classification task.\n",
    "features = ['fj_jetNTracks', # Number of tracks associated with the AK8 jet\n",
    "            'fj_nSV',  # Number of SVs associated with the AK8 jet (∆R < 0.7)\n",
    "            'fj_tau0_trackEtaRel_0',# Smallest track pseudorapidity ∆η, relative to the jet axis, associated to the 1st N-subjettiness axis\n",
    "            'fj_tau0_trackEtaRel_1', # Second smallest ...\n",
    "            'fj_tau0_trackEtaRel_2', # Third smallest ...\n",
    "            'fj_tau1_trackEtaRel_0', # Smallest track pseudorapidity ∆η, relative to the jet axis, associated to the 2nd N-subjettiness axis\n",
    "            'fj_tau1_trackEtaRel_1', # Second smallest ...\n",
    "            'fj_tau1_trackEtaRel_2', # Thrid smallest ...\n",
    "            'fj_tau_flightDistance2dSig_0', # Transverse (2D) flight distance significance between the PV and the SV with the smallest uncertainty on the 3D flight distance associated to the first N-subjettiness axis\n",
    "            'fj_tau_flightDistance2dSig_1', # ... associated to the second N-subjettiness axis\n",
    "            'fj_tau_vertexDeltaR_0',  # Pseudoangular distance ∆R between the first N-subjettiness axis and SV direction\n",
    "            'fj_tau_vertexEnergyRatio_0', # SV vertex energy ratio for the first N-subjettiness axis, defined as the total energy of all SVs associated with the first N-subjettiness axis divided by the total energy of all the tracks associated with the AK8 jet that are consistent with the PV\n",
    "            'fj_tau_vertexEnergyRatio_1', # SV vertex energy ratio for the second N-subjettiness axis\n",
    "            'fj_tau_vertexMass_0', \n",
    "            'fj_tau_vertexMass_1',\n",
    "            'fj_trackSip2dSigAboveBottom_0',\n",
    "            'fj_trackSip2dSigAboveBottom_1',\n",
    "            'fj_trackSip2dSigAboveCharm_0',\n",
    "            'fj_trackSipdSig_0',\n",
    "            'fj_trackSipdSig_0_0',\n",
    "            'fj_trackSipdSig_0_1',\n",
    "            'fj_trackSipdSig_1',\n",
    "            'fj_trackSipdSig_1_0',\n",
    "            'fj_trackSipdSig_1_1',\n",
    "            'fj_trackSipdSig_2',\n",
    "            'fj_trackSipdSig_3',\n",
    "            'fj_z_ratio']\n",
    "\n",
    "# spectators to define mass/pT window\n",
    "spectators = ['fj_sdmass', # Soft drop mass of the AK8 jet\n",
    "              'fj_pt'] # Transverse momentum of the AK8 jet\n",
    "\n",
    "# 2 labels: QCD or Hbb\n",
    "labels = ['fj_isQCD*sample_isQCD',\n",
    "          'fj_isH*fj_isBB']\n",
    "\n",
    "nfeatures = len(features)\n",
    "nspectators = len(spectators)\n",
    "nlabels = len(labels)\n",
    "\n",
    "def get_features_labels(file_name, remove_mass_pt_window=True):\n",
    "\n",
    "    # load file\n",
    "    h5file = tables.open_file(file_name, 'r')\n",
    "    njets = getattr(h5file.root,features[0]).shape[0]\n",
    "\n",
    "    # allocate arrays\n",
    "    feature_array = np.zeros((njets,nfeatures))\n",
    "    spec_array = np.zeros((njets,nspectators))\n",
    "    label_array = np.zeros((njets,nlabels))\n",
    "\n",
    "    # load feature arrays\n",
    "    for (i, feat) in enumerate(features):\n",
    "        feature_array[:,i] = getattr(h5file.root,feat)[:]\n",
    "\n",
    "    # load spectator arrays\n",
    "    for (i, spec) in enumerate(spectators):\n",
    "        spec_array[:,i] = getattr(h5file.root,spec)[:]\n",
    "\n",
    "    # load labels arrays\n",
    "    for (i, label) in enumerate(labels):\n",
    "        prods = label.split('*')\n",
    "        prod0 = prods[0]\n",
    "        prod1 = prods[1]\n",
    "        fact0 = getattr(h5file.root,prod0)[:]\n",
    "        fact1 = getattr(h5file.root,prod1)[:]\n",
    "        label_array[:,i] = np.multiply(fact0,fact1)\n",
    "\n",
    "    # remove samples outside mass/pT window\n",
    "    if remove_mass_pt_window:\n",
    "        feature_array = feature_array[(spec_array[:,0] > 40) & (spec_array[:,0] < 200) & (spec_array[:,1] > 300) & (spec_array[:,1] < 2000)]\n",
    "        label_array = label_array[(spec_array[:,0] > 40) & (spec_array[:,0] < 200) & (spec_array[:,1] > 300) & (spec_array[:,1] < 2000)]\n",
    "\n",
    "    feature_array = feature_array[np.sum(label_array,axis=1)==1]\n",
    "    label_array = label_array[np.sum(label_array,axis=1)==1]\n",
    "\n",
    "    h5file.close()\n",
    "    return feature_array, label_array\n",
    "\n",
    "# load training file\n",
    "train_filename = 'ntuple_merged_10.h5'\n",
    "if not os.path.isfile(train_filename):\n",
    "     !wget http://opendata.cern.ch/eos/opendata/cms/datascience/HiggsToBBNtupleProducerTool/HiggsToBBNTuple_HiggsToBB_QCD_RunII_13TeV_MC/train/ntuple_merged_10.h5 .\n",
    "feature_array, label_array = get_features_labels(train_filename, remove_mass_pt_window=False)\n",
    "print(feature_array.shape, label_array.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Decision Tree Classifier**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import tree\n",
    "clf = tree.DecisionTreeClassifier(max_depth=5)\n",
    "clf = clf.fit(feature_array, label_array[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Support Vector Machine Classifier**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import linear_model\n",
    "svm = linear_model.SGDClassifier()\n",
    "svm.fit(feature_array, label_array[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Fully Connected Neural Network Classifier**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense, BatchNormalization\n",
    "from keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau\n",
    "\n",
    "# define dense keras model\n",
    "inputs = Input(shape=(nfeatures,), name = 'input')  \n",
    "x = BatchNormalization(name='bn_1')(inputs)\n",
    "x = Dense(64, name = 'dense_1', activation='relu')(x)\n",
    "x = Dense(32, name = 'dense_2', activation='relu')(x)\n",
    "x = Dense(32, name = 'dense_3', activation='relu')(x)\n",
    "outputs = Dense(nlabels, name = 'output', activation='softmax')(x)\n",
    "keras_model = Model(inputs=inputs, outputs=outputs)\n",
    "keras_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "print(\"Define dense keras model:\")\n",
    "print(keras_model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Train FCNN model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define callbacks\n",
    "early_stopping = EarlyStopping(monitor='val_loss', patience=20)\n",
    "reduce_lr = ReduceLROnPlateau(patience=5,factor=0.5)\n",
    "model_checkpoint = ModelCheckpoint('keras_model_best.h5', monitor='val_loss', save_best_only=True)\n",
    "callbacks = [early_stopping, model_checkpoint, reduce_lr]\n",
    "\n",
    "# fit keras model\n",
    "ep = 100 # epoch\n",
    "history  = keras_model.fit(feature_array, label_array, batch_size=1024, \n",
    "                epochs=ep, validation_split=0.2, shuffle=False,\n",
    "                callbacks = callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.plot(history.history['loss'],label='Loss')\n",
    "plt.plot(history.history['val_loss'],label='Val. loss')\n",
    "plt.xlabel('Epoch')\n",
    "plt.legend()\n",
    "plt.title(\"Loss vs Epoch during training on ntuple_merged_0.h5\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Test and Compare All 3 Models**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-05-04 14:02:02--  http://opendata.cern.ch/eos/opendata/cms/datascience/HiggsToBBNtupleProducerTool/HiggsToBBNTuple_HiggsToBB_QCD_RunII_13TeV_MC/test/ntuple_merged_0.h5\n",
      "Resolving opendata.cern.ch (opendata.cern.ch)... 188.184.28.138, 188.184.93.89, 188.185.82.144\n",
      "Connecting to opendata.cern.ch (opendata.cern.ch)|188.184.28.138|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1506140388 (1.4G) [application/octet-stream]\n",
      "Saving to: ‘ntuple_merged_0.h5’\n",
      "\n",
      "ntuple_merged_0.h5    0%[                    ]   8.01M  46.4KB/s    eta 3h 47m ^C\n"
     ]
    }
   ],
   "source": [
    "test_filename = 'ntuple_merged_0.h5'\n",
    "if not os.path.isfile(test_filename): \n",
    "    !wget http://opendata.cern.ch/eos/opendata/cms/datascience/HiggsToBBNtupleProducerTool/HiggsToBBNTuple_HiggsToBB_QCD_RunII_13TeV_MC/test/ntuple_merged_0.h5 ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load training file\n",
    "feature_array_test, label_array_test = get_features_labels(test_filename, remove_mass_pt_window=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test on file ntuple_merged_11.h5\n",
      "187653/187653 [==============================] - 2s 10us/step\n",
      "Neural network:\n",
      " Test_loss: 0.219699, Test_accuracy: 0.915221\n",
      "Decision Tree Classifier \n",
      " Test_accuracy: 0.902522\n",
      "SVM \n",
      " Test_accuracy: 0.889445\n"
     ]
    }
   ],
   "source": [
    "print(\"Test on file %s\" %test_filename)\n",
    "# run model inference on test data set\n",
    "predict_array_nn = np.argmax(keras_model.predict(feature_array_test),axis=1)\n",
    "predict_array_tree = np.argmax(clf.predict_proba(feature_array_test),axis=1)\n",
    "predict_array_svm = svm.decision_function(feature_array_test)\n",
    "predict_array_svm = (predict_array_svm>0).astype(int)\n",
    "\n",
    "# print test accuracy\n",
    "[nn_test_loss, nn_test_accuracy] = keras_model.evaluate(feature_array_test, label_array_test)\n",
    "print(\"Neural network:\\n Test_loss: %f, Test_accuracy: %f\" %(nn_test_loss, nn_test_accuracy))\n",
    "\n",
    "acc_tree = np.sum((predict_array_tree==label_array_test[:,1]).astype(int))/len(predict_array_tree)\n",
    "print(\"Decision Tree Classifier \\n Test_accuracy: %f\"%acc_tree )\n",
    "\n",
    "acc_svm = np.sum((predict_array_svm==label_array_test[:,1]).astype(int))/len(predict_array_svm)\n",
    "print(\"SVM \\n Test_accuracy: %f\"%acc_svm )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Regression: Energy Reconstruction \n",
    "\n",
    "In energy reconstruction example, the regression target is **fj_gen_pt**. It is the Transverse momentum of the generator-level, geometrically matched heavy particle: H, W, Z, t, etc. (default = -999)\n",
    "\n",
    "**Input**: \n",
    "    \n",
    "    features: dimension (n,27)\n",
    "    \n",
    "**Output**: \n",
    "    \n",
    "    fj_gen_pt: (n,1)\n",
    "    \n",
    "**Model**: \n",
    "    \n",
    "    Fully connected nueral network\n",
    "    \n",
    "In the following example, we use the same set of features as in the classification example to reconstruct the **fj_gen_pt** we target at.\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "gen_pts = ['fj_gen_pt']\n",
    "\n",
    "def get_features_gen(file_name, remove_mass_pt_window=False):\n",
    "\n",
    "    # load file\n",
    "    h5file = tables.open_file(file_name, 'r')\n",
    "    njets = getattr(h5file.root,features[0]).shape[0]\n",
    "\n",
    "    # allocate arrays\n",
    "    feature_array = np.zeros((njets,nfeatures))\n",
    "    spec_array = np.zeros((njets,nspectators))\n",
    "    label_array = np.zeros((njets,nlabels))\n",
    "    gen_pt_array = np.zeros((njets,1))\n",
    "\n",
    "    # load feature arrays\n",
    "    for (i, feat) in enumerate(features):\n",
    "        feature_array[:,i] = getattr(h5file.root,feat)[:]\n",
    "\n",
    "    for gen in gen_pts:\n",
    "        gen_pt_array[:,0] = getattr(h5file.root,gen)[:]\n",
    "        \n",
    "    # load labels arrays\n",
    "    for (i, label) in enumerate(labels):\n",
    "        prods = label.split('*')\n",
    "        prod0 = prods[0]\n",
    "        prod1 = prods[1]\n",
    "        fact0 = getattr(h5file.root,prod0)[:]\n",
    "        fact1 = getattr(h5file.root,prod1)[:]\n",
    "        label_array[:,i] = np.multiply(fact0,fact1)\n",
    "\n",
    "    feature_array = feature_array[np.sum(label_array,axis=1)==1]\n",
    "    gen_pt_array = gen_pt_array[np.sum(label_array,axis=1)==1]\n",
    "    label_array = label_array[np.sum(label_array,axis=1)==1]\n",
    "\n",
    "    feature_array = feature_array[gen_pt_array[:,0]>=0]\n",
    "    label_array = label_array[gen_pt_array[:,0]>=0]\n",
    "    gen_pt_array = gen_pt_array[gen_pt_array[:,0]>=0]\n",
    "    \n",
    "    h5file.close()\n",
    "    return feature_array, label_array, gen_pt_array\n",
    "\n",
    "\n",
    "# load training file\n",
    "train_filename = 'ntuple_merged_11.h5'\n",
    "feature_array2, label_array2, gen_pt_array = get_features_gen(train_filename, remove_mass_pt_window=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Losses and Metrics\n",
    "def rmse(y_true, y_pred):\n",
    "    from keras import backend\n",
    "    return backend.sqrt(backend.mean(backend.square(y_pred - y_true), axis=-1))\n",
    "\n",
    "# mean squared error (mse) for regression  (only for Keras tensors)\n",
    "def mse(y_true, y_pred):\n",
    "    from keras import backend\n",
    "    return backend.mean(backend.square(y_pred - y_true), axis=-1)\n",
    "\n",
    "# coefficient of determination (R^2) for regression  (only for Keras tensors)\n",
    "def r_square(y_true, y_pred):\n",
    "    from keras import backend as K\n",
    "    SS_res =  K.sum(K.square(y_true - y_pred)) \n",
    "    SS_tot = K.sum(K.square(y_true - K.mean(y_true))) \n",
    "    return ( 1 - SS_res/(SS_tot + K.epsilon()) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Define dense keras model:\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input (InputLayer)           (None, 27)                0         \n",
      "_________________________________________________________________\n",
      "bn_1 (BatchNormalization)    (None, 27)                108       \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 64)                1792      \n",
      "_________________________________________________________________\n",
      "output (Dense)               (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 1,965\n",
      "Trainable params: 1,911\n",
      "Non-trainable params: 54\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "inputs = Input(shape=(nfeatures,), name = 'input')  \n",
    "x = BatchNormalization(name='bn_1')(inputs)\n",
    "x = Dense(64, kernel_initializer='he_uniform', activation='relu')(x)\n",
    "outputs = Dense(1, name = 'output')(x)\n",
    "keras_model = Model(inputs=inputs, outputs=outputs)\n",
    "keras_model.compile(optimizer='adam', loss=mse, metrics=[r_square, rmse])\n",
    "print(\"Define dense keras model:\")\n",
    "print(keras_model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 19824 samples, validate on 4956 samples\n",
      "Epoch 1/300\n",
      "19824/19824 [==============================] - 0s 25us/step - loss: 1264881.6499 - r_square: -4.8171 - rmse: 1023.2984 - val_loss: 1266823.0690 - val_r_square: -4.9133 - val_rmse: 1025.7532\n",
      "Epoch 2/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 1261852.5013 - r_square: -4.8031 - rmse: 1021.8276 - val_loss: 1263659.1794 - val_r_square: -4.8986 - val_rmse: 1024.2299\n",
      "Epoch 3/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 1258434.1799 - r_square: -4.7874 - rmse: 1020.1701 - val_loss: 1259960.2968 - val_r_square: -4.8813 - val_rmse: 1022.4518\n",
      "Epoch 4/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 1254353.6929 - r_square: -4.7686 - rmse: 1018.1949 - val_loss: 1255459.5257 - val_r_square: -4.8603 - val_rmse: 1020.2927\n",
      "Epoch 5/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 1249348.2527 - r_square: -4.7456 - rmse: 1015.7760 - val_loss: 1249895.2522 - val_r_square: -4.8343 - val_rmse: 1017.6284\n",
      "Epoch 6/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 1243142.8711 - r_square: -4.7170 - rmse: 1012.7807 - val_loss: 1242968.2684 - val_r_square: -4.8020 - val_rmse: 1014.3157\n",
      "Epoch 7/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 1235406.6896 - r_square: -4.6814 - rmse: 1009.0481 - val_loss: 1234313.7101 - val_r_square: -4.7616 - val_rmse: 1010.1755\n",
      "Epoch 8/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 1225734.4892 - r_square: -4.6369 - rmse: 1004.3784 - val_loss: 1223478.1369 - val_r_square: -4.7110 - val_rmse: 1004.9832\n",
      "Epoch 9/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 1213613.3454 - r_square: -4.5811 - rmse: 998.5129 - val_loss: 1209870.0053 - val_r_square: -4.6475 - val_rmse: 998.4393\n",
      "Epoch 10/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 1198375.8187 - r_square: -4.5110 - rmse: 991.1077 - val_loss: 1192724.4449 - val_r_square: -4.5675 - val_rmse: 990.1437\n",
      "Epoch 11/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 1179144.7457 - r_square: -4.4225 - rmse: 981.6977 - val_loss: 1171045.9799 - val_r_square: -4.4663 - val_rmse: 979.5607\n",
      "Epoch 12/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 1154802.5528 - r_square: -4.3105 - rmse: 969.6656 - val_loss: 1143589.7877 - val_r_square: -4.3382 - val_rmse: 965.9876\n",
      "Epoch 13/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 1124081.2582 - r_square: -4.1691 - rmse: 954.2533 - val_loss: 1109093.1889 - val_r_square: -4.1772 - val_rmse: 948.6169\n",
      "Epoch 14/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 1085713.3817 - r_square: -3.9925 - rmse: 934.5976 - val_loss: 1066288.8477 - val_r_square: -3.9774 - val_rmse: 926.5324\n",
      "Epoch 15/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 1038733.9536 - r_square: -3.7764 - rmse: 909.8422 - val_loss: 1014623.5659 - val_r_square: -3.7363 - val_rmse: 898.9901\n",
      "Epoch 16/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 982929.2553 - r_square: -3.5196 - rmse: 879.3406 - val_loss: 954169.3646 - val_r_square: -3.4542 - val_rmse: 865.4201\n",
      "Epoch 17/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 918715.7245 - r_square: -3.2242 - rmse: 842.6410 - val_loss: 885711.4477 - val_r_square: -3.1347 - val_rmse: 825.4815\n",
      "Epoch 18/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 847252.3776 - r_square: -2.8954 - rmse: 799.6749 - val_loss: 810741.0981 - val_r_square: -2.7848 - val_rmse: 779.2119\n",
      "Epoch 19/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 770401.7311 - r_square: -2.5419 - rmse: 750.9530 - val_loss: 731544.8306 - val_r_square: -2.4151 - val_rmse: 727.7491\n",
      "Epoch 20/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 690859.8178 - r_square: -2.1760 - rmse: 698.2203 - val_loss: 651235.1291 - val_r_square: -2.0403 - val_rmse: 673.6933\n",
      "Epoch 21/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 611947.3072 - r_square: -1.8131 - rmse: 644.3519 - val_loss: 573271.1054 - val_r_square: -1.6764 - val_rmse: 619.8718\n",
      "Epoch 22/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 537100.1026 - r_square: -1.4688 - rmse: 592.0441 - val_loss: 500970.1903 - val_r_square: -1.3389 - val_rmse: 569.1183\n",
      "Epoch 23/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 469445.3621 - r_square: -1.1577 - rmse: 543.5656 - val_loss: 437195.7364 - val_r_square: -1.0412 - val_rmse: 523.2298\n",
      "Epoch 24/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 411349.7351 - r_square: -0.8906 - rmse: 500.8611 - val_loss: 383795.7298 - val_r_square: -0.7919 - val_rmse: 483.5141\n",
      "Epoch 25/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 364013.8299 - r_square: -0.6730 - rmse: 466.5154 - val_loss: 341365.8157 - val_r_square: -0.5938 - val_rmse: 452.2070\n",
      "Epoch 26/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 327440.3780 - r_square: -0.5049 - rmse: 441.0080 - val_loss: 309340.3375 - val_r_square: -0.4442 - val_rmse: 429.6099\n",
      "Epoch 27/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 300513.5780 - r_square: -0.3811 - rmse: 423.2959 - val_loss: 286162.9061 - val_r_square: -0.3359 - val_rmse: 414.3091\n",
      "Epoch 28/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 281385.4232 - r_square: -0.2932 - rmse: 411.3286 - val_loss: 269833.9480 - val_r_square: -0.2596 - val_rmse: 403.9129\n",
      "Epoch 29/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 268022.0992 - r_square: -0.2318 - rmse: 403.5788 - val_loss: 258376.3125 - val_r_square: -0.2060 - val_rmse: 396.9247\n",
      "Epoch 30/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 258615.2187 - r_square: -0.1886 - rmse: 398.4319 - val_loss: 250185.4271 - val_r_square: -0.1677 - val_rmse: 392.1967\n",
      "Epoch 31/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 251775.8861 - r_square: -0.1572 - rmse: 394.8277 - val_loss: 244096.7912 - val_r_square: -0.1392 - val_rmse: 388.7106\n",
      "Epoch 32/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 246571.6063 - r_square: -0.1333 - rmse: 392.0838 - val_loss: 239349.4408 - val_r_square: -0.1170 - val_rmse: 385.9603\n",
      "Epoch 33/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 242409.5407 - r_square: -0.1141 - rmse: 389.8228 - val_loss: 235483.9653 - val_r_square: -0.0989 - val_rmse: 383.6302\n",
      "Epoch 34/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 238940.6844 - r_square: -0.0982 - rmse: 387.8457 - val_loss: 232220.8866 - val_r_square: -0.0837 - val_rmse: 381.5600\n",
      "Epoch 35/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 235965.1052 - r_square: -0.0845 - rmse: 386.0561 - val_loss: 229391.9745 - val_r_square: -0.0704 - val_rmse: 379.7000\n",
      "Epoch 36/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 233358.9942 - r_square: -0.0725 - rmse: 384.4226 - val_loss: 226896.4540 - val_r_square: -0.0587 - val_rmse: 378.0315\n",
      "Epoch 37/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 231044.2045 - r_square: -0.0619 - rmse: 382.9255 - val_loss: 224670.9519 - val_r_square: -0.0483 - val_rmse: 376.4985\n",
      "Epoch 38/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 228970.6248 - r_square: -0.0523 - rmse: 381.5420 - val_loss: 222669.6764 - val_r_square: -0.0390 - val_rmse: 375.0838\n",
      "Epoch 39/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 227100.1643 - r_square: -0.0437 - rmse: 380.2654 - val_loss: 220859.1709 - val_r_square: -0.0305 - val_rmse: 373.8257\n",
      "Epoch 40/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 225403.3760 - r_square: -0.0359 - rmse: 379.0973 - val_loss: 219215.0284 - val_r_square: -0.0228 - val_rmse: 372.6880\n",
      "Epoch 41/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 223856.6494 - r_square: -0.0288 - rmse: 378.0268 - val_loss: 217712.9222 - val_r_square: -0.0158 - val_rmse: 371.6415\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 42/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 222442.8662 - r_square: -0.0223 - rmse: 377.0398 - val_loss: 216335.2309 - val_r_square: -0.0093 - val_rmse: 370.6843\n",
      "Epoch 43/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 221144.7966 - r_square: -0.0163 - rmse: 376.1347 - val_loss: 215069.0489 - val_r_square: -0.0034 - val_rmse: 369.8082\n",
      "Epoch 44/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 219948.6643 - r_square: -0.0108 - rmse: 375.3064 - val_loss: 213899.7148 - val_r_square: 0.0021 - val_rmse: 368.9916\n",
      "Epoch 45/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 218843.8016 - r_square: -0.0057 - rmse: 374.5359 - val_loss: 212818.7993 - val_r_square: 0.0071 - val_rmse: 368.2317\n",
      "Epoch 46/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 217820.6498 - r_square: -0.0010 - rmse: 373.8146 - val_loss: 211818.6226 - val_r_square: 0.0118 - val_rmse: 367.5407\n",
      "Epoch 47/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 216870.9686 - r_square: 0.0033 - rmse: 373.1408 - val_loss: 210889.3515 - val_r_square: 0.0162 - val_rmse: 366.8905\n",
      "Epoch 48/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 215986.4166 - r_square: 0.0074 - rmse: 372.5095 - val_loss: 210024.3909 - val_r_square: 0.0202 - val_rmse: 366.2746\n",
      "Epoch 49/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 215161.1866 - r_square: 0.0112 - rmse: 371.9139 - val_loss: 209218.2767 - val_r_square: 0.0240 - val_rmse: 365.6946\n",
      "Epoch 50/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 214388.4945 - r_square: 0.0148 - rmse: 371.3572 - val_loss: 208463.2322 - val_r_square: 0.0275 - val_rmse: 365.1556\n",
      "Epoch 51/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 213663.2882 - r_square: 0.0181 - rmse: 370.8410 - val_loss: 207755.9869 - val_r_square: 0.0308 - val_rmse: 364.6491\n",
      "Epoch 52/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 212981.2218 - r_square: 0.0212 - rmse: 370.3546 - val_loss: 207093.1497 - val_r_square: 0.0339 - val_rmse: 364.1751\n",
      "Epoch 53/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 212337.9226 - r_square: 0.0242 - rmse: 369.8940 - val_loss: 206469.5591 - val_r_square: 0.0368 - val_rmse: 363.7233\n",
      "Epoch 54/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 211730.4956 - r_square: 0.0270 - rmse: 369.4593 - val_loss: 205882.2047 - val_r_square: 0.0396 - val_rmse: 363.2915\n",
      "Epoch 55/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 211156.4075 - r_square: 0.0296 - rmse: 369.0447 - val_loss: 205328.0229 - val_r_square: 0.0422 - val_rmse: 362.8806\n",
      "Epoch 56/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 210612.5489 - r_square: 0.0321 - rmse: 368.6508 - val_loss: 204804.1012 - val_r_square: 0.0446 - val_rmse: 362.4990\n",
      "Epoch 57/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 210096.6216 - r_square: 0.0345 - rmse: 368.2721 - val_loss: 204308.5592 - val_r_square: 0.0470 - val_rmse: 362.1366\n",
      "Epoch 58/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 209606.2114 - r_square: 0.0368 - rmse: 367.9132 - val_loss: 203839.0445 - val_r_square: 0.0491 - val_rmse: 361.7920\n",
      "Epoch 59/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 209138.8168 - r_square: 0.0389 - rmse: 367.5721 - val_loss: 203393.6116 - val_r_square: 0.0512 - val_rmse: 361.4604\n",
      "Epoch 60/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 208693.2098 - r_square: 0.0410 - rmse: 367.2488 - val_loss: 202970.5516 - val_r_square: 0.0532 - val_rmse: 361.1415\n",
      "Epoch 61/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 208267.4475 - r_square: 0.0429 - rmse: 366.9418 - val_loss: 202567.8504 - val_r_square: 0.0551 - val_rmse: 360.8377\n",
      "Epoch 62/300\n",
      "19824/19824 [==============================] - 0s 4us/step - loss: 207859.8927 - r_square: 0.0448 - rmse: 366.6468 - val_loss: 202183.5938 - val_r_square: 0.0569 - val_rmse: 360.5445\n",
      "Epoch 63/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 207469.1944 - r_square: 0.0466 - rmse: 366.3633 - val_loss: 201817.2758 - val_r_square: 0.0586 - val_rmse: 360.2615\n",
      "Epoch 64/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 207094.5070 - r_square: 0.0483 - rmse: 366.0904 - val_loss: 201467.4413 - val_r_square: 0.0602 - val_rmse: 359.9877\n",
      "Epoch 65/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 206734.9931 - r_square: 0.0500 - rmse: 365.8262 - val_loss: 201133.1035 - val_r_square: 0.0618 - val_rmse: 359.7216\n",
      "Epoch 66/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 206389.7789 - r_square: 0.0515 - rmse: 365.5738 - val_loss: 200813.4198 - val_r_square: 0.0633 - val_rmse: 359.4625\n",
      "Epoch 67/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 206058.3576 - r_square: 0.0531 - rmse: 365.3316 - val_loss: 200507.7246 - val_r_square: 0.0647 - val_rmse: 359.2151\n",
      "Epoch 68/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 205740.0126 - r_square: 0.0545 - rmse: 365.0991 - val_loss: 200215.7922 - val_r_square: 0.0661 - val_rmse: 358.9805\n",
      "Epoch 69/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 205433.8455 - r_square: 0.0559 - rmse: 364.8746 - val_loss: 199936.5787 - val_r_square: 0.0674 - val_rmse: 358.7552\n",
      "Epoch 70/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 205139.3092 - r_square: 0.0573 - rmse: 364.6578 - val_loss: 199669.0977 - val_r_square: 0.0686 - val_rmse: 358.5400\n",
      "Epoch 71/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 204855.7677 - r_square: 0.0586 - rmse: 364.4469 - val_loss: 199412.8049 - val_r_square: 0.0698 - val_rmse: 358.3303\n",
      "Epoch 72/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 204582.4705 - r_square: 0.0599 - rmse: 364.2414 - val_loss: 199166.6338 - val_r_square: 0.0710 - val_rmse: 358.1297\n",
      "Epoch 73/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 204318.8402 - r_square: 0.0611 - rmse: 364.0416 - val_loss: 198929.6698 - val_r_square: 0.0721 - val_rmse: 357.9391\n",
      "Epoch 74/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 204064.5274 - r_square: 0.0622 - rmse: 363.8491 - val_loss: 198701.2119 - val_r_square: 0.0732 - val_rmse: 357.7565\n",
      "Epoch 75/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 203818.4504 - r_square: 0.0634 - rmse: 363.6630 - val_loss: 198481.2830 - val_r_square: 0.0742 - val_rmse: 357.5800\n",
      "Epoch 76/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 203580.3524 - r_square: 0.0645 - rmse: 363.4817 - val_loss: 198270.0309 - val_r_square: 0.0752 - val_rmse: 357.4129\n",
      "Epoch 77/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 203350.2680 - r_square: 0.0655 - rmse: 363.3061 - val_loss: 198066.4950 - val_r_square: 0.0761 - val_rmse: 357.2519\n",
      "Epoch 78/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 203127.2680 - r_square: 0.0666 - rmse: 363.1369 - val_loss: 197870.0585 - val_r_square: 0.0771 - val_rmse: 357.0987\n",
      "Epoch 79/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 202911.3092 - r_square: 0.0675 - rmse: 362.9713 - val_loss: 197680.3817 - val_r_square: 0.0779 - val_rmse: 356.9497\n",
      "Epoch 80/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 202701.8589 - r_square: 0.0685 - rmse: 362.8075 - val_loss: 197497.4916 - val_r_square: 0.0788 - val_rmse: 356.8043\n",
      "Epoch 81/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 202499.1107 - r_square: 0.0694 - rmse: 362.6481 - val_loss: 197320.5260 - val_r_square: 0.0796 - val_rmse: 356.6640\n",
      "Epoch 82/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 202302.6652 - r_square: 0.0703 - rmse: 362.4930 - val_loss: 197150.0759 - val_r_square: 0.0804 - val_rmse: 356.5268\n",
      "Epoch 83/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 202112.0937 - r_square: 0.0712 - rmse: 362.3429 - val_loss: 196985.6585 - val_r_square: 0.0812 - val_rmse: 356.3961\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 84/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 201927.2309 - r_square: 0.0721 - rmse: 362.1972 - val_loss: 196827.2966 - val_r_square: 0.0819 - val_rmse: 356.2726\n",
      "Epoch 85/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 201747.8334 - r_square: 0.0729 - rmse: 362.0539 - val_loss: 196673.8698 - val_r_square: 0.0826 - val_rmse: 356.1537\n",
      "Epoch 86/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 201573.6224 - r_square: 0.0737 - rmse: 361.9151 - val_loss: 196525.4057 - val_r_square: 0.0833 - val_rmse: 356.0376\n",
      "Epoch 87/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 201404.7546 - r_square: 0.0745 - rmse: 361.7797 - val_loss: 196382.2781 - val_r_square: 0.0840 - val_rmse: 355.9251\n",
      "Epoch 88/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 201240.9084 - r_square: 0.0752 - rmse: 361.6472 - val_loss: 196243.9157 - val_r_square: 0.0846 - val_rmse: 355.8152\n",
      "Epoch 89/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 201081.9347 - r_square: 0.0760 - rmse: 361.5161 - val_loss: 196110.4891 - val_r_square: 0.0853 - val_rmse: 355.7063\n",
      "Epoch 90/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 200927.6296 - r_square: 0.0767 - rmse: 361.3889 - val_loss: 195981.4031 - val_r_square: 0.0859 - val_rmse: 355.6026\n",
      "Epoch 91/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 200777.7890 - r_square: 0.0774 - rmse: 361.2654 - val_loss: 195856.6676 - val_r_square: 0.0865 - val_rmse: 355.5010\n",
      "Epoch 92/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 200631.9528 - r_square: 0.0780 - rmse: 361.1453 - val_loss: 195735.7366 - val_r_square: 0.0870 - val_rmse: 355.4027\n",
      "Epoch 93/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 200490.1547 - r_square: 0.0787 - rmse: 361.0290 - val_loss: 195618.6444 - val_r_square: 0.0876 - val_rmse: 355.3107\n",
      "Epoch 94/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 200352.1972 - r_square: 0.0793 - rmse: 360.9149 - val_loss: 195504.9512 - val_r_square: 0.0881 - val_rmse: 355.2222\n",
      "Epoch 95/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 200218.0249 - r_square: 0.0799 - rmse: 360.8037 - val_loss: 195394.8220 - val_r_square: 0.0886 - val_rmse: 355.1358\n",
      "Epoch 96/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 200087.3644 - r_square: 0.0805 - rmse: 360.6961 - val_loss: 195287.9981 - val_r_square: 0.0891 - val_rmse: 355.0522\n",
      "Epoch 97/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 199959.9598 - r_square: 0.0811 - rmse: 360.5909 - val_loss: 195184.3477 - val_r_square: 0.0896 - val_rmse: 354.9707\n",
      "Epoch 98/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 199835.7821 - r_square: 0.0817 - rmse: 360.4887 - val_loss: 195083.8715 - val_r_square: 0.0901 - val_rmse: 354.8890\n",
      "Epoch 99/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 199714.7289 - r_square: 0.0822 - rmse: 360.3875 - val_loss: 194986.2964 - val_r_square: 0.0905 - val_rmse: 354.8079\n",
      "Epoch 100/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 199596.4531 - r_square: 0.0828 - rmse: 360.2890 - val_loss: 194891.8619 - val_r_square: 0.0910 - val_rmse: 354.7294\n",
      "Epoch 101/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 199480.8883 - r_square: 0.0833 - rmse: 360.1948 - val_loss: 194799.9332 - val_r_square: 0.0914 - val_rmse: 354.6531\n",
      "Epoch 102/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 199368.0128 - r_square: 0.0838 - rmse: 360.1017 - val_loss: 194711.0631 - val_r_square: 0.0918 - val_rmse: 354.5769\n",
      "Epoch 103/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 199257.7408 - r_square: 0.0844 - rmse: 360.0086 - val_loss: 194624.7631 - val_r_square: 0.0922 - val_rmse: 354.5040\n",
      "Epoch 104/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 199149.9662 - r_square: 0.0848 - rmse: 359.9159 - val_loss: 194540.8450 - val_r_square: 0.0926 - val_rmse: 354.4328\n",
      "Epoch 105/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 199044.4438 - r_square: 0.0853 - rmse: 359.8258 - val_loss: 194458.8894 - val_r_square: 0.0930 - val_rmse: 354.3648\n",
      "Epoch 106/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 198941.2485 - r_square: 0.0858 - rmse: 359.7389 - val_loss: 194378.2215 - val_r_square: 0.0934 - val_rmse: 354.3004\n",
      "Epoch 107/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 198840.1656 - r_square: 0.0863 - rmse: 359.6533 - val_loss: 194299.0572 - val_r_square: 0.0937 - val_rmse: 354.2378\n",
      "Epoch 108/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 198741.1032 - r_square: 0.0867 - rmse: 359.5682 - val_loss: 194221.8401 - val_r_square: 0.0941 - val_rmse: 354.1757\n",
      "Epoch 109/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 198643.8399 - r_square: 0.0872 - rmse: 359.4832 - val_loss: 194146.4739 - val_r_square: 0.0944 - val_rmse: 354.1133\n",
      "Epoch 110/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 198548.3369 - r_square: 0.0876 - rmse: 359.3999 - val_loss: 194072.7308 - val_r_square: 0.0948 - val_rmse: 354.0560\n",
      "Epoch 111/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 198454.5213 - r_square: 0.0880 - rmse: 359.3179 - val_loss: 194000.4566 - val_r_square: 0.0951 - val_rmse: 353.9993\n",
      "Epoch 112/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 198362.2693 - r_square: 0.0885 - rmse: 359.2370 - val_loss: 193929.5367 - val_r_square: 0.0955 - val_rmse: 353.9420\n",
      "Epoch 113/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 198271.4656 - r_square: 0.0889 - rmse: 359.1575 - val_loss: 193859.9407 - val_r_square: 0.0958 - val_rmse: 353.8852\n",
      "Epoch 114/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 198182.1720 - r_square: 0.0893 - rmse: 359.0793 - val_loss: 193791.7570 - val_r_square: 0.0961 - val_rmse: 353.8299\n",
      "Epoch 115/300\n",
      "19824/19824 [==============================] - 0s 4us/step - loss: 198094.2850 - r_square: 0.0897 - rmse: 359.0017 - val_loss: 193724.6867 - val_r_square: 0.0964 - val_rmse: 353.7774\n",
      "Epoch 116/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 198007.7727 - r_square: 0.0901 - rmse: 358.9256 - val_loss: 193657.8430 - val_r_square: 0.0967 - val_rmse: 353.7268\n",
      "Epoch 117/300\n",
      "19824/19824 [==============================] - 0s 4us/step - loss: 197922.5798 - r_square: 0.0905 - rmse: 358.8502 - val_loss: 193591.8028 - val_r_square: 0.0970 - val_rmse: 353.6751\n",
      "Epoch 118/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 197838.5285 - r_square: 0.0909 - rmse: 358.7745 - val_loss: 193526.5581 - val_r_square: 0.0973 - val_rmse: 353.6241\n",
      "Epoch 119/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 197755.5211 - r_square: 0.0913 - rmse: 358.6994 - val_loss: 193461.8072 - val_r_square: 0.0976 - val_rmse: 353.5737\n",
      "Epoch 120/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 197673.4952 - r_square: 0.0916 - rmse: 358.6249 - val_loss: 193397.7085 - val_r_square: 0.0979 - val_rmse: 353.5226\n",
      "Epoch 121/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 197592.2556 - r_square: 0.0920 - rmse: 358.5504 - val_loss: 193333.4391 - val_r_square: 0.0982 - val_rmse: 353.4720\n",
      "Epoch 122/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 197511.9127 - r_square: 0.0924 - rmse: 358.4778 - val_loss: 193269.7537 - val_r_square: 0.0985 - val_rmse: 353.4214\n",
      "Epoch 123/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 197432.4218 - r_square: 0.0927 - rmse: 358.4049 - val_loss: 193206.8023 - val_r_square: 0.0988 - val_rmse: 353.3699\n",
      "Epoch 124/300\n",
      "19824/19824 [==============================] - 0s 4us/step - loss: 197353.7072 - r_square: 0.0931 - rmse: 358.3309 - val_loss: 193144.5525 - val_r_square: 0.0991 - val_rmse: 353.3197\n",
      "Epoch 125/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 197275.4545 - r_square: 0.0935 - rmse: 358.2562 - val_loss: 193082.5373 - val_r_square: 0.0994 - val_rmse: 353.2707\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 126/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 197197.8139 - r_square: 0.0938 - rmse: 358.1823 - val_loss: 193020.8839 - val_r_square: 0.0997 - val_rmse: 353.2216\n",
      "Epoch 127/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 197120.7441 - r_square: 0.0942 - rmse: 358.1087 - val_loss: 192959.5518 - val_r_square: 0.1000 - val_rmse: 353.1729\n",
      "Epoch 128/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 197044.1422 - r_square: 0.0945 - rmse: 358.0353 - val_loss: 192898.1562 - val_r_square: 0.1003 - val_rmse: 353.1234\n",
      "Epoch 129/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 196967.7869 - r_square: 0.0949 - rmse: 357.9614 - val_loss: 192836.8223 - val_r_square: 0.1006 - val_rmse: 353.0752\n",
      "Epoch 130/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 196891.8841 - r_square: 0.0952 - rmse: 357.8890 - val_loss: 192775.5547 - val_r_square: 0.1008 - val_rmse: 353.0273\n",
      "Epoch 131/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 196816.4000 - r_square: 0.0956 - rmse: 357.8154 - val_loss: 192714.3123 - val_r_square: 0.1011 - val_rmse: 352.9779\n",
      "Epoch 132/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 196741.1843 - r_square: 0.0959 - rmse: 357.7408 - val_loss: 192653.0932 - val_r_square: 0.1014 - val_rmse: 352.9292\n",
      "Epoch 133/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 196666.0315 - r_square: 0.0963 - rmse: 357.6663 - val_loss: 192591.9619 - val_r_square: 0.1017 - val_rmse: 352.8798\n",
      "Epoch 134/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 196590.9202 - r_square: 0.0966 - rmse: 357.5913 - val_loss: 192530.6902 - val_r_square: 0.1020 - val_rmse: 352.8296\n",
      "Epoch 135/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 196515.9121 - r_square: 0.0970 - rmse: 357.5163 - val_loss: 192469.3324 - val_r_square: 0.1023 - val_rmse: 352.7793\n",
      "Epoch 136/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 196441.0782 - r_square: 0.0973 - rmse: 357.4421 - val_loss: 192407.6195 - val_r_square: 0.1026 - val_rmse: 352.7285\n",
      "Epoch 137/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 196366.4892 - r_square: 0.0976 - rmse: 357.3677 - val_loss: 192345.9717 - val_r_square: 0.1029 - val_rmse: 352.6767\n",
      "Epoch 138/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 196292.1313 - r_square: 0.0980 - rmse: 357.2934 - val_loss: 192283.8311 - val_r_square: 0.1031 - val_rmse: 352.6253\n",
      "Epoch 139/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 196217.8507 - r_square: 0.0983 - rmse: 357.2195 - val_loss: 192220.9969 - val_r_square: 0.1034 - val_rmse: 352.5744\n",
      "Epoch 140/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 196143.4871 - r_square: 0.0987 - rmse: 357.1457 - val_loss: 192157.8135 - val_r_square: 0.1037 - val_rmse: 352.5235\n",
      "Epoch 141/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 196069.1379 - r_square: 0.0990 - rmse: 357.0698 - val_loss: 192094.3174 - val_r_square: 0.1040 - val_rmse: 352.4705\n",
      "Epoch 142/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 195994.9068 - r_square: 0.0994 - rmse: 356.9937 - val_loss: 192030.7931 - val_r_square: 0.1043 - val_rmse: 352.4165\n",
      "Epoch 143/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 195920.6557 - r_square: 0.0997 - rmse: 356.9162 - val_loss: 191966.5804 - val_r_square: 0.1046 - val_rmse: 352.3619\n",
      "Epoch 144/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 195846.3669 - r_square: 0.1000 - rmse: 356.8391 - val_loss: 191902.0522 - val_r_square: 0.1049 - val_rmse: 352.3072\n",
      "Epoch 145/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 195772.0264 - r_square: 0.1004 - rmse: 356.7626 - val_loss: 191837.3830 - val_r_square: 0.1052 - val_rmse: 352.2520\n",
      "Epoch 146/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 195697.5613 - r_square: 0.1007 - rmse: 356.6857 - val_loss: 191772.3444 - val_r_square: 0.1055 - val_rmse: 352.1958\n",
      "Epoch 147/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 195622.9505 - r_square: 0.1011 - rmse: 356.6086 - val_loss: 191706.8260 - val_r_square: 0.1058 - val_rmse: 352.1386\n",
      "Epoch 148/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 195548.0456 - r_square: 0.1014 - rmse: 356.5307 - val_loss: 191640.5424 - val_r_square: 0.1061 - val_rmse: 352.0798\n",
      "Epoch 149/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 195472.7204 - r_square: 0.1018 - rmse: 356.4522 - val_loss: 191573.4877 - val_r_square: 0.1065 - val_rmse: 352.0205\n",
      "Epoch 150/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 195397.1543 - r_square: 0.1021 - rmse: 356.3744 - val_loss: 191506.0718 - val_r_square: 0.1068 - val_rmse: 351.9616\n",
      "Epoch 151/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 195321.4379 - r_square: 0.1025 - rmse: 356.2968 - val_loss: 191438.2502 - val_r_square: 0.1071 - val_rmse: 351.9011\n",
      "Epoch 152/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 195245.4738 - r_square: 0.1028 - rmse: 356.2182 - val_loss: 191369.6880 - val_r_square: 0.1074 - val_rmse: 351.8397\n",
      "Epoch 153/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 195169.2268 - r_square: 0.1032 - rmse: 356.1399 - val_loss: 191300.1451 - val_r_square: 0.1077 - val_rmse: 351.7769\n",
      "Epoch 154/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 195092.5583 - r_square: 0.1035 - rmse: 356.0608 - val_loss: 191229.7940 - val_r_square: 0.1081 - val_rmse: 351.7131\n",
      "Epoch 155/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 195015.4857 - r_square: 0.1039 - rmse: 355.9814 - val_loss: 191158.1784 - val_r_square: 0.1084 - val_rmse: 351.6488\n",
      "Epoch 156/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 194937.9540 - r_square: 0.1042 - rmse: 355.9011 - val_loss: 191085.7533 - val_r_square: 0.1087 - val_rmse: 351.5844\n",
      "Epoch 157/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 194860.0209 - r_square: 0.1046 - rmse: 355.8209 - val_loss: 191012.7108 - val_r_square: 0.1091 - val_rmse: 351.5184\n",
      "Epoch 158/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 194781.6216 - r_square: 0.1049 - rmse: 355.7389 - val_loss: 190938.9172 - val_r_square: 0.1094 - val_rmse: 351.4513\n",
      "Epoch 159/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 194702.5149 - r_square: 0.1053 - rmse: 355.6562 - val_loss: 190864.3656 - val_r_square: 0.1098 - val_rmse: 351.3834\n",
      "Epoch 160/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 194622.6848 - r_square: 0.1057 - rmse: 355.5735 - val_loss: 190788.8614 - val_r_square: 0.1101 - val_rmse: 351.3147\n",
      "Epoch 161/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 194542.2704 - r_square: 0.1060 - rmse: 355.4921 - val_loss: 190712.5292 - val_r_square: 0.1105 - val_rmse: 351.2472\n",
      "Epoch 162/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 194461.3873 - r_square: 0.1064 - rmse: 355.4104 - val_loss: 190635.6903 - val_r_square: 0.1108 - val_rmse: 351.1784\n",
      "Epoch 163/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 194380.0159 - r_square: 0.1068 - rmse: 355.3276 - val_loss: 190557.4267 - val_r_square: 0.1112 - val_rmse: 351.1087\n",
      "Epoch 164/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 194298.1036 - r_square: 0.1072 - rmse: 355.2449 - val_loss: 190478.3106 - val_r_square: 0.1116 - val_rmse: 351.0379\n",
      "Epoch 165/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 194215.6945 - r_square: 0.1075 - rmse: 355.1617 - val_loss: 190398.5033 - val_r_square: 0.1120 - val_rmse: 350.9669\n",
      "Epoch 166/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 194132.6910 - r_square: 0.1079 - rmse: 355.0792 - val_loss: 190318.4212 - val_r_square: 0.1123 - val_rmse: 350.8965\n",
      "Epoch 167/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 194049.1710 - r_square: 0.1083 - rmse: 354.9953 - val_loss: 190236.9945 - val_r_square: 0.1127 - val_rmse: 350.8237\n",
      "Epoch 168/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 193965.1992 - r_square: 0.1087 - rmse: 354.9097 - val_loss: 190154.3103 - val_r_square: 0.1131 - val_rmse: 350.7479\n",
      "Epoch 169/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 193880.5047 - r_square: 0.1091 - rmse: 354.8227 - val_loss: 190070.6148 - val_r_square: 0.1135 - val_rmse: 350.6721\n",
      "Epoch 170/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 193795.1938 - r_square: 0.1095 - rmse: 354.7375 - val_loss: 189985.7776 - val_r_square: 0.1139 - val_rmse: 350.5952\n",
      "Epoch 171/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 193709.2067 - r_square: 0.1099 - rmse: 354.6494 - val_loss: 189899.6088 - val_r_square: 0.1143 - val_rmse: 350.5138\n",
      "Epoch 172/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 193622.5257 - r_square: 0.1103 - rmse: 354.5605 - val_loss: 189812.4715 - val_r_square: 0.1147 - val_rmse: 350.4325\n",
      "Epoch 173/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 193535.3467 - r_square: 0.1107 - rmse: 354.4712 - val_loss: 189724.3774 - val_r_square: 0.1151 - val_rmse: 350.3500\n",
      "Epoch 174/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 193447.5777 - r_square: 0.1111 - rmse: 354.3819 - val_loss: 189635.0704 - val_r_square: 0.1155 - val_rmse: 350.2676\n",
      "Epoch 175/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 193359.1781 - r_square: 0.1115 - rmse: 354.2922 - val_loss: 189545.0366 - val_r_square: 0.1159 - val_rmse: 350.1835\n",
      "Epoch 176/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 193270.1334 - r_square: 0.1119 - rmse: 354.1993 - val_loss: 189453.8521 - val_r_square: 0.1164 - val_rmse: 350.0967\n",
      "Epoch 177/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 193180.4471 - r_square: 0.1123 - rmse: 354.1052 - val_loss: 189361.7499 - val_r_square: 0.1168 - val_rmse: 350.0092\n",
      "Epoch 178/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 193090.3444 - r_square: 0.1127 - rmse: 354.0121 - val_loss: 189268.9201 - val_r_square: 0.1172 - val_rmse: 349.9216\n",
      "Epoch 179/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 192999.7684 - r_square: 0.1131 - rmse: 353.9178 - val_loss: 189175.1868 - val_r_square: 0.1177 - val_rmse: 349.8357\n",
      "Epoch 180/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 192908.5772 - r_square: 0.1136 - rmse: 353.8229 - val_loss: 189080.2877 - val_r_square: 0.1181 - val_rmse: 349.7499\n",
      "Epoch 181/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 192816.9198 - r_square: 0.1140 - rmse: 353.7278 - val_loss: 188984.3716 - val_r_square: 0.1186 - val_rmse: 349.6626\n",
      "Epoch 182/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 192724.7353 - r_square: 0.1144 - rmse: 353.6315 - val_loss: 188887.6005 - val_r_square: 0.1190 - val_rmse: 349.5734\n",
      "Epoch 183/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 192631.9345 - r_square: 0.1148 - rmse: 353.5348 - val_loss: 188789.5359 - val_r_square: 0.1195 - val_rmse: 349.4835\n",
      "Epoch 184/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 192538.5371 - r_square: 0.1153 - rmse: 353.4381 - val_loss: 188690.1945 - val_r_square: 0.1199 - val_rmse: 349.3913\n",
      "Epoch 185/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 192444.2721 - r_square: 0.1157 - rmse: 353.3394 - val_loss: 188589.6599 - val_r_square: 0.1204 - val_rmse: 349.2989\n",
      "Epoch 186/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 192349.4426 - r_square: 0.1161 - rmse: 353.2408 - val_loss: 188487.7649 - val_r_square: 0.1209 - val_rmse: 349.2057\n",
      "Epoch 187/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 192254.0960 - r_square: 0.1166 - rmse: 353.1408 - val_loss: 188384.8136 - val_r_square: 0.1214 - val_rmse: 349.1121\n",
      "Epoch 188/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 192158.3372 - r_square: 0.1170 - rmse: 353.0411 - val_loss: 188281.0451 - val_r_square: 0.1218 - val_rmse: 349.0180\n",
      "Epoch 189/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 192062.0919 - r_square: 0.1174 - rmse: 352.9404 - val_loss: 188176.3949 - val_r_square: 0.1223 - val_rmse: 348.9209\n",
      "Epoch 190/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 191965.1284 - r_square: 0.1179 - rmse: 352.8383 - val_loss: 188070.5064 - val_r_square: 0.1228 - val_rmse: 348.8228\n",
      "Epoch 191/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 191867.4637 - r_square: 0.1183 - rmse: 352.7369 - val_loss: 187963.5202 - val_r_square: 0.1233 - val_rmse: 348.7245\n",
      "Epoch 192/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 191769.3903 - r_square: 0.1188 - rmse: 352.6329 - val_loss: 187855.6166 - val_r_square: 0.1238 - val_rmse: 348.6224\n",
      "Epoch 193/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 191670.9124 - r_square: 0.1192 - rmse: 352.5286 - val_loss: 187747.1739 - val_r_square: 0.1243 - val_rmse: 348.5197\n",
      "Epoch 194/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 191572.0661 - r_square: 0.1197 - rmse: 352.4253 - val_loss: 187637.8149 - val_r_square: 0.1248 - val_rmse: 348.4168\n",
      "Epoch 195/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 191473.0221 - r_square: 0.1202 - rmse: 352.3226 - val_loss: 187527.6292 - val_r_square: 0.1254 - val_rmse: 348.3123\n",
      "Epoch 196/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 191373.6412 - r_square: 0.1206 - rmse: 352.2186 - val_loss: 187416.4337 - val_r_square: 0.1259 - val_rmse: 348.2066\n",
      "Epoch 197/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 191274.0530 - r_square: 0.1211 - rmse: 352.1133 - val_loss: 187304.9627 - val_r_square: 0.1264 - val_rmse: 348.0997\n",
      "Epoch 198/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 191174.1829 - r_square: 0.1215 - rmse: 352.0071 - val_loss: 187192.4948 - val_r_square: 0.1269 - val_rmse: 347.9931\n",
      "Epoch 199/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 191074.0850 - r_square: 0.1220 - rmse: 351.9017 - val_loss: 187079.3801 - val_r_square: 0.1275 - val_rmse: 347.8854\n",
      "Epoch 200/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 190973.6960 - r_square: 0.1224 - rmse: 351.7951 - val_loss: 186965.4269 - val_r_square: 0.1280 - val_rmse: 347.7769\n",
      "Epoch 201/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 190873.1348 - r_square: 0.1229 - rmse: 351.6879 - val_loss: 186851.2092 - val_r_square: 0.1285 - val_rmse: 347.6694\n",
      "Epoch 202/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 190772.5486 - r_square: 0.1234 - rmse: 351.5806 - val_loss: 186736.3839 - val_r_square: 0.1291 - val_rmse: 347.5665\n",
      "Epoch 203/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 190672.1149 - r_square: 0.1238 - rmse: 351.4749 - val_loss: 186621.6695 - val_r_square: 0.1296 - val_rmse: 347.4627\n",
      "Epoch 204/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 190571.7360 - r_square: 0.1243 - rmse: 351.3670 - val_loss: 186506.3832 - val_r_square: 0.1301 - val_rmse: 347.3572\n",
      "Epoch 205/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 190471.3854 - r_square: 0.1248 - rmse: 351.2583 - val_loss: 186390.3932 - val_r_square: 0.1307 - val_rmse: 347.2512\n",
      "Epoch 206/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 190371.0827 - r_square: 0.1252 - rmse: 351.1490 - val_loss: 186274.5965 - val_r_square: 0.1312 - val_rmse: 347.1455\n",
      "Epoch 207/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 190271.1311 - r_square: 0.1257 - rmse: 351.0416 - val_loss: 186158.5657 - val_r_square: 0.1318 - val_rmse: 347.0406\n",
      "Epoch 208/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 190171.6914 - r_square: 0.1261 - rmse: 350.9338 - val_loss: 186042.5347 - val_r_square: 0.1323 - val_rmse: 346.9341\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 209/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 190072.7418 - r_square: 0.1266 - rmse: 350.8265 - val_loss: 185926.4245 - val_r_square: 0.1328 - val_rmse: 346.8260\n",
      "Epoch 210/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 189974.2783 - r_square: 0.1270 - rmse: 350.7204 - val_loss: 185811.2544 - val_r_square: 0.1334 - val_rmse: 346.7181\n",
      "Epoch 211/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 189876.3116 - r_square: 0.1275 - rmse: 350.6148 - val_loss: 185696.1027 - val_r_square: 0.1339 - val_rmse: 346.6103\n",
      "Epoch 212/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 189778.7309 - r_square: 0.1279 - rmse: 350.5102 - val_loss: 185582.0919 - val_r_square: 0.1344 - val_rmse: 346.5049\n",
      "Epoch 213/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 189681.5231 - r_square: 0.1284 - rmse: 350.4068 - val_loss: 185467.4597 - val_r_square: 0.1350 - val_rmse: 346.4004\n",
      "Epoch 214/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 189584.9454 - r_square: 0.1288 - rmse: 350.3043 - val_loss: 185353.4846 - val_r_square: 0.1355 - val_rmse: 346.2956\n",
      "Epoch 215/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 189488.9480 - r_square: 0.1293 - rmse: 350.2009 - val_loss: 185239.7197 - val_r_square: 0.1360 - val_rmse: 346.1916\n",
      "Epoch 216/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 189393.6259 - r_square: 0.1297 - rmse: 350.0968 - val_loss: 185126.6315 - val_r_square: 0.1366 - val_rmse: 346.0867\n",
      "Epoch 217/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 189298.6823 - r_square: 0.1302 - rmse: 349.9916 - val_loss: 185014.3211 - val_r_square: 0.1371 - val_rmse: 345.9837\n",
      "Epoch 218/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 189204.1257 - r_square: 0.1306 - rmse: 349.8880 - val_loss: 184902.5775 - val_r_square: 0.1376 - val_rmse: 345.8813\n",
      "Epoch 219/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 189110.0218 - r_square: 0.1310 - rmse: 349.7855 - val_loss: 184791.8682 - val_r_square: 0.1381 - val_rmse: 345.7782\n",
      "Epoch 220/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 189016.4762 - r_square: 0.1315 - rmse: 349.6842 - val_loss: 184681.5170 - val_r_square: 0.1387 - val_rmse: 345.6755\n",
      "Epoch 221/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 188923.4755 - r_square: 0.1319 - rmse: 349.5842 - val_loss: 184571.1348 - val_r_square: 0.1392 - val_rmse: 345.5722\n",
      "Epoch 222/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 188831.0749 - r_square: 0.1323 - rmse: 349.4826 - val_loss: 184461.6296 - val_r_square: 0.1397 - val_rmse: 345.4676\n",
      "Epoch 223/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 188739.3692 - r_square: 0.1327 - rmse: 349.3823 - val_loss: 184352.9825 - val_r_square: 0.1402 - val_rmse: 345.3653\n",
      "Epoch 224/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 188648.2554 - r_square: 0.1331 - rmse: 349.2811 - val_loss: 184244.8391 - val_r_square: 0.1407 - val_rmse: 345.2622\n",
      "Epoch 225/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 188558.0213 - r_square: 0.1336 - rmse: 349.1819 - val_loss: 184137.5138 - val_r_square: 0.1412 - val_rmse: 345.1620\n",
      "Epoch 226/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 188468.8978 - r_square: 0.1340 - rmse: 349.0855 - val_loss: 184032.6642 - val_r_square: 0.1417 - val_rmse: 345.0614\n",
      "Epoch 227/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 188380.3453 - r_square: 0.1344 - rmse: 348.9876 - val_loss: 183928.7679 - val_r_square: 0.1422 - val_rmse: 344.9603\n",
      "Epoch 228/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 188292.5237 - r_square: 0.1348 - rmse: 348.8905 - val_loss: 183825.7664 - val_r_square: 0.1426 - val_rmse: 344.8615\n",
      "Epoch 229/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 188205.6203 - r_square: 0.1352 - rmse: 348.7955 - val_loss: 183724.1864 - val_r_square: 0.1431 - val_rmse: 344.7645\n",
      "Epoch 230/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 188120.0026 - r_square: 0.1356 - rmse: 348.7027 - val_loss: 183623.7640 - val_r_square: 0.1436 - val_rmse: 344.6674\n",
      "Epoch 231/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 188035.4103 - r_square: 0.1360 - rmse: 348.6103 - val_loss: 183524.3831 - val_r_square: 0.1441 - val_rmse: 344.5695\n",
      "Epoch 232/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 187951.6926 - r_square: 0.1363 - rmse: 348.5178 - val_loss: 183426.3620 - val_r_square: 0.1445 - val_rmse: 344.4733\n",
      "Epoch 233/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 187869.1317 - r_square: 0.1367 - rmse: 348.4256 - val_loss: 183329.3597 - val_r_square: 0.1450 - val_rmse: 344.3770\n",
      "Epoch 234/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 187787.5845 - r_square: 0.1371 - rmse: 348.3335 - val_loss: 183232.7229 - val_r_square: 0.1454 - val_rmse: 344.2818\n",
      "Epoch 235/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 187707.0792 - r_square: 0.1375 - rmse: 348.2427 - val_loss: 183137.4102 - val_r_square: 0.1459 - val_rmse: 344.1869\n",
      "Epoch 236/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 187627.5971 - r_square: 0.1378 - rmse: 348.1531 - val_loss: 183043.3739 - val_r_square: 0.1463 - val_rmse: 344.0930\n",
      "Epoch 237/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 187549.2954 - r_square: 0.1382 - rmse: 348.0627 - val_loss: 182951.2036 - val_r_square: 0.1467 - val_rmse: 343.9978\n",
      "Epoch 238/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 187471.7368 - r_square: 0.1386 - rmse: 347.9736 - val_loss: 182860.7613 - val_r_square: 0.1472 - val_rmse: 343.9067\n",
      "Epoch 239/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 187395.5155 - r_square: 0.1389 - rmse: 347.8860 - val_loss: 182772.4094 - val_r_square: 0.1476 - val_rmse: 343.8143\n",
      "Epoch 240/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 187320.3179 - r_square: 0.1393 - rmse: 347.7990 - val_loss: 182685.1689 - val_r_square: 0.1480 - val_rmse: 343.7252\n",
      "Epoch 241/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 187246.5069 - r_square: 0.1396 - rmse: 347.7143 - val_loss: 182599.5739 - val_r_square: 0.1484 - val_rmse: 343.6404\n",
      "Epoch 242/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 187174.3674 - r_square: 0.1399 - rmse: 347.6342 - val_loss: 182515.7775 - val_r_square: 0.1488 - val_rmse: 343.5583\n",
      "Epoch 243/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 187103.6950 - r_square: 0.1402 - rmse: 347.5533 - val_loss: 182433.5258 - val_r_square: 0.1491 - val_rmse: 343.4739\n",
      "Epoch 244/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 187033.9319 - r_square: 0.1406 - rmse: 347.4728 - val_loss: 182352.7009 - val_r_square: 0.1495 - val_rmse: 343.3901\n",
      "Epoch 245/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186965.0774 - r_square: 0.1409 - rmse: 347.3915 - val_loss: 182273.3344 - val_r_square: 0.1499 - val_rmse: 343.3073\n",
      "Epoch 246/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186897.2751 - r_square: 0.1412 - rmse: 347.3122 - val_loss: 182195.6662 - val_r_square: 0.1503 - val_rmse: 343.2253\n",
      "Epoch 247/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186830.4169 - r_square: 0.1415 - rmse: 347.2316 - val_loss: 182119.3548 - val_r_square: 0.1506 - val_rmse: 343.1420\n",
      "Epoch 248/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186764.2546 - r_square: 0.1418 - rmse: 347.1533 - val_loss: 182044.2554 - val_r_square: 0.1510 - val_rmse: 343.0612\n",
      "Epoch 249/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186699.0960 - r_square: 0.1421 - rmse: 347.0764 - val_loss: 181971.2630 - val_r_square: 0.1513 - val_rmse: 342.9816\n",
      "Epoch 250/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186635.0802 - r_square: 0.1424 - rmse: 347.0004 - val_loss: 181898.4956 - val_r_square: 0.1516 - val_rmse: 342.9013\n",
      "Epoch 251/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186571.9869 - r_square: 0.1427 - rmse: 346.9248 - val_loss: 181827.3447 - val_r_square: 0.1520 - val_rmse: 342.8228\n",
      "Epoch 252/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186509.5729 - r_square: 0.1430 - rmse: 346.8495 - val_loss: 181757.8229 - val_r_square: 0.1523 - val_rmse: 342.7447\n",
      "Epoch 253/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186447.8966 - r_square: 0.1433 - rmse: 346.7745 - val_loss: 181689.7163 - val_r_square: 0.1526 - val_rmse: 342.6681\n",
      "Epoch 254/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186387.0474 - r_square: 0.1435 - rmse: 346.6996 - val_loss: 181623.1730 - val_r_square: 0.1529 - val_rmse: 342.5905\n",
      "Epoch 255/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186326.9773 - r_square: 0.1438 - rmse: 346.6257 - val_loss: 181557.0689 - val_r_square: 0.1532 - val_rmse: 342.5160\n",
      "Epoch 256/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186268.0661 - r_square: 0.1441 - rmse: 346.5534 - val_loss: 181491.4306 - val_r_square: 0.1535 - val_rmse: 342.4405\n",
      "Epoch 257/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186210.0515 - r_square: 0.1444 - rmse: 346.4823 - val_loss: 181427.2086 - val_r_square: 0.1538 - val_rmse: 342.3679\n",
      "Epoch 258/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186153.0833 - r_square: 0.1446 - rmse: 346.4127 - val_loss: 181364.4960 - val_r_square: 0.1541 - val_rmse: 342.2939\n",
      "Epoch 259/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186096.7692 - r_square: 0.1449 - rmse: 346.3417 - val_loss: 181302.9469 - val_r_square: 0.1544 - val_rmse: 342.2209\n",
      "Epoch 260/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 186041.0984 - r_square: 0.1451 - rmse: 346.2725 - val_loss: 181242.6862 - val_r_square: 0.1547 - val_rmse: 342.1503\n",
      "Epoch 261/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185986.0130 - r_square: 0.1454 - rmse: 346.2039 - val_loss: 181183.1063 - val_r_square: 0.1550 - val_rmse: 342.0805\n",
      "Epoch 262/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 185931.8386 - r_square: 0.1456 - rmse: 346.1371 - val_loss: 181124.7029 - val_r_square: 0.1553 - val_rmse: 342.0103\n",
      "Epoch 263/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185878.4637 - r_square: 0.1459 - rmse: 346.0694 - val_loss: 181068.0367 - val_r_square: 0.1555 - val_rmse: 341.9408\n",
      "Epoch 264/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185825.5693 - r_square: 0.1461 - rmse: 346.0035 - val_loss: 181011.9916 - val_r_square: 0.1558 - val_rmse: 341.8730\n",
      "Epoch 265/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185773.2971 - r_square: 0.1464 - rmse: 345.9374 - val_loss: 180957.5926 - val_r_square: 0.1560 - val_rmse: 341.8050\n",
      "Epoch 266/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185721.4571 - r_square: 0.1466 - rmse: 345.8736 - val_loss: 180903.5664 - val_r_square: 0.1563 - val_rmse: 341.7395\n",
      "Epoch 267/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185670.4660 - r_square: 0.1468 - rmse: 345.8101 - val_loss: 180850.8668 - val_r_square: 0.1565 - val_rmse: 341.6724\n",
      "Epoch 268/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185620.0143 - r_square: 0.1471 - rmse: 345.7469 - val_loss: 180798.3083 - val_r_square: 0.1568 - val_rmse: 341.6068\n",
      "Epoch 269/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 185570.1903 - r_square: 0.1473 - rmse: 345.6848 - val_loss: 180746.5735 - val_r_square: 0.1570 - val_rmse: 341.5434\n",
      "Epoch 270/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185520.7234 - r_square: 0.1475 - rmse: 345.6219 - val_loss: 180695.5758 - val_r_square: 0.1573 - val_rmse: 341.4803\n",
      "Epoch 271/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185471.7551 - r_square: 0.1477 - rmse: 345.5601 - val_loss: 180644.7591 - val_r_square: 0.1575 - val_rmse: 341.4175\n",
      "Epoch 272/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185423.5120 - r_square: 0.1480 - rmse: 345.5000 - val_loss: 180594.5066 - val_r_square: 0.1577 - val_rmse: 341.3553\n",
      "Epoch 273/300\n",
      "19824/19824 [==============================] - 0s 3us/step - loss: 185376.0359 - r_square: 0.1482 - rmse: 345.4388 - val_loss: 180545.5396 - val_r_square: 0.1580 - val_rmse: 341.2920\n",
      "Epoch 274/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185328.7562 - r_square: 0.1484 - rmse: 345.3790 - val_loss: 180497.9300 - val_r_square: 0.1582 - val_rmse: 341.2316\n",
      "Epoch 275/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185281.9345 - r_square: 0.1486 - rmse: 345.3212 - val_loss: 180451.3103 - val_r_square: 0.1584 - val_rmse: 341.1731\n",
      "Epoch 276/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185235.7583 - r_square: 0.1488 - rmse: 345.2637 - val_loss: 180405.7411 - val_r_square: 0.1586 - val_rmse: 341.1147\n",
      "Epoch 277/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185190.0250 - r_square: 0.1490 - rmse: 345.2072 - val_loss: 180359.8653 - val_r_square: 0.1588 - val_rmse: 341.0578\n",
      "Epoch 278/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185144.9470 - r_square: 0.1493 - rmse: 345.1519 - val_loss: 180315.2213 - val_r_square: 0.1590 - val_rmse: 341.0005\n",
      "Epoch 279/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185100.3927 - r_square: 0.1495 - rmse: 345.0965 - val_loss: 180271.4029 - val_r_square: 0.1592 - val_rmse: 340.9448\n",
      "Epoch 280/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185056.2397 - r_square: 0.1497 - rmse: 345.0427 - val_loss: 180228.8133 - val_r_square: 0.1594 - val_rmse: 340.8909\n",
      "Epoch 281/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 185012.3099 - r_square: 0.1499 - rmse: 344.9881 - val_loss: 180187.2630 - val_r_square: 0.1596 - val_rmse: 340.8378\n",
      "Epoch 282/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184968.7590 - r_square: 0.1501 - rmse: 344.9358 - val_loss: 180145.9200 - val_r_square: 0.1598 - val_rmse: 340.7839\n",
      "Epoch 283/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184925.4196 - r_square: 0.1503 - rmse: 344.8811 - val_loss: 180105.5638 - val_r_square: 0.1600 - val_rmse: 340.7299\n",
      "Epoch 284/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184882.4460 - r_square: 0.1505 - rmse: 344.8303 - val_loss: 180064.6254 - val_r_square: 0.1602 - val_rmse: 340.6779\n",
      "Epoch 285/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184840.3085 - r_square: 0.1507 - rmse: 344.7799 - val_loss: 180024.4968 - val_r_square: 0.1604 - val_rmse: 340.6247\n",
      "Epoch 286/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184798.4271 - r_square: 0.1508 - rmse: 344.7304 - val_loss: 179985.0163 - val_r_square: 0.1606 - val_rmse: 340.5733\n",
      "Epoch 287/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184757.2901 - r_square: 0.1510 - rmse: 344.6817 - val_loss: 179946.4657 - val_r_square: 0.1608 - val_rmse: 340.5214\n",
      "Epoch 288/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184716.3841 - r_square: 0.1512 - rmse: 344.6325 - val_loss: 179908.8064 - val_r_square: 0.1609 - val_rmse: 340.4705\n",
      "Epoch 289/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184675.9789 - r_square: 0.1514 - rmse: 344.5842 - val_loss: 179871.4190 - val_r_square: 0.1611 - val_rmse: 340.4203\n",
      "Epoch 290/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184635.8233 - r_square: 0.1516 - rmse: 344.5382 - val_loss: 179834.0986 - val_r_square: 0.1613 - val_rmse: 340.3710\n",
      "Epoch 291/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184595.7804 - r_square: 0.1518 - rmse: 344.4900 - val_loss: 179797.0818 - val_r_square: 0.1615 - val_rmse: 340.3207\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 292/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184555.8277 - r_square: 0.1520 - rmse: 344.4433 - val_loss: 179760.8578 - val_r_square: 0.1616 - val_rmse: 340.2712\n",
      "Epoch 293/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184515.9922 - r_square: 0.1521 - rmse: 344.3956 - val_loss: 179724.3786 - val_r_square: 0.1618 - val_rmse: 340.2202\n",
      "Epoch 294/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184476.1262 - r_square: 0.1523 - rmse: 344.3480 - val_loss: 179686.4839 - val_r_square: 0.1620 - val_rmse: 340.1712\n",
      "Epoch 295/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184436.5783 - r_square: 0.1525 - rmse: 344.3028 - val_loss: 179649.2909 - val_r_square: 0.1621 - val_rmse: 340.1203\n",
      "Epoch 296/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184397.0899 - r_square: 0.1527 - rmse: 344.2563 - val_loss: 179611.8187 - val_r_square: 0.1623 - val_rmse: 340.0709\n",
      "Epoch 297/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184357.8062 - r_square: 0.1529 - rmse: 344.2098 - val_loss: 179574.8697 - val_r_square: 0.1625 - val_rmse: 340.0196\n",
      "Epoch 298/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184318.5620 - r_square: 0.1530 - rmse: 344.1631 - val_loss: 179538.4202 - val_r_square: 0.1627 - val_rmse: 339.9690\n",
      "Epoch 299/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184279.2882 - r_square: 0.1532 - rmse: 344.1168 - val_loss: 179502.3362 - val_r_square: 0.1628 - val_rmse: 339.9195\n",
      "Epoch 300/300\n",
      "19824/19824 [==============================] - 0s 2us/step - loss: 184240.2819 - r_square: 0.1534 - rmse: 344.0720 - val_loss: 179466.7076 - val_r_square: 0.1630 - val_rmse: 339.8706\n"
     ]
    }
   ],
   "source": [
    "# define callbacks\n",
    "early_stopping = EarlyStopping(monitor='val_loss', patience=20)\n",
    "reduce_lr = ReduceLROnPlateau(patience=5,factor=0.5)\n",
    "model_checkpoint = ModelCheckpoint('keras_regression_best.h5', monitor='val_loss', save_best_only=True)\n",
    "callbacks = [early_stopping, model_checkpoint, reduce_lr]\n",
    "\n",
    "# fit keras model\n",
    "ep = 300\n",
    "history  = keras_model.fit(feature_array2, gen_pt_array, batch_size=1024, \n",
    "                epochs=ep, validation_split=0.2, shuffle=False,\n",
    "                callbacks = callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEWCAYAAABIVsEJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZxcVZn/8c+3qnpNOukl+0YSQSCEECG/gIIIBAGZkTCKYxAkKDP8UBHGbQCdGRgWBWcUQRReIMgiQ4gogiKE/FhExggk7GExIQSyka3Tne4k3enl+f1xTyWVTq/Vy+3qPO/X677q1rnnnPvcutX91Ln31i2ZGc4551xXJeIOwDnnXG7yBOKccy4rnkCcc85lxROIc865rHgCcc45lxVPIM4557LiCcR1maSJkkxSKsv250p6thvrP0vS49m2722SbpH07z1dd18h6WlJ/xR3HH1F0p2Sru7hPldKOrEn+2yNJxD67sXuLeGf+TZJtRnTv8YdV28xs3vN7KTe6Lsn3gtmdoGZXdXTdfurfe0fflwkzZL0lqTtkp6StF+W/aQ/AGb+v8jqQ0xWnyBdv3SYmS2PO4jeJillZo376vpdvOLa/5KGAb8F/gn4PXAVcD9wVDe6Le3utvgIpAOS/lnSckmVkh6WNCaUS9L1kjZIqpb0qqSpYdmpkt6QVCNpjaRvt9JvgaSqdJtQNlzSDkkjJA2T9IdQp1LSnyV1eX9JukLSA5LuD/G8KOmwjOUHh0+QVZKWSjotY1mRpB9Jei9s47OSijK6P0vS+5I2SfpeOzFUhNduq6TngQ9lLNvrcFjmJ9pwuOt/w2tdCVzR8hBYaH+BpGWStkj6mSSFZcmwDZskvSvpwrYOv0m6B5gA/D49isuI7zxJ7wNPhrq/lvRBeF2ekXRIRj+7DklIOk7SaknfCu+VdZK+lGXdCkm/D6/jC5KuVjuHAiWdFvZpVXhND85YtlLSt8P7tjq8Pwrb6OfcsO//O7y+70r6VFh2DfBx4Kbwmt3UhX3607DutyTNamc7vizpzbDuBerEJ++w/q+G90SNpKskfUjSovD6zZeUn1H/7yW9HF6rv0ia1uK1ukTSq8A2SSlJh0t6KfT96/D6Xd3J/j6i6O+wRtL9QKuvewufAZaa2a/NrA64AjhM0kHttJnemf3bLWa2z0/ASuDEVspPADYBhwMFwE+BZ8Kyk4ElQCkg4GBgdFi2Dvh4mC8DDm9jvXcA12Q8/xrwWJj/AXALkBemjwNqox8D9m9j2RVAA3BG6OfbwLsZ/S4Hvgvkh+2tAQ4MbX8GPA2MBZLAx8LrMDGs8zagCDgMqAcObiOGecB8YBAwFVgDPBuWpftKZdR/GvinMH8u0Ah8nWjEXBTKnm2x/X8I+2ICsBE4JSy7AHgDGBf2xf9rub723gsZ8d0d4i8K5V8GSsLr8RPg5Yw2dwJXh/njQvxXhtf7VGA7UJZF3XlhKgamAKsyX4cW2/FhYBvwydDXv4Z9nZ+xnc8DY4By4E3ggjb6OpfoPfTP4X3wFWAt4f2Yub+6uE+/EWL7PFANlLdS9/QQ98Fh//8b8JdO/E0b8DAwBDiE6P35BDAZGBreE3ND3cOBDcCRYfvmhtenIOO1ehkYT/T+ywfeAy4O8X8G2JmxH9vsL6NtetvPCK/t1R1szw3AzS3KXgc+2877uNX9m7F/1gCrgV8Cw7L639ndf74DYaLtBHI78MOM54PDzp5I9M/2b0RDyESLdu8D/xcY0sF6TwRWZDz/X+CcMH8l8BBtJIZW/li2AlUZ08lh2RXAXzPqJggJLkwfZMYP3BfaJIAdRIfGWq4v/QYcl1H2PDCnlbrJ8JodlFH2fbqWQN5v0ee57J1Ajsl4Ph+4NMw/CfzfFq95Nglkcjuvf2moMzQ8v5M9k8KOFtu3ATiqK3UzXscDM5ZdTdsJ5N+B+S32+xrguIztPDtj+Q+BW9ro61xgecbz4rC9o1rury7s010JKOP988VW6j4KnNdiO7YD+3Xib+LojOdLgEsynv8I+EmYvxm4qkX7t4FPZLxWX85Ydmx4LTPjfzZjP7bZX2jbctv/QscJ5Hbg2hZl/wuc2877uNX9S/R/bAZRQh4JPAAsaG/9bU1+CKt9Y4g+LQBgZrXAZmCsmT0J3ET0KX29pFslDQlVP0v06fE9SX+S9NE2+n8SKJJ0ZBiWTwceDMv+i+iT1+OSVki6tINYDzez0oxpQcayVRnb0Ez0qWNMmFaFsrT3iEYcw4iG1u+0s84PMua3E70xWxpO9EZdlVH2Xiv12rOq4yptxjKmRfvO9NVuDOGw2LWS3pG0leiPFaLXrDWbbc9jzW29Vu3Vbe11bG9bWr53m0P9sRl1OrP/9qprZtvDbHv1O7LGwn+z4D2imFvaD7ghHAqqAiqJRvxjW6nb0vqM+R2tPE/Hvx/wrfQ6wnrGt4gn87Ue00r8mcvb66+1tp35e6glGk1lGkJ0xKAtre5fM6s1s8Vm1mhm64ELgZMy/n91mieQ9q0lejMAIGkQUEH06QMzu9HMjiAaIn8Y+E4of8HMZgMjgN8RfSLeS/ijng+cCXwB+IOZ1YRlNWb2LTObDHwa+GZ7x4k7MD5jGxJEh3PWhmm89jy3MiFs3yagjozzFVnaSHS4YnxG2YSM+W3hsTijbFSLPozsrSPa3rTxbVXsYF2Z5V8AZhONZoYSfeKG6B9bb0m/jp3dlpbvXYX6a3ohtpavWWf26dgQU9oEophbWkU0gsz8cFRkZn/pXsh7reOaFusoNrP7MupkbuO6VuLP3Bft9dda28y/h7YsJTpUDOz6X/ShUN5d6W3r8vvXE8hueZIKM6YU8D/AlyRNl1RAdOjlOTNbKen/hJFDHtEfTB3QJClf0fcUhppZA9GhpaZ21vs/RMeAzwrzwK6TcPuHN1q6j/b6ac8Rkj4TtulfiI4H/xV4LsT+r5LyJB1HlKzmheR2B/BjSWPCp+6Phteh08ysiejqkSskFUuaQnRMOL18I9E/tbPDOr5M95NWpvnAxZLGSioFLumg/nqi4+TtKSF6DTcT/ZP8frej7EArr+NBwDntNJkP/J2iSz/zgG+FmHvyH2/aHq9ZJ/fpCOCi8L77HNE5jj+20vctwGUKFylIGhrq96TbgAvC37MkDZL0d5JK2qi/iOhv8cJwQn02MLOT/S0i+iBwUWj7mRZt2/IgMFXSZ8PJ8P8AXjWzt7q6sSGuAyUlJFUANwJPm1l1V/vyBLLbH4mGtenpCjN7guhY8m+IPjl8CJgT6g8heqNsIRqCbgb+Oyz7IrAyHN64ADi7rZWaWfqf+Bii471pBxCd8K0letP93Myebif+V7Tndd0/yVj2EFGS2hJi+4yZNZjZTuA04FNEI46fE52DSb8pvw28BrxAdOjgOrJ7z1xINHz+gOiY/y9bLP9notHbZqLRXE/+k7sNeBx4FXiJaD830nYy/gHwb+HQw15XzwV3E+3zNUQnY//ag/G250KiEc8HwD1E56vqW6toZm8Tve9+SrRvPw18OuzznnYDcIaiq6RuDGUd7dPniN7jm4BrgDPMbHMr2/Eg0ftuXvh7ep3o/dpjzGxxiPcmor+R5UTnadqqv5PoxPl5ROcbzya6iKO+o/4y2p4bln2e6INBRzFuJDo0fk1odyS7/xch6buSHm2jeUuTgceIDn+9HuI+s5Nt95C+isINUJKuIDoR32YS25couvz0FjPbr8PK/Zyk64hOZM/tsHI/IulcopPkx8QdS0+R9BzR+6rlh6MBzUcgbkBT9F2WU8PhgrHA5ey+UCGnSDpI0rRwWGQm0SfgnNyWXCfpE5JGhffVXGAa0af6fYonEDfQCfhPomH/S0TXw/9HrBFlr4TocMc2onMcPyI6PLnPkfTxFodsd019FMKBwCtE31/5FtEhuHXd6TAchmptmzp7aKrP+SEs55xzWfERiHPOuazsUzdTHDZsmE2cODHuMJxzLqcsWbJkk5kNb1m+TyWQiRMnsnjx4rjDcM65nCKp1W/L+yEs55xzWfEE4pxzLiueQJxzzmVlnzoH4pzbdzQ0NLB69Wrq6uriDiVnFBYWMm7cOPLy8jpV3xOIc25AWr16NSUlJUycOJE9b37rWmNmbN68mdWrVzNp0qROtfFDWM65Aamuro6KigpPHp0kiYqKii6N2DyBOOcGLE8eXdPV18sTSCdsXPQ/bHv+XvDbvjjn3C6eQDph1dN3MOiPX2XdjSfSsL7Lv9/inNtHDR7cnV/97f88gXTC4HMf4JdlF1Fc+QZNNx/Lur/Mizsk55yLnSeQTvjw6FLOvehKXvr0At5mP0YuuIC3F9wWd1jOuRz03nvvMWvWLKZNm8asWbN4//33Afj1r3/N1KlTOeywwzj22GMBWLp0KTNnzmT69OlMmzaNZcuWxRn6Xvwy3k6SxHEzpvHBxAW8cstspv7lEt4aOpaDjjo17tCccx34z98v5Y21W3u0zyljhnD5pw/pcrsLL7yQc845h7lz53LHHXdw0UUX8bvf/Y4rr7ySBQsWMHbsWKqqqgC45ZZbuPjiiznrrLPYuXMnTU1t/RJzPHwE0kWjhpUz8asPsi45imGPXcD69d36DRnn3D5m0aJFfOELXwDgi1/8Is8++ywARx99NOeeey633XbbrkTx0Y9+lO9///tcd911vPfeexQVFcUWd2t8BJKFsvJh1J5xB6X3f4q/3HURI74z3y8XdK4fy2ak0FfS/ztuueUWnnvuOR555BGmT5/Oyy+/zBe+8AWOPPJIHnnkEU4++WR+8YtfcMIJJ8Qc8W4+AsnS+ClH8dakczh2++P8+dk/xR2Ocy5HfOxjH2PevOhCnHvvvZdjjjkGgHfeeYcjjzySK6+8kmHDhrFq1SpWrFjB5MmTueiiizjttNN49dVX4wx9L55AuuHgz11BrQahp66moak57nCcc/3M9u3bGTdu3K7pxz/+MTfeeCO//OUvmTZtGvfccw833HADAN/5znc49NBDmTp1KsceeyyHHXYY999/P1OnTmX69Om89dZbnHPOOTFv0Z72qd9EnzFjhvX0D0qt+PX3mLz0JhbO+iOf/PjRPdq3cy57b775JgcffHDcYeSc1l43SUvMbEbLuh2OQCTdIWmDpNczyv5L0luSXpX0oKTSjGWXSVou6W1JJ2eUHyHptbDsRoUDf5IKJN0fyp+TNDGjzVxJy8I0N6N8Uqi7LLTN7/Sr08MmnnQhjSTZ+uxt7EvJ2DnnOnMI607glBZlC4GpZjYN+BtwGYCkKcAc4JDQ5ueSkqHNzcD5wAFhSvd5HrDFzPYHrgeuC32VA5cDRwIzgcsllYU21wHXm9kBwJbQRywSQ0ezZtQJHF+3kNdXbY4rDOec63MdJhAzewaobFH2uJk1hqd/BcaF+dnAPDOrN7N3geXATEmjgSFmtsiij+l3A6dntLkrzD8AzAqjk5OBhWZWaWZbiJLWKWHZCaEuoW26r1gM++gXKVctLz3z+zjDcM65PtUTJ9G/DDwa5scCqzKWrQ5lY8N8y/I92oSkVA1UtNNXBVCVkcAy+9qLpPMlLZa0eOPGjV3euM4YNOUk6lXIoHceoanZD2M55/YN3Uogkr4HNAL3potaqWbtlGfTpr2+9l5gdquZzTCzGcOHD2+rWvfkFbFpzPEc2/wcL79f2XF955wbALJOIOGk9t8DZ9nus8ergfEZ1cYBa0P5uFbK92gjKQUMJTpk1lZfm4DSULdlX7EpO+zvGK6tvP7SorhDcc65PpFVApF0CnAJcJqZbc9Y9DAwJ1xZNYnoZPnzZrYOqJF0VDiHcQ7wUEab9BVWZwBPhoS0ADhJUlk4eX4SsCAseyrUJbRN9xWb4gOPB6B+2dPxBuKc6xeOO+44FixYsEfZT37yE7761a+2264zt4A/7rjj6OmvJGSjM5fx3gcsAg6UtFrSecBNQAmwUNLLkm4BMLOlwHzgDeAx4Gtmlr7711eAXxCdWH+H3edNbgcqJC0HvglcGvqqBK4CXgjTlaEMouT1zdCmIvQRr6HjqCqawKSaJVRt3xl3NM65mJ155pm7vnGeNm/ePM4888yYIup5nbkK60wzG21meWY2zsxuN7P9zWy8mU0P0wUZ9a8xsw+Z2YFm9mhG+WIzmxqWXZg+7GVmdWb2udDnTDNbkdHmjlC+v5n9MqN8Rai7f2hb33MvSfbqxh3NkYk3WbLSL+d1bl93xhln8Ic//IH6+ujf08qVK1m7di3HHHMMtbW1zJo1i8MPP5xDDz2Uhx7K/iDKfffdt+sb7JdccgkATU1NnHvuuUydOpVDDz2U66+/HoAbb7yRKVOmMG3aNObMmdPtbfSbKfag8gOPIX/Zfax4+xVmTRkddzjOubRHL4UPXuvZPkcdCp+6ts3FFRUVzJw5k8cee4zZs2czb948Pv/5zyOJwsJCHnzwQYYMGcKmTZs46qijOO2007p8U9a1a9dyySWXsGTJEsrKyjjppJP43e9+x/jx41mzZg2vvx59/zt9e/hrr72Wd999l4KCgl1l3eH3wupB+ROOAKBu5QsxR+Kc6w8yD2NlHr4yM7773e8ybdo0TjzxRNasWcP69eu73P8LL7zAcccdx/Dhw0mlUpx11lk888wzTJ48mRUrVvD1r3+dxx57jCFDhgAwbdo0zjrrLH71q1+RSnV//OAjkJ407MPsTBQydMvrNDY1k0p6fnauX2hnpNCbTj/9dL75zW/y4osvsmPHDg4//HAgugvvxo0bWbJkCXl5eUycOJG6urou99/W7ZPKysp45ZVXWLBgAT/72c+YP38+d9xxB4888gjPPPMMDz/8MFdddRVLly7tViLx/3A9KZFka+kUprCCFZu2xR2Ncy5mgwcP5rjjjuPLX/7yHifPq6urGTFiBHl5eTz11FO89957WfV/5JFH8qc//YlNmzbR1NTEfffdxyc+8Qk2bdpEc3Mzn/3sZ7nqqqt48cUXaW5uZtWqVRx//PH88Ic/pKqqitra2m5tn49AepjGfoRDNt/D42u38OGRJXGH45yL2ZlnnslnPvOZPa7IOuuss/j0pz/NjBkzmD59OgcddFCrbdM/LNWW0aNH84Mf/IDjjz8eM+PUU09l9uzZvPLKK3zpS1+iuTn6mYkf/OAHNDU1cfbZZ1NdXY2Z8Y1vfIPS0tI2++4Mv517D2tcfDepP3ydWw77DRf8w4m9ui7nXNv8du7Z6dHbubuuSY2MXvjta9+IORLnnOtdnkB62vAPA5BX+beYA3HOud7lCaSnFQ5lW/4wRu98j+odDXFH49w+bV86RN8Tuvp6eQLpBXWlB7C/1rDSr8RyLjaFhYVs3rzZk0gnmRmbN2+msLCw0238KqxekBxxIPuvf4knNtVy2PjuXeXgnMvOuHHjWL16Nb31O0ADUWFhIePGjeu4YuAJpBcMGnMQea/X8cHaVfCRzu8M51zPycvLY9KkSXGHMaD5IaxekDdsMgDbNrwTcyTOOdd7PIH0htL9ALDNK+ONwznnepEnkN5QOgGAvNpVfgLPOTdgeQLpDfnFbM8fxsjGD/xSXufcgOUJpJfUl4xnvDawpmpH3KE451yv8ATSW0r3Y0JiA2u2eAJxzg1MnkB6ScHwyYxmM+sqt8YdinPO9QpPIL2kaNh+JGVUb1wTdyjOOdcrPIH0Eg0dC8DOytUxR+Kcc73DE0hvGTIGgKatPgJxzg1MnkB6S8loAPJq18UciHPO9Q5PIL2lqIyGRAGDd26kvrEp7micc67HeQLpLRJ1RaMYrc1s2FofdzTOOdfjPIH0osZBoxilSjbU1MUdinPO9ThPIL1IQ8YyWpWs9xGIc24A8t8D6UUF5eMoZgsbqrfHHYpzzvW4Dkcgku6QtEHS6xll5ZIWSloWHssyll0mabmktyWdnFF+hKTXwrIbJSmUF0i6P5Q/J2liRpu5YR3LJM3NKJ8U6i4LbfO7/1L0vILyseSriZot6+MOxTnnelxnDmHdCZzSouxS4AkzOwB4IjxH0hRgDnBIaPNzScnQ5mbgfOCAMKX7PA/YYmb7A9cD14W+yoHLgSOBmcDlGYnqOuD6sP4toY9+J1EyEoC6LX4pr3Nu4OkwgZjZM0Bli+LZwF1h/i7g9IzyeWZWb2bvAsuBmZJGA0PMbJFFP5Bxd4s26b4eAGaF0cnJwEIzqzSzLcBC4JSw7IRQt+X6+5fBUQJprvERiHNu4Mn2JPpIM1sHEB5HhPKxwKqMeqtD2dgw37J8jzZm1ghUAxXt9FUBVIW6Lfvai6TzJS2WtHjjxo1d3MxuCgmE2g19u17nnOsDPX0Vllops3bKs2nTXl97LzC71cxmmNmM4cOHt1WtdwyK1pfasalv1+ucc30g2wSyPhyWIjymP2KvBsZn1BsHrA3l41op36ONpBQwlOiQWVt9bQJKQ92WffUvBSU0JAooaaykrsG/je6cG1iyTSAPA+mrouYCD2WUzwlXVk0iOln+fDjMVSPpqHAO45wWbdJ9nQE8Gc6TLABOklQWTp6fBCwIy54KdVuuv3+RqC8YxjBVs6nWvwvinBtYOnMZ733AIuBASaslnQdcC3xS0jLgk+E5ZrYUmA+8ATwGfM3M0h+9vwL8gujE+jvAo6H8dqBC0nLgm4QrusysErgKeCFMV4YygEuAb4Y2FaGPfqmxeATDqWJz7c64Q3HOuR6l6AP9vmHGjBm2ePHiPl1n1S//kQ/efYO1X3iCEw4a2afrds65niBpiZnNaFnutzLpZakhIxmuKjbV+AjEOTew+K1MellB6WiKqaWypjbuUJxzrkf5CKSX5Q0ZQULGjqo+/g6Kc871Mk8gva14GAD1NZ5AnHMDiyeQ3lZcAUBTzeaYA3HOuZ7lCaS3hQRi2z2BOOcGFk8gvS0kkGRdy/tROudcbvME0tuKywHI37mF5uZ95zs3zrmBzxNIb0vmUZ8aTBk1bNnu3wVxzg0cnkD6QENBGWWqYcv2hrhDcc65HuMJpA80F5ZTTg1VPgJxzg0gnkD6QnG5j0CccwOOJ5A+kBw0jHLVsGWbj0CccwOHJ5A+kDdkuJ9Ed84NOH4zxT6QN3gY+aqnprYm7lCcc67H+AikD2hQ9GXCnVv9t9GdcwOHJ5C+EL6N3ljrCcQ5N3B4AukLIYHI74flnBtAPIH0hXQC8fthOecGEE8gfSEkkPz6LTEH4pxzPccTSF8oLMUQBQ1VmPkNFZ1zA4MnkL6QTFGfGkKpbWVrXWPc0TjnXI/wBNJHGgrKKJffD8s5N3B4AukjTYXl4dvofj8s59zA4Amkrwwqp1y1fj8s59yA4Qmkj0Q3VNzq98Nyzg0YnkD6SF5JuKGij0CccwNEtxKIpG9IWirpdUn3SSqUVC5poaRl4bEso/5lkpZLelvSyRnlR0h6LSy7UZJCeYGk+0P5c5ImZrSZG9axTNLc7mxHX8gfMpwCNVJbUxV3KM451yOyTiCSxgIXATPMbCqQBOYAlwJPmNkBwBPhOZKmhOWHAKcAP5eUDN3dDJwPHBCmU0L5ecAWM9sfuB64LvRVDlwOHAnMBC7PTFT9UaK4HICGrRtjjsQ553pGdw9hpYAiSSmgGFgLzAbuCsvvAk4P87OBeWZWb2bvAsuBmZJGA0PMbJFF37K7u0WbdF8PALPC6ORkYKGZVZrZFmAhu5NO/xQSSOM2v52Jc25gyDqBmNka4L+B94F1QLWZPQ6MNLN1oc46YERoMhZYldHF6lA2Nsy3LN+jjZk1AtVARTt97UXS+ZIWS1q8cWOMn/6LogTS7AnEOTdAdOcQVhnRCGESMAYYJOns9pq0UmbtlGfbZs9Cs1vNbIaZzRg+fHg74fWyougIm+r8fljOuYGhO4ewTgTeNbONZtYA/Bb4GLA+HJYiPG4I9VcD4zPajyM65LU6zLcs36NNOEw2FKhsp6/+KxzCStX7SXTn3MDQnQTyPnCUpOJwXmIW8CbwMJC+Kmou8FCYfxiYE66smkR0svz5cJirRtJRoZ9zWrRJ93UG8GQ4T7IAOElSWRgJnRTK+q/CUgDyd/oNFZ1zA0PWv4luZs9JegB4EWgEXgJuBQYD8yWdR5RkPhfqL5U0H3gj1P+amTWF7r4C3AkUAY+GCeB24B5Jy4lGHnNCX5WSrgJeCPWuNLP+fXIhlc/O5CAGN9ayo6GJ4nz/OXrnXG7TvvRpeMaMGbZ48eLY1r/tuoN5rHYyR33rAcaWFsUWh3POdYWkJWY2o2W5fxO9DzUWlFHKNr8jr3NuQPAE0oesqJQy1VDld+R1zg0AnkD6UKK4gqFs8wTinBsQPIH0odTgcspU43fkdc4NCH4pUB/KH1xBAduo3l4fdyjOOddtnkD6UGpwBcjYsbV/X3HsnHOd4Yew+lK4H1aD3w/LOTcAeALpS+F+WH5DRefcQOAJpC+F+2HZdk8gzrnc5wmkL4URSKLe78jrnMt9nkD6UjgHkud35HXODQCeQPpS4VAAChqq/Y68zrmc5wmkLyVT1KdKKLFaausb447GOee6xRNIH2vIH+r3w3LODQieQPpY06478noCcc7lNk8gfcyKyihVDVU7/H5Yzrnc5gmkjyUGlfsIxDk3IHgC6WOpQeXhHIiPQJxzuc1vptjHCkqGU6ztVG+rizsU55zrFh+B9LHkoOjLhDtq/HYmzrnc5gmkr4X7YTXWbo45EOec6x5PIH3N78jrnBsgPIH0tXA/LHZ4AnHO5TZPIH2tqBTwO/I653KfJ5C+Vpy+I291zIE451z3eALpawVDaSZBQUMVzc1+R17nXO7yBNLXEgl25g1hKNuo8TvyOudyWLcSiKRSSQ9IekvSm5I+Kqlc0kJJy8JjWUb9yyQtl/S2pJMzyo+Q9FpYdqMkhfICSfeH8uckTcxoMzesY5mkud3Zjr6WviNvtd/OxDmXw7o7ArkBeMzMDgIOA94ELgWeMLMDgCfCcyRNAeYAhwCnAD+XlAz93AycDxwQplNC+XnAFjPbH7geuC70VQ5cDhwJzAQuz0xU/V1TYRlD2cYWv52Jcy6HZZ1AJA0BjgVuBzCznWZWBcwG7grV7gJOD/OzgXlmVm9m7wLLgZmSRgNDzGyRRT/Td3eLNum+HgBmhdHJycBCM6s0sy3AQvJalwAAABVySURBVHYnnX5PRWXR/bB2+AjEOZe7ujMCmQxsBH4p6SVJv5A0CBhpZusAwuOIUH8ssCqj/epQNjbMtyzfo42ZNQLVQEU7fe1F0vmSFktavHHjxmy3tUepOH1HXh+BOOdyV3cSSAo4HLjZzD4CbCMcrmqDWimzdsqzbbNnodmtZjbDzGYMHz68nfD6Tt7gCkpV67d0d87ltO4kkNXAajN7Ljx/gCihrA+HpQiPGzLqj89oPw5YG8rHtVK+RxtJKWAoUNlOXzkhv2QYJdrB1todcYfinHNZyzqBmNkHwCpJB4aiWcAbwMNA+qqoucBDYf5hYE64smoS0cny58NhrhpJR4XzG+e0aJPu6wzgyXCeZAFwkqSycPL8pFCWE9J35K2r2RRzJM45l73u/h7I14F7JeUDK4AvESWl+ZLOA94HPgdgZkslzSdKMo3A18ysKfTzFeBOoAh4NEwQnaC/R9JyopHHnNBXpaSrgBdCvSvNLHduLhVuqNi4ze/I65zLXd1KIGb2MjCjlUWz2qh/DXBNK+WLgamtlNcRElAry+4A7uhKvP2G35HXOTcA+DfR41CcviOv31DROZe7PIHEIYxAknWeQJxzucsTSBzCb4LkNfgdeZ1zucsTSBwKSmhSksKGKpr8jrzOuRzlCSQOEjvzhlLKNmrq/MuEzrnc5AkkJg35pZSqhi3+bXTnXI7yBBKT5sIyStlG5Ta/H5ZzLjd5AolJoriMMtV6AnHO5SxPIDFJDqpgqGqp3FYfdyjOOZcVTyAxyS+poIxaNvsIxDmXo7p7LyyXpbzBFeSpnuqtNXGH4pxzWfERSFzCt9HrtvoNFZ1zuckTSFzC/bAaa/2W7s653OQJJC6Dwq8jbu8fP7PrnHNd5QkkLoOin4pPbfcRiHMuN3kCicugYQAU1FcS/ciic87lFk8gcSkqo0kpSq2K7TubOq7vnHP9jCeQuEjszC+ngq1srvXvgjjnco8nkBg1FlVQoWo2+bfRnXM5yBNInAaPYJiq2bDVE4hzLvd4AolRqmQEw7SVjbWeQJxzuccTSIwKSkcxjGo2Vu+IOxTnnOsyTyAxSgweTqEaqKquijsU55zrMk8gcQpfJtxZ/UHMgTjnXNd5AonT4CiBWI0nEOdc7vEEEqchYwBIbVsfcyDOOdd1nkDiVDIagOL6DTQ1++1MnHO5xRNInAqH0pgoZASV/tvozrmc0+0EIikp6SVJfwjPyyUtlLQsPJZl1L1M0nJJb0s6OaP8CEmvhWU3SlIoL5B0fyh/TtLEjDZzwzqWSZrb3e2IhUR90UhGqZINNXVxR+Occ13SEyOQi4E3M55fCjxhZgcAT4TnSJoCzAEOAU4Bfi4pGdrcDJwPHBCmU0L5ecAWM9sfuB64LvRVDlwOHAnMBC7PTFS5pLlkFCO1hQ+qPYE453JLtxKIpHHA3wG/yCieDdwV5u8CTs8on2dm9Wb2LrAcmClpNDDEzBZZdF/zu1u0Sff1ADArjE5OBhaaWaWZbQEWsjvp5JS80rGMopI1Vf5lQudcbunuCOQnwL8CzRllI81sHUB4HBHKxwKrMuqtDmVjw3zL8j3amFkjUA1UtNPXXiSdL2mxpMUbN/a/X/8rKBvHSFWxZsv2uENxzrkuyTqBSPp7YIOZLelsk1bKrJ3ybNvsWWh2q5nNMLMZw4cP71SgfUlDx1CgBqoqN8QdinPOdUl3RiBHA6dJWgnMA06Q9CtgfTgsRXhM/2dcDYzPaD8OWBvKx7VSvkcbSSlgKFDZTl+5p2QUAA2bV3VQ0Tnn+pesE4iZXWZm48xsItHJ8SfN7GzgYSB9VdRc4KEw/zAwJ1xZNYnoZPnz4TBXjaSjwvmNc1q0Sfd1RliHAQuAkySVhZPnJ4Wy3FM6AYBkzeoOKjrnXP+S6oU+rwXmSzoPeB/4HICZLZU0H3gDaAS+Zmbp33L9CnAnUAQ8GiaA24F7JC0nGnnMCX1VSroKeCHUu9LMKnthW3pf2SQASurW0NDUTF7Sv5rjnMsNPZJAzOxp4OkwvxmY1Ua9a4BrWilfDExtpbyOkIBaWXYHcEe2MfcbRWU0pAYxvnEDH1TXMb68OO6InHOuU/zjbtwk6ksmMEEbWFXpV2I553KHJ5B+IFk+ifHawIpN2+IOxTnnOs0TSD9QOHwyE7SBFRtq4w7FOec6zRNIP6DyiRSqgU3r3487FOec6zRPIP1BuBKredPymANxzrnO8wTSH4w4CIDS2neoa2jqoLJzzvUPnkD6gyFjaUiVcKBW8d5mvxLLOZcbPIH0BxINww7iwMQq3lhXHXc0zjnXKZ5A+onCsVM5SKt45f2quENxzrlO8QTSTyRGHsIQbWfN+34i3TmXGzyB9BejpgGQt+FVGpqaO6jsnHPx8wTSX4yZTlMin4/Ym7z9QU3c0TjnXIc8gfQXqQKaxsxgZuIt/rxsU9zROOdchzyB9CP5k49mamIli95cGXcozjnXIU8g/cl+R5OkmYLVf6F6e0Pc0TjnXLs8gfQn+x1NY/5QPpX4K48tXRd3NM451y5PIP1JKp/kIbM5JbmE+/73b0S/3uucc/2TJ5B+Rod+lmJ2sP/Gx1m0YnPc4TjnXJs8gfQ3kz5B84ipfD3/91z98Gs0+ndCnHP9lCeQ/kYi8YnvsJ+t5fBND3HDE8vijsg551rlCaQ/Ovg0bPLx/HvBPP741J/41V/fizsi55zbiyeQ/iiRQLN/Rn7RYO4v/i9ueehJ/v13r7N9Z2PckTnn3C6eQPqroWPR2b+hIq+eBYP+k1XP/45Tb/gzC99Y71dnOef6BU8g/dnow9A/PcGgspHcmf9f/Hvdj7j87sf4+58+y29fXM2Onf7rhc65+Ghf+jQ7Y8YMW7x4cdxhdF1jPTz7E+zP/401N/N48lh+vu143i34MJ+cMopPHjySj394OIMLUnFH6pwbgCQtMbMZe5V7AskhVatg0U3YkrtQ4w425I/n1zs/xu/rP8KKxH5Mn1DGjP3K+D8Tyzl03FCGDS6IO2Ln3ADgCYQBkEDSdlTBmw/Dq/Nh5Z8BqM0rZ3HiMJ7YNolXmibxlk1gyOBBHDiqhINGDWHy8EGMLytmQnkxY0qLyE/50UvnXOf0eAKRNB64GxgFNAO3mtkNksqB+4GJwErgH81sS2hzGXAe0ARcZGYLQvkRwJ1AEfBH4GIzM0kFYR1HAJuBz5vZytBmLvBvIZyrzeyujmIeMAkk09a18M6T8M5T8O6fYNtGAJqUYl3BZP5m43htewXLm0byro1ipY1iu4oZPbSI0UMLGV5SwLDBBQwvCVOYLx+Uz5CiPEoKUiQSinkjnXNx6o0EMhoYbWYvSioBlgCnA+cClWZ2raRLgTIzu0TSFOA+YCYwBvh/wIfNrEnS88DFwF+JEsiNZvaopK8C08zsAklzgH8ws8+HJLUYmAFYWPcR6UTVlgGZQDKZQdX7sO5lWPsSrH0ZNv0Ntq7Zo9qO1FC2JMvZSDnrmkt5v2Eo7zcMZZMNpYrBVNlgqmwQVQymXgWUFKQYWpzHkMI8hhbtfiwpTFFckKI4P0lxfpKivCTF+SmKC5IUh/mi/CSDCpIU50XzeUkheUJyLpe0lUCyPutqZuuAdWG+RtKbwFhgNnBcqHYX8DRwSSifZ2b1wLuSlgMzJa0EhpjZohDo3USJ6NHQ5orQ1wPATYr++5wMLDSzytBmIXAKUYLad0lQtl80TZm9u3zndtjyLmxeDpvfoah6NUU1HzCmZh2H1bwJjeshr/UruhoTBWxPllDbVELttkHU1hZS01xAdVMB1U35bG3OZ6sV8gGFbKOQbVbIdgqoJ596y2MnqWiePOotjwalsGQhlioglcojP5mgIC9JfjJBfiqaCsJjuqwgldxVnkqIVDJBXlIkEyIvmSCZUFQelu35mF6WIJkUeYlEaNdGnfA8mfk8oT0ePQE6F+mRy3YkTQQ+AjwHjAzJBTNbJ2lEqDaWaISRtjqUNYT5luXpNqtCX42SqoGKzPJW2rSM7XzgfIAJEyZktX05L78YRh4STa1pboLaDbB9E+zYsseU2l7JkB1bGLJjC9RVw85tsHNTeKzB6muRZXc5cXNjgsamfBoa8tmpPBqURyMpGi1JIwkaSdJgYSLBzjDfaAl2WlhOkiZL0kg07STJNpI0EfWRXtZMInoeyptD/00kaLTosdVl6bYWLW8iQbOSoBSWSKJEEkukIJGERAqFeWWWJaPyVLK1hBQlsWSyjfL082Qb5buW7y5PKjMJttEm2VZfiVZi2TOZ+iFNl9btBCJpMPAb4F/MbGs7n85aW2DtlGfbZs9Cs1uBWyE6hNVWcPu0RBKGjI6mLpIZNO0MCaU2PG6Dxrro8uPG+mi+aWdGWR007iTRWEd+Yx35TTsZlF7W1ADNjbunzOe75ndCcyMWnlsrdRSeq/W3Rc8yorN6HeTR5pCkmpTYPb9rUkhguxNW464kGCW/RkvQsCvZJVpJjNHzBktQx5710uto3JVw90yUTRl9NIV17krCZM5H6yCRpFmpKIEqSpaWSCElIZmERF5IpJmJNUqmUeLN3yPptZ3QQnmyjfKQ0JKCZDJBUiKZgIRCMs1IqonwmFm+u17UJpVIkEiwR5vUHvXURt/s6ntfGqF2K4FIyiNKHvea2W9D8XpJo8PoYzSwIZSvBsZnNB8HrA3l41opz2yzWlIKGApUhvLjWrR5ujvb4rIkQaogmorL+3bVLR5b1dwM1pSRlJrC1Nh2efp5e8t3LWvZrhGsudV2ieZGEtZE3q7yFo/Wop/m5hbra8KaG6GpEbOm6DEstz1iiNrJdver5kbU3BSV9YbmMEF0TKETdiemVJTolNqV4BpCImsgSYOlaCQRRqLJkFxTYZSa2pVoo9FraJPuk9Tuviy1q8/0Ohv2GPGmdiXWJhI0I5ozn1tU1kQCC49tzUMCUxJTAlMCwjy7nkfLM5+jJImQwBKKEpWUnic83z2fEC2eZ9RP7F3/6tOnMqa0qEd3e9YJJJyLuB1408x+nLHoYWAucG14fCij/H8k/ZjoJPoBwPPhJHqNpKOIDoGdA/y0RV+LgDOAJ8PVWQuA70sqC/VOAi7LdlvcAJaI/phJ5sUdSY/oVNLsyF6JqTOJsLWE1loibK1duu+G3Y9NjSSbG0k2N5Df3BRGjmEk2dSYMZ/ZJpq3poaQSBuw5rrd9ZvS64vqqXn3aDTbw6y9xmj1mEkzwoiSSno+Kk9gCssAY/dyg131mxGEx+aQ0CxdvuXXUHpwj25Gd0YgRwNfBF6T9HIo+y5R4pgv6TzgfeBzAGa2VNJ84A2gEfia2a69+hV2X8b7aJggSlD3hBPulcCc0FelpKuAF0K9K9Mn1J1zHUgkIJEP5McdSVZEFgk0M/mlE9leSStjmTXvntKjwz3mLaNeUzfbNIWRcjOJzDZYWG5hyuifFs9b1mltecXQnt4V/kVC55xz7WvrMl7/OrJzzrmseAJxzjmXFU8gzjnnsuIJxDnnXFY8gTjnnMuKJxDnnHNZ8QTinHMuK55AnHPOZWWf+iKhpI3Ae1k2HwZs6sFw4uTb0j/5tvRPA2VburMd+5nZ8JaF+1QC6Q5Ji1v7JmYu8m3pn3xb+qeBsi29sR1+CMs551xWPIE455zLiieQzrs17gB6kG9L/+Tb0j8NlG3p8e3wcyDOOeey4iMQ55xzWfEE4pxzLiueQDog6RRJb0taLunSuOPpKkkrJb0m6WVJi0NZuaSFkpaFx7KO+omDpDskbZD0ekZZm7FLuizsp7clnRxP1K1rY1uukLQm7JuXJZ2asaw/b8t4SU9JelPSUkkXh/Kc2zftbEvO7RtJhZKel/RK2Jb/DOW9t1/MzKc2JiAJvANMJvr9z1eAKXHH1cVtWAkMa1H2Q+DSMH8pcF3ccbYR+7HA4cDrHcUOTAn7pwCYFPZbMu5t6GBbrgC+3Urd/r4to4HDw3wJ8LcQc87tm3a2Jef2DdEv/Q4O83nAc8BRvblffATSvpnAcjNbYWY7gXnA7Jhj6gmzgbvC/F3A6THG0iYzewZo+Vv3bcU+G5hnZvVm9i6wnGj/9QttbEtb+vu2rDOzF8N8DfAmMJYc3DftbEtb+vO2mJnVhqd5YTJ6cb94AmnfWGBVxvPVtP/m6o8MeFzSEknnh7KRZrYOoj8gYERs0XVdW7Hn6r66UNKr4RBX+tBCzmyLpInAR4g+7eb0vmmxLZCD+0ZSUtLLwAZgoZn16n7xBNI+tVKWa9c9H21mhwOfAr4m6di4A+olubivbgY+BEwH1gE/CuU5sS2SBgO/Af7FzLa2V7WVsn61Pa1sS07uGzNrMrPpwDhgpqSp7VTv9rZ4AmnfamB8xvNxwNqYYsmKma0NjxuAB4mGqOsljQYIjxvii7DL2oo95/aVma0Pf/DNwG3sPnzQ77dFUh7RP9x7zey3oTgn901r25LL+wbAzKqAp4FT6MX94gmkfS8AB0iaJCkfmAM8HHNMnSZpkKSS9DxwEvA60TbMDdXmAg/FE2FW2or9YWCOpAJJk4ADgOdjiK/T0n/UwT8Q7Rvo59siScDtwJtm9uOMRTm3b9rallzcN5KGSyoN80XAicBb9OZ+ifvKgf4+AacSXZnxDvC9uOPpYuyTia6yeAVYmo4fqACeAJaFx/K4Y20j/vuIDh80EH1aOq+92IHvhf30NvCpuOPvxLbcA7wGvBr+mEfnyLYcQ3So41Xg5TCdmov7pp1tybl9A0wDXgoxvw78Ryjvtf3itzJxzjmXFT+E5ZxzLiueQJxzzmXFE4hzzrmseAJxzjmXFU8gzjnnsuIJxLkeJKkp4w6uL6sH7+AsaWLm3Xydi1sq7gCcG2B2WHQrCecGPB+BONcHFP0uy3Xh9xqel7R/KN9P0hPhpn1PSJoQykdKejD8tsMrkj4WukpKui383sPj4RvHzsXCE4hzPauoxSGsz2cs22pmM4GbgJ+EspuAu81sGnAvcGMovxH4k5kdRvQ7IktD+QHAz8zsEKAK+Gwvb49zbfJvojvXgyTVmtngVspXAieY2Ypw874PzKxC0iai22Q0hPJ1ZjZM0kZgnJnVZ/QxkegW3QeE55cAeWZ2de9vmXN78xGIc33H2phvq05r6jPmm/DzmC5GnkCc6zufz3hcFOb/QnSXZ4CzgGfD/BPAV2DXjwQN6asgness//TiXM8qCr8Il/aYmaUv5S2Q9BzRB7czQ9lFwB2SvgNsBL4Uyi8GbpV0HtFI4ytEd/N1rt/wcyDO9YFwDmSGmW2KOxbneoofwnLOOZcVH4E455zLio9AnHPOZcUTiHPOuax4AnHOOZcVTyDOOeey4gnEOedcVv4/aJWOHa3FHLkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure()\n",
    "plt.plot(history.history['loss'],label='Loss')\n",
    "plt.plot(history.history['val_loss'],label='Val. loss')\n",
    "plt.xlabel('Epoch')\n",
    "plt.legend()\n",
    "plt.title(\"Loss vs Epoch during training on ntuple_merged_0.h5\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test and Evaluate \n",
    "# load training file\n",
    "test_filename = 'ntuple_merged_0.h5'\n",
    "feature_array_test2, label_array_test2, gen_pt_array_test = get_features_gen(test_filename, remove_mass_pt_window=False)\n",
    "predict_array_nn = keras_model.predict(feature_array_test2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24780/24780 [==============================] - 0s 11us/step\n",
      "Neural network:\n",
      " Test_loss: 183829.230307, Test_r_square: 0.119075, Test_rmse: 343.150444\n"
     ]
    }
   ],
   "source": [
    "[nn_test_mse, nn_test_rsquare, nn_test_rmse] = keras_model.evaluate(feature_array_test2, gen_pt_array_test)\n",
    "print(\"Neural network:\\n Test_loss: %f, Test_r_square: %f, Test_rmse: %f\" %(nn_test_mse, nn_test_rsquare, nn_test_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tfvenv",
   "language": "python",
   "name": "tfvenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}