tutorials/intro/Intro_Tutorial_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro. to Snorkel: Extracting Spouse Relations from the News"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part II: Generating _and modeling_ noisy training labels\n",
    "\n",
    "In this part of the tutorial, we will write **labeling functions** which express various heuristics, patterns, and [_weak supervision_](http://hazyresearch.github.io/snorkel/blog/weak_supervision.html) strategies to label our data.\n",
    "\n",
    "In the wild, hand-labeled training data is rare and expensive. A common scenario is to have access to tons of unlabeled training data, and have some idea of how to label them programmatically. For example:\n",
    "\n",
    "* We may be able to think of text patterns that would indicate to people mention in a sentence are married, such as seeing the word \"spouse\" between the mentions.\n",
    "* We may have access to an external knowledge base that lists some pairs of famous people who are married, and can use these to noisily label some of our mention pairs, such as Barack and Michelle Obama.\n",
    "\n",
    "Our labeling functions will capture these types of strategies. We know that these labeling functions will not be perfect, and some may be quite low-quality, so we will _model_ their accuracies with a generative model, which Snorkel will help us easily apply.\n",
    "\n",
    "This will ultimately produce a single set of **noise-aware training labels**, which we will then use to train an end extraction model in the next notebook.  For more technical details of this overall approach, see our [NIPS 2016 paper](https://arxiv.org/abs/1605.07723)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "\n",
    "# TO USE A DATABASE OTHER THAN SQLITE, USE THIS LINE\n",
    "# Note that this is necessary for parallel execution amongst other things...\n",
    "# os.environ['SNORKELDB'] = 'postgres:///snorkel-intro'\n",
    "\n",
    "import numpy as np\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We repeat our definition of the `Spouse` `Candidate` subclass from Parts II and III."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.models import candidate_subclass\n",
    "\n",
    "Spouse = candidate_subclass('Spouse', ['person1', 'person2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using a _development set_\n",
    "\n",
    "In our setting here, we will use the phrase \"development set\" to refer to a set of examples (here, a subset of our training set) which we label by hand and use to help us develop and refine labeling functions.  Unlike the _test set_, which we do not look at and use for final evaluation, we can inspect the development set while writing labeling functions.\n",
    "\n",
    "In our case, we already loaded labels for a development set (`split` 1), so we can load them again now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import load_gold_labels\n",
    "\n",
    "L_gold_dev = load_gold_labels(session, annotator_name='gold', split=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating and Modeling a Noisy Training Set\n",
    "\n",
    "Our biggest step in the data programming pipeline is the creation - _and modeling_ - of a noisy training set.  We'll approach this in three main steps:\n",
    "\n",
    "1. **Creating labeling functions (LFs):** This is where most of our development time would actually go into if this were a real application. Labeling functions encode our heuristics and weak supervision signals to generate (noisy) labels for our training candidates.\n",
    "\n",
    "2. **Applying the LFs:** Here, we actually use them to label our candidates!\n",
    "\n",
    "3. **Training a generative model of our training set:** Here we learn a model over our LFs, learning their respective accuracies automatically. This will allow us to combine them into a single, higher-quality label set.\n",
    "\n",
    "We'll also add some detail on how to go about _developing labeling functions_ and then _debugging our model_ of them to improve performance.\n",
    "\n",
    "## 1. Creating Labeling Functions\n",
    "\n",
    "In Snorkel, our primary interface through which we provide training signal to the end extraction model we are training is by writing **labeling functions (LFs)** (as opposed to hand-labeling massive training sets).  We'll go through some examples for our spouse extraction task below.\n",
    "\n",
    "A labeling function isn't anything special. It's just a Python function that accepts a `Candidate` as the input argument and returns `1` if it says the `Candidate` should be marked as true, `-1` if it says the `Candidate` should be marked as false, and `0` if it doesn't know how to vote and abstains. In practice, many labeling functions are unipolar: it labels only `1`s and `0`s, or it labels only `-1`s and `0`s.\n",
    "\n",
    "Recall that our goal is to ultimately train a high-performance classification model that predicts which of our `Candidate`s are true mentions of spouse relations.  It turns out that we can do this by writing potentially low-quality labeling functions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "from snorkel.lf_helpers import (\n",
    "    get_left_tokens, get_right_tokens, get_between_tokens,\n",
    "    get_text_between, get_tagged_text,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pattern-based LFs\n",
    "These LFs express some common sense text patterns which indicate that a person pair might be married. For example, `LF_husband_wife` looks for words in `spouses` between the person mentions, and `LF_same_last_name` checks to see if the two people have the same last name (but aren't the same whole name)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "spouses = {'spouse', 'wife', 'husband', 'ex-wife', 'ex-husband'}\n",
    "family = {'father', 'mother', 'sister', 'brother', 'son', 'daughter',\n",
    "              'grandfather', 'grandmother', 'uncle', 'aunt', 'cousin'}\n",
    "family = family | {f + '-in-law' for f in family}\n",
    "other = {'boyfriend', 'girlfriend' 'boss', 'employee', 'secretary', 'co-worker'}\n",
    "\n",
    "# Helper function to get last name\n",
    "def last_name(s):\n",
    "    name_parts = s.split(' ')\n",
    "    return name_parts[-1] if len(name_parts) > 1 else None    \n",
    "\n",
    "def LF_husband_wife(c):\n",
    "    return 1 if len(spouses.intersection(get_between_tokens(c))) > 0 else 0\n",
    "\n",
    "def LF_husband_wife_left_window(c):\n",
    "    if len(spouses.intersection(get_left_tokens(c[0], window=2))) > 0:\n",
    "        return 1\n",
    "    elif len(spouses.intersection(get_left_tokens(c[1], window=2))) > 0:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "    \n",
    "def LF_same_last_name(c):\n",
    "    p1_last_name = last_name(c.person1.get_span())\n",
    "    p2_last_name = last_name(c.person2.get_span())\n",
    "    if p1_last_name and p2_last_name and p1_last_name == p2_last_name:\n",
    "        if c.person1.get_span() != c.person2.get_span():\n",
    "            return 1\n",
    "    return 0\n",
    "\n",
    "def LF_no_spouse_in_sentence(c):\n",
    "    return -1 if np.random.rand() < 0.75 and len(spouses.intersection(c.get_parent().words)) == 0 else 0\n",
    "\n",
    "def LF_and_married(c):\n",
    "    return 1 if 'and' in get_between_tokens(c) and 'married' in get_right_tokens(c) else 0\n",
    "    \n",
    "def LF_familial_relationship(c):\n",
    "    return -1 if len(family.intersection(get_between_tokens(c))) > 0 else 0\n",
    "\n",
    "def LF_family_left_window(c):\n",
    "    if len(family.intersection(get_left_tokens(c[0], window=2))) > 0:\n",
    "        return -1\n",
    "    elif len(family.intersection(get_left_tokens(c[1], window=2))) > 0:\n",
    "        return -1\n",
    "    else:\n",
    "        return 0\n",
    "\n",
    "def LF_other_relationship(c):\n",
    "    return -1 if len(other.intersection(get_between_tokens(c))) > 0 else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distant Supervision LFs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to writing labeling functions that describe text pattern-based heuristics for labeling training examples, we can also write labeling functions that distantly supervise examples. Here, we'll load in a list of known spouse pairs and check to see if the candidate pair matches one of these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import bz2\n",
    "\n",
    "# Function to remove special characters from text\n",
    "def strip_special(s):\n",
    "    return ''.join(c for c in s if ord(c) < 128)\n",
    "\n",
    "# Read in known spouse pairs and save as set of tuples\n",
    "with bz2.BZ2File('data/spouses_dbpedia.csv.bz2', 'rb') as f:\n",
    "    known_spouses = set(\n",
    "        tuple(strip_special(x).strip().split(',')) for x in f.readlines()\n",
    "    )\n",
    "# Last name pairs for known spouses\n",
    "last_names = set([(last_name(x), last_name(y)) for x, y in known_spouses if last_name(x) and last_name(y)])\n",
    "    \n",
    "def LF_distant_supervision(c):\n",
    "    p1, p2 = c.person1.get_span(), c.person2.get_span()\n",
    "    return 1 if (p1, p2) in known_spouses or (p2, p1) in known_spouses else 0\n",
    "\n",
    "def LF_distant_supervision_last_names(c):\n",
    "    p1, p2 = c.person1.get_span(), c.person2.get_span()\n",
    "    p1n, p2n = last_name(p1), last_name(p2)\n",
    "    return 1 if (p1 != p2) and ((p1n, p2n) in last_names or (p2n, p1n) in last_names) else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For later convenience we group the labeling functions into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "LFs = [\n",
    "    LF_distant_supervision, LF_distant_supervision_last_names, \n",
    "    LF_husband_wife, LF_husband_wife_left_window, LF_same_last_name,\n",
    "    LF_no_spouse_in_sentence, LF_and_married, LF_familial_relationship, \n",
    "    LF_family_left_window, LF_other_relationship\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Developing Labeling Functions\n",
    "\n",
    "Above, we've written a bunch of labeling functions already, which should give you some sense about how to go about it. While writing them, we probably want to check to make sure that they at least work as intended before adding to our set. Suppose we're thinking about writing a simple LF: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def LF_wife_in_sentence(c):\n",
    "    \"\"\"A simple example of a labeling function\"\"\"\n",
    "    return 1 if 'wife' in c.get_parent().words else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One simple thing we can do is quickly test it on our development set (or any other set), without saving it to the database.  This is simple to do. For example, we can easily get every candidate that this LF labels as true:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Number labeled:', 276)\n"
     ]
    }
   ],
   "source": [
    "labeled = []\n",
    "for c in session.query(Spouse).filter(Spouse.split == 1).all():\n",
    "    if LF_wife_in_sentence(c) != 0:\n",
    "        labeled.append(c)\n",
    "print(\"Number labeled:\", len(labeled))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then easily put this into the Viewer as usual (try it out!):\n",
    "```\n",
    "SentenceNgramViewer(labeled, session)\n",
    "```\n",
    "\n",
    "We also have a simple helper function for getting the empirical accuracy of a single LF with respect to the development set labels for example. This function also returns the evaluation buckets of the candidates (true positive, false positive, true negative, false negative):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "Scores (Un-adjusted)\n",
      "========================================\n",
      "Pos. class accuracy: 1.0\n",
      "Neg. class accuracy: 0.0\n",
      "Precision            0.239\n",
      "Recall               1.0\n",
      "F1                   0.386\n",
      "----------------------------------------\n",
      "TP: 66 | FP: 210 | TN: 0 | FN: 0\n",
      "========================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from snorkel.lf_helpers import test_LF\n",
    "tp, fp, tn, fn = test_LF(session, LF_wife_in_sentence, split=1, annotator_name='gold')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Applying the Labeling Functions\n",
    "\n",
    "Next, we need to actually run the LFs over all of our training candidates, producing a set of `Labels` and `LabelKeys` (just the names of the LFs) in the database.  We'll do this using the `LabelAnnotator` class, a UDF which we will again run with `UDFRunner`.  **Note that this will delete any existing `Labels` and `LabelKeys` for this candidate set.**  We start by setting up the class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import LabelAnnotator\n",
    "labeler = LabelAnnotator(lfs=LFs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we run the `labeler`. Note that we set a random seed for reproducibility, since some of the LFs involve random number generators. Again, this can be run in parallel, given an appropriate database like Postgres is being used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clearing existing...\n",
      "Running UDF...\n",
      "[========================================] 100%\n",
      "\n",
      "CPU times: user 2min 12s, sys: 957 ms, total: 2min 13s\n",
      "Wall time: 2min 16s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<22195x10 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 22215 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(1701)\n",
    "%time L_train = labeler.apply(split=0)\n",
    "L_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we've already created the labels (saved in the database), we can load them in as a sparse matrix here too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<22195x10 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 22215 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_train = labeler.load_matrix(session, split=0)\n",
    "L_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the returned matrix is a special subclass of the `scipy.sparse.csr_matrix` class, with some special features which we demonstrate below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Spouse(Span(\"Earl\", sentence=40643, chars=[8,11], words=[2,2]), Span(\"Robert\", sentence=40643, chars=[0,5], words=[0,0]))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_train.get_candidate(session, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LabelKey (LF_distant_supervision)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_train.get_key(session, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also view statistics about the resulting label matrix.\n",
    "\n",
    "* **Coverage** is the fraction of candidates that the labeling function emits a non-zero label for.\n",
    "* **Overlap** is the fraction candidates that the labeling function emits a non-zero label for and that another labeling function emits a non-zero label for.\n",
    "* **Conflict** is the fraction candidates that the labeling function emits a non-zero label for and that another labeling function emits a *conflicting* non-zero label for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>j</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Overlaps</th>\n",
       "      <th>Conflicts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LF_distant_supervision</th>\n",
       "      <td>0</td>\n",
       "      <td>0.001442</td>\n",
       "      <td>0.001397</td>\n",
       "      <td>0.000451</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_distant_supervision_last_names</th>\n",
       "      <td>1</td>\n",
       "      <td>0.008245</td>\n",
       "      <td>0.008065</td>\n",
       "      <td>0.004596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_husband_wife</th>\n",
       "      <td>2</td>\n",
       "      <td>0.103942</td>\n",
       "      <td>0.066501</td>\n",
       "      <td>0.017797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_husband_wife_left_window</th>\n",
       "      <td>3</td>\n",
       "      <td>0.077675</td>\n",
       "      <td>0.057535</td>\n",
       "      <td>0.010813</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_same_last_name</th>\n",
       "      <td>4</td>\n",
       "      <td>0.017076</td>\n",
       "      <td>0.015589</td>\n",
       "      <td>0.010273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_no_spouse_in_sentence</th>\n",
       "      <td>5</td>\n",
       "      <td>0.604506</td>\n",
       "      <td>0.083082</td>\n",
       "      <td>0.010228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_and_married</th>\n",
       "      <td>6</td>\n",
       "      <td>0.000991</td>\n",
       "      <td>0.000766</td>\n",
       "      <td>0.000631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_familial_relationship</th>\n",
       "      <td>7</td>\n",
       "      <td>0.104348</td>\n",
       "      <td>0.091913</td>\n",
       "      <td>0.021491</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_family_left_window</th>\n",
       "      <td>8</td>\n",
       "      <td>0.073305</td>\n",
       "      <td>0.067538</td>\n",
       "      <td>0.012165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_other_relationship</th>\n",
       "      <td>9</td>\n",
       "      <td>0.009371</td>\n",
       "      <td>0.006938</td>\n",
       "      <td>0.001126</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   j  Coverage  Overlaps  Conflicts\n",
       "LF_distant_supervision             0  0.001442  0.001397   0.000451\n",
       "LF_distant_supervision_last_names  1  0.008245  0.008065   0.004596\n",
       "LF_husband_wife                    2  0.103942  0.066501   0.017797\n",
       "LF_husband_wife_left_window        3  0.077675  0.057535   0.010813\n",
       "LF_same_last_name                  4  0.017076  0.015589   0.010273\n",
       "LF_no_spouse_in_sentence           5  0.604506  0.083082   0.010228\n",
       "LF_and_married                     6  0.000991  0.000766   0.000631\n",
       "LF_familial_relationship           7  0.104348  0.091913   0.021491\n",
       "LF_family_left_window              8  0.073305  0.067538   0.012165\n",
       "LF_other_relationship              9  0.009371  0.006938   0.001126"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_train.lf_stats(session)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Fitting the Generative Model\n",
    "Now, we'll train a model of the LFs to estimate their accuracies. Once the model is trained, we can combine the outputs of the LFs into a single, noise-aware training label set for our extractor. Intuitively, we'll model the LFs by observing how they overlap and conflict with each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inferred cardinality: 2\n"
     ]
    }
   ],
   "source": [
    "from snorkel.learning import GenerativeModel\n",
    "\n",
    "gen_model = GenerativeModel()\n",
    "gen_model.train(L_train, epochs=100, decay=0.95, step_size=0.1 / L_train.shape[0], reg_param=1e-6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.07365367,  0.07356034,  0.12076262,  0.1127853 ,  0.07221189,\n",
       "        0.69236749,  0.07425521,  0.15913109,  0.13906755,  0.08099007])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen_model.weights.lf_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now apply the generative model to the training candidates to get the noise-aware training label set. We'll refer to these as the training marginals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_marginals = gen_model.marginals(L_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll look at the distribution of the training marginals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEr9JREFUeJzt3XGs3eV93/H3Z/ZCSSJSCLcuu6azq7hrwUrUcstYOlXZ\n3Am3aWqiUequHV5rgTpYm07rOrNJzaTJEtGmtUMqTFaSYboorkXTYjVxG2SaRVtryCVhBUMoboBg\nD/AtSeMl20hMvvvjPpTDfezc43Ou77lc3i/p6Dzn+T3P73wfzr187u/3O+c4VYUkSYP+2qQLkCSt\nPIaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOmsnXcCoLr744tqwYcOky5Ck15QH\nH3zwL6pqarFxi4ZDkg8DPw6cqKrNre/fA+8Bvg78OfBzVfWXbdstwE7gJeCXquoPW/8VwJ3A+cAn\ngPdVVSU5D7gLuAJ4Afipqnpqsbo2bNjA7OzsYsMkSQOSPD3MuGFOK90JbF3Qdy+wuareDvwZcEt7\n0suA7cDlbc7tSda0OXcANwCb2u3lfe4EvlxVbwN+HfjAMIVLks6dRcOhqj4NfGlB3yer6lR7eBhY\n39rbgH1V9WJVPQkcBa5McglwQVUdrvlv+rsLuGZgzt7WvhvYkiTjLEqSNJ6luCD988DB1p4GnhnY\ndqz1Tbf2wv5XzWmB8xXgrUtQlyRpRGOFQ5J/A5wCPrI05Sz6fDcmmU0yOzc3txxPKUmvSyOHQ5J/\nwvyF6p+pV/5RiOPApQPD1re+47xy6mmw/1VzkqwF3sL8helOVe2pqpmqmpmaWvRiuyRpRCOFQ5Kt\nwK8CP1FV/2dg0wFge5Lzkmxk/sLzA1X1LHAyyVXtesL1wD0Dc3a09rXAfeW/QCRJEzXMW1k/CrwL\nuDjJMeD9zL876Tzg3nbt+HBV/UJVHUmyH3iU+dNNN1fVS21XN/HKW1kP8sp1ig8Bv5XkKPMXvrcv\nzdIkSaPKa/WP9JmZmfJzDpJ0dpI8WFUzi43z6zMkSZ3X7NdnvB5t2PXxkec+deu7l7ASSaudRw6S\npI7hIEnqGA6SpI7hIEnqGA6SpI7hIEnqGA6SpI7hIEnqGA6SpI7hIEnqGA6SpI7hIEnqGA6SpI7h\nIEnqGA6SpI7hIEnqGA6SpI7hIEnqGA6SpI7hIEnqGA6SpI7hIEnqGA6SpI7hIEnqGA6SpM6i4ZDk\nw0lOJHlkoO+iJPcmeaLdXziw7ZYkR5M8nuTqgf4rkjzctt2WJK3/vCS/3frvT7JhaZcoSTpbwxw5\n3AlsXdC3CzhUVZuAQ+0xSS4DtgOXtzm3J1nT5twB3ABsareX97kT+HJVvQ34deADoy5GkrQ0Fg2H\nqvo08KUF3duAva29F7hmoH9fVb1YVU8CR4Erk1wCXFBVh6uqgLsWzHl5X3cDW14+qpAkTcao1xzW\nVdWzrf0csK61p4FnBsYda33Trb2w/1VzquoU8BXgrad70iQ3JplNMjs3Nzdi6ZKkxYx9QbodCdQS\n1DLMc+2pqpmqmpmamlqOp5Sk16VRw+H5dqqIdn+i9R8HLh0Yt771HW/thf2vmpNkLfAW4IUR65Ik\nLYFRw+EAsKO1dwD3DPRvb+9A2sj8hecH2imok0muatcTrl8w5+V9XQvc145GJEkTsnaxAUk+CrwL\nuDjJMeD9wK3A/iQ7gaeB6wCq6kiS/cCjwCng5qp6qe3qJubf+XQ+cLDdAD4E/FaSo8xf+N6+JCuT\nJI1s0XCoqp8+w6YtZxi/G9h9mv5ZYPNp+v8f8JOL1SFJWj5+QlqS1DEcJEkdw0GS1DEcJEkdw0GS\n1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEc\nJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEkdw0GS1DEcJEmdscIhyT9PciTJI0k+muTbklyU\n5N4kT7T7CwfG35LkaJLHk1w90H9FkofbttuSZJy6JEnjGTkckkwDvwTMVNVmYA2wHdgFHKqqTcCh\n9pgkl7XtlwNbgduTrGm7uwO4AdjUbltHrUuSNL5xTyutBc5PshZ4I/C/gG3A3rZ9L3BNa28D9lXV\ni1X1JHAUuDLJJcAFVXW4qgq4a2COJGkCRg6HqjoO/Afgi8CzwFeq6pPAuqp6tg17DljX2tPAMwO7\nONb6plt7Yb8kaULGOa10IfNHAxuBvwG8KcnPDo5pRwI1VoWvfs4bk8wmmZ2bm1uq3UqSFhjntNKP\nAE9W1VxVfQP4GPBO4Pl2qoh2f6KNPw5cOjB/fes73toL+ztVtaeqZqpqZmpqaozSJUnfyjjh8EXg\nqiRvbO8u2gI8BhwAdrQxO4B7WvsAsD3JeUk2Mn/h+YF2Cupkkqvafq4fmCNJmoC1o06sqvuT3A18\nFjgFfA7YA7wZ2J9kJ/A0cF0bfyTJfuDRNv7mqnqp7e4m4E7gfOBgu0mSJmTkcACoqvcD71/Q/SLz\nRxGnG78b2H2a/llg8zi1SJKWjp+QliR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdw\nkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1\nDAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1xgqHJN+e5O4kn0/yWJK/k+SiJPcmeaLdXzgw/pYk\nR5M8nuTqgf4rkjzctt2WJOPUJUkaz7hHDv8J+IOq+l7gHcBjwC7gUFVtAg61xyS5DNgOXA5sBW5P\nsqbt5w7gBmBTu20dsy5J0hhGDockbwF+GPgQQFV9var+EtgG7G3D9gLXtPY2YF9VvVhVTwJHgSuT\nXAJcUFWHq6qAuwbmSJImYJwjh43AHPBfknwuyQeTvAlYV1XPtjHPAetaexp4ZmD+sdY33doL+ztJ\nbkwym2R2bm5ujNIlSd/KOOGwFvgB4I6q+n7ga7RTSC9rRwI1xnO8SlXtqaqZqpqZmppaqt1KkhYY\nJxyOAceq6v72+G7mw+L5dqqIdn+ibT8OXDowf33rO97aC/slSRMycjhU1XPAM0n+VuvaAjwKHAB2\ntL4dwD2tfQDYnuS8JBuZv/D8QDsFdTLJVe1dStcPzJEkTcDaMef/IvCRJG8AvgD8HPOBsz/JTuBp\n4DqAqjqSZD/zAXIKuLmqXmr7uQm4EzgfONhukqQJGSscquohYOY0m7acYfxuYPdp+meBzePUIkla\nOn5CWpLUMRwkSR3DQZLUMRwkSR3DQZLUMRwkSR3DQZLUMRwkSR3DQZLUMRwkSR3DQZLUGfeL9ySt\nEht2fXzkuU/d+u4lrEQrgUcOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO\n4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6owdDknWJPlckt9vjy9Kcm+SJ9r9hQNjb0lyNMnjSa4e6L8i\nycNt221JMm5dkqTRLcWRw/uAxwYe7wIOVdUm4FB7TJLLgO3A5cBW4PYka9qcO4AbgE3ttnUJ6pIk\njWiscEiyHng38MGB7m3A3tbeC1wz0L+vql6sqieBo8CVSS4BLqiqw1VVwF0DcyRJEzDukcNvAL8K\nfHOgb11VPdvazwHrWnsaeGZg3LHWN93aC/slSRMycjgk+XHgRFU9eKYx7UigRn2O0zznjUlmk8zO\nzc0t1W4lSQuMc+TwQ8BPJHkK2Af8/ST/FXi+nSqi3Z9o448Dlw7MX9/6jrf2wv5OVe2pqpmqmpma\nmhqjdEnStzJyOFTVLVW1vqo2MH+h+b6q+lngALCjDdsB3NPaB4DtSc5LspH5C88PtFNQJ5Nc1d6l\ndP3AHEnSBKw9B/u8FdifZCfwNHAdQFUdSbIfeBQ4BdxcVS+1OTcBdwLnAwfbTZI0IUsSDlX1KeBT\nrf0CsOUM43YDu0/TPwtsXopaJEnj8xPSkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ\n6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgO\nkqSO4SBJ6hgOkqSO4SBJ6hgOkqSO4SBJ6hgOkqTOyOGQ5NIkf5Tk0SRHkryv9V+U5N4kT7T7Cwfm\n3JLkaJLHk1w90H9FkofbttuSZLxlSZLGMc6RwyngX1TVZcBVwM1JLgN2AYeqahNwqD2mbdsOXA5s\nBW5Psqbt6w7gBmBTu20doy5J0phGDoeqeraqPtva/xt4DJgGtgF727C9wDWtvQ3YV1UvVtWTwFHg\nyiSXABdU1eGqKuCugTmSpAlYkmsOSTYA3w/cD6yrqmfbpueAda09DTwzMO1Y65tu7YX9p3ueG5PM\nJpmdm5tbitIlSacxdjgkeTPwO8AvV9XJwW3tSKDGfY6B/e2pqpmqmpmamlqq3UqSFhgrHJL8deaD\n4SNV9bHW/Xw7VUS7P9H6jwOXDkxf3/qOt/bCfknShIzzbqUAHwIeq6r/OLDpALCjtXcA9wz0b09y\nXpKNzF94fqCdgjqZ5Kq2z+sH5kiSJmDtGHN/CPjHwMNJHmp9/xq4FdifZCfwNHAdQFUdSbIfeJT5\ndzrdXFUvtXk3AXcC5wMH202SNCEjh0NV/XfgTJ9H2HKGObuB3afpnwU2j1qLJGlp+QlpSVLHcJAk\ndQwHSVLHcJAkdcZ5t5IkAbBh18fHmv/Ure9eokq0VDxykCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJ\nUsdwkCR1/JyDpNe0cT5j4ecrzswjB0lSxyOHEfiXiqTVziMHSVLHcJAkdQwHSVLndXnNYdxvkJSk\n1c4jB0lSx3CQJHVel6eVpHPJf/hGq4FHDpKkjuEgSep4Wkmrlp9kl0bnkYMkqbNiwiHJ1iSPJzma\nZNek65Gk17MVcVopyRrgN4F/ABwDPpPkQFU9OtnKlp4fwNO55M+XlsqKCAfgSuBoVX0BIMk+YBuw\n6sJBWoz/g9dKsFLCYRp4ZuDxMeBvT6iWVcmLs2fH/0Evr0n99/b34sxWSjgMJcmNwI3t4VeTPH6W\nu7gY+IulrWoilnUd+cA527Wvx8riOs7COfy9GHQu1vI3hxm0UsLhOHDpwOP1re9VqmoPsGfUJ0ky\nW1Uzo85fKVzHyuI6VpbVsg6Y7FpWyruVPgNsSrIxyRuA7cCBCdckSa9bK+LIoapOJflnwB8Ca4AP\nV9WRCZclSa9bKyIcAKrqE8AnzvHTjHxKaoVxHSuL61hZVss6YIJrSVVN6rklSSvUSrnmIElaQVZd\nOCz2NRxJvjfJnyR5McmvTKLGYQ2xlp9J8qdJHk7yx0neMYk6FzPEOra1dTyUZDbJ351EnYsZ9ite\nkvxgklNJrl3O+oY1xOvxriRfaa/HQ0l+bRJ1LmaY16Ot5aEkR5L8t+WucRhDvB7/cuC1eCTJS0ku\nOueFVdWquTF/MfvPge8G3gD8T+CyBWO+A/hBYDfwK5Ouecy1vBO4sLV/FLh/0nWPuI4388opzrcD\nn5903aOsY2DcfcxfP7t20nWP+Hq8C/j9Sde6BOv4dua/ZeG72uPvmHTdo/5cDYx/D3DfctS22o4c\n/uprOKrq68DLX8PxV6rqRFV9BvjGJAo8C8Os5Y+r6svt4WHmPx+y0gyzjq9W+8kH3gSsxAthi66j\n+UXgd4ATy1ncWRh2HSvdMOv4R8DHquqLMP+7v8w1DuNsX4+fBj66HIWttnA43ddwTE+olnGd7Vp2\nAgfPaUWjGWodSd6b5PPAx4GfX6bazsai60gyDbwXuGMZ6zpbw/5cvbOd6juY5PLlKe2sDLOO7wEu\nTPKpJA8muX7Zqhve0L/nSd4IbGX+j49zbsW8lVWjS/L3mA+HFXmufhhV9bvA7yb5YeDfAT8y4ZJG\n8RvAv6qqbyaZdC3j+Czzp2K+muTHgN8DNk24plGsBa4AtgDnA3+S5HBV/dlkyxrZe4D/UVVfWo4n\nW23hMNTXcLxGDLWWJG8HPgj8aFW9sEy1nY2zek2q6tNJvjvJxVW1kr7nZ5h1zAD7WjBcDPxYklNV\n9XvLU+JQFl1HVZ0caH8iye2v0dfjGPBCVX0N+FqSTwPvAFZSOJzN78d2lumUErDqLkivBb4AbOSV\nizuXn2Hsv2VlX5BedC3AdwFHgXdOut4x1/E2Xrkg/QPM/3Jk0rWP+rPVxt/JyrwgPczr8Z0Dr8eV\nwBdfi68H8H3AoTb2jcAjwOZJ1z7KzxXwFuBLwJuWq7ZVdeRQZ/gajiS/0Lb/5yTfCcwCFwDfTPLL\nzL874OQZdzwBw6wF+DXgrcDt7a/VU7XCvnBsyHX8Q+D6JN8A/i/wU9V+I1aKIdex4g25jmuBf5rk\nFPOvx/bX4utRVY8l+QPgT4FvAh+sqkcmV3XvLH6u3gt8suaPgpaFn5CWJHVW27uVJElLwHCQJHUM\nB0lSx3CQJHUMB0lSx3CQJHUMB0lSx3CQJHX+P+RTZqf1r3tBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x120222550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.hist(train_marginals, bins=20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can view the learned accuracy parameters, and other statistics about the LFs learned by the generative model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.534331</td>\n",
       "      <td>0.6685</td>\n",
       "      <td>0.533610</td>\n",
       "      <td>0.359681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.537407</td>\n",
       "      <td>0.6710</td>\n",
       "      <td>0.533373</td>\n",
       "      <td>0.357285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.566811</td>\n",
       "      <td>0.6683</td>\n",
       "      <td>0.572629</td>\n",
       "      <td>0.380838</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.544246</td>\n",
       "      <td>0.6622</td>\n",
       "      <td>0.543686</td>\n",
       "      <td>0.351497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.531741</td>\n",
       "      <td>0.6616</td>\n",
       "      <td>0.529915</td>\n",
       "      <td>0.346507</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.805757</td>\n",
       "      <td>0.7156</td>\n",
       "      <td>0.803320</td>\n",
       "      <td>0.579641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.530631</td>\n",
       "      <td>0.6660</td>\n",
       "      <td>0.532365</td>\n",
       "      <td>0.351297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.580750</td>\n",
       "      <td>0.6743</td>\n",
       "      <td>0.587330</td>\n",
       "      <td>0.388623</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.576654</td>\n",
       "      <td>0.6725</td>\n",
       "      <td>0.570839</td>\n",
       "      <td>0.386826</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.544443</td>\n",
       "      <td>0.6739</td>\n",
       "      <td>0.541886</td>\n",
       "      <td>0.369261</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Accuracy  Coverage  Precision    Recall\n",
       "0  0.534331    0.6685   0.533610  0.359681\n",
       "1  0.537407    0.6710   0.533373  0.357285\n",
       "2  0.566811    0.6683   0.572629  0.380838\n",
       "3  0.544246    0.6622   0.543686  0.351497\n",
       "4  0.531741    0.6616   0.529915  0.346507\n",
       "5  0.805757    0.7156   0.803320  0.579641\n",
       "6  0.530631    0.6660   0.532365  0.351297\n",
       "7  0.580750    0.6743   0.587330  0.388623\n",
       "8  0.576654    0.6725   0.570839  0.386826\n",
       "9  0.544443    0.6739   0.541886  0.369261"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen_model.learned_lf_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Model to Iterate on Labeling Functions\n",
    "\n",
    "Now that we have learned the generative model, we can stop here and use this to potentially debug and/or improve our labeling function set.  First, we apply the LFs to our development set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clearing existing...\n",
      "Running UDF...\n",
      "[========================================] 100%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "L_dev = labeler.apply_existing(split=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, we get the score of the generative model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "Scores (Un-adjusted)\n",
      "========================================\n",
      "Pos. class accuracy: 0.546\n",
      "Neg. class accuracy: 0.932\n",
      "Precision            0.378\n",
      "Recall               0.546\n",
      "F1                   0.447\n",
      "----------------------------------------\n",
      "TP: 107 | FP: 176 | TN: 2424 | FN: 89\n",
      "========================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tp, fp, tn, fn = gen_model.error_analysis(session, L_dev, L_gold_dev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting Generative Model Performance\n",
    "\n",
    "At this point, we should be getting an F1 score of around 0.4 to 0.5 on the development set, which is pretty good!  However, we should be very careful in interpreting this. Since we developed our labeling functions using this development set as a guide, and our generative model is composed of these labeling functions, we expect it to score very well here!  \n",
    "\n",
    "In fact, it is probably somewhat _overfit_ to this set. However this is fine, since in the next tutorial, we'll train a more powerful end extraction model which will generalize beyond the development set, and which we will evaluate on a _blind_ test set (i.e. one we never looked at during development)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Doing Some Error Analysis\n",
    "\n",
    "At this point, we might want to look at some examples in one of the error buckets. For example, one of the false negatives that we did not correctly label as true mentions.  To do this, we can again just use the `Viewer`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "require.undef('viewer');\n",
       "\n",
       "// NOTE: all elements should be selected using this.$el.find to avoid collisions with other Viewers\n",
       "\n",
       "define('viewer', [\"jupyter-js-widgets\"], function(widgets) {\n",
       "    var ViewerView = widgets.DOMWidgetView.extend({\n",
       "        render: function() {\n",
       "            this.cids   = this.model.get('cids');\n",
       "            this.nPages = this.cids.length;\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Insert the html payload\n",
       "            this.$el.append(this.model.get('html'));\n",
       "\n",
       "            // Initialize all labels from previous sessions\n",
       "            this.labels = this.deserializeDict(this.model.get('_labels_serialized'));\n",
       "            for (var i=0; i < this.nPages; i++) {\n",
       "                this.pid = i;\n",
       "                for (var j=0; j < this.cids[i].length; j++) {\n",
       "                    this.cxid = j;\n",
       "                    for (var k=0; k < this.cids[i][j].length; k++) {\n",
       "                        this.cid = k;\n",
       "                        if (this.cids[i][j][k] in this.labels) {\n",
       "                            this.markCurrentCandidate(false);\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Enable button functionality for navigation\n",
       "            var that = this;\n",
       "            this.$el.find(\"#next-cand\").click(function() {\n",
       "                that.switchCandidate(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-cand\").click(function() {\n",
       "                that.switchCandidate(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-context\").click(function() {\n",
       "                that.switchContext(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-context\").click(function() {\n",
       "                that.switchContext(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-page\").click(function() {\n",
       "                that.switchPage(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-page\").click(function() {\n",
       "                that.switchPage(-1);\n",
       "            });\n",
       "            this.$el.find(\"#label-true\").click(function() {\n",
       "                that.labelCandidate(true, true);\n",
       "            });\n",
       "            this.$el.find(\"#label-false\").click(function() {\n",
       "                that.labelCandidate(false, true);\n",
       "            });\n",
       "\n",
       "            // Arrow key functionality\n",
       "            this.$el.keydown(function(e) {\n",
       "                switch(e.which) {\n",
       "                    case 74: // j\n",
       "                    that.switchCandidate(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 73: // i\n",
       "                    that.switchPage(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 76: // l\n",
       "                    that.switchCandidate(1);\n",
       "                    break;\n",
       "\n",
       "                    case 75: // k\n",
       "                    that.switchPage(1);\n",
       "                    break;\n",
       "\n",
       "                    case 84: // t\n",
       "                    that.labelCandidate(true, true);\n",
       "                    break;\n",
       "\n",
       "                    case 70: // f\n",
       "                    that.labelCandidate(false, true);\n",
       "                    break;\n",
       "                }\n",
       "            });\n",
       "\n",
       "            // Show the first page and highlight the first candidate\n",
       "            this.$el.find(\"#viewer-page-0\").show();\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Get candidate selector for currently selected candidate, escaping id properly\n",
       "        getCandidate: function() {\n",
       "            return this.$el.find(\".\"+this.cids[this.pid][this.cxid][this.cid]);\n",
       "        },  \n",
       "\n",
       "        // Color the candidate correctly according to registered label, as well as set highlighting\n",
       "        markCurrentCandidate: function(highlight) {\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var tags = this.$el.find(\".\"+cid);\n",
       "\n",
       "            // Clear color classes\n",
       "            tags.removeClass(\"candidate-h\");\n",
       "            tags.removeClass(\"true-candidate\");\n",
       "            tags.removeClass(\"true-candidate-h\");\n",
       "            tags.removeClass(\"false-candidate\");\n",
       "            tags.removeClass(\"false-candidate-h\");\n",
       "            tags.removeClass(\"highlighted\");\n",
       "\n",
       "            if (highlight) {\n",
       "                if (cid in this.labels) {\n",
       "                    tags.addClass(String(this.labels[cid]) + \"-candidate-h\");\n",
       "                } else {\n",
       "                    tags.addClass(\"candidate-h\");\n",
       "                }\n",
       "            \n",
       "            // If un-highlighting, leave with first non-null coloring\n",
       "            } else {\n",
       "                var that = this;\n",
       "                tags.each(function() {\n",
       "                    var cids = $(this).attr('class').split(/\\s+/).map(function(item) {\n",
       "                        return parseInt(item);\n",
       "                    });\n",
       "                    cids.sort();\n",
       "                    for (var i in cids) {\n",
       "                        if (cids[i] in that.labels) {\n",
       "                            var label = that.labels[cids[i]];\n",
       "                            $(this).addClass(String(label) + \"-candidate\");\n",
       "                            $(this).removeClass(String(!label) + \"-candidate\");\n",
       "                            break;\n",
       "                        }\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "\n",
       "            // Extra highlighting css\n",
       "            if (highlight) {\n",
       "                tags.addClass(\"highlighted\");\n",
       "            }\n",
       "\n",
       "            // Classes for showing direction of relation\n",
       "            if (highlight) {\n",
       "                this.$el.find(\".\"+cid+\"-0\").addClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").addClass(\"right-candidate\");\n",
       "            } else {\n",
       "                this.$el.find(\".\"+cid+\"-0\").removeClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").removeClass(\"right-candidate\");\n",
       "            }\n",
       "        },\n",
       "\n",
       "        // Cycle through candidates and highlight, by increment inc\n",
       "        switchCandidate: function(inc) {\n",
       "            var N = this.cids[this.pid].length\n",
       "            var M = this.cids[this.pid][this.cxid].length;\n",
       "            if (N == 0 || M == 0) { return false; }\n",
       "\n",
       "            // Clear highlighting from previous candidate\n",
       "            if (inc != 0) {\n",
       "                this.markCurrentCandidate(false);\n",
       "\n",
       "                // Increment the cid counter\n",
       "\n",
       "                // Move to next context\n",
       "                if (this.cid + inc >= M) {\n",
       "                    while (this.cid + inc >= M) {\n",
       "                        \n",
       "                        // At last context on page, halt\n",
       "                        if (this.cxid == N - 1) {\n",
       "                            this.cid = M - 1;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to next context\n",
       "                        } else {\n",
       "                            inc -= M - this.cid;\n",
       "                            this.cxid += 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = 0;\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                // Move to previous context\n",
       "                } else if (this.cid + inc < 0) {\n",
       "                    while (this.cid + inc < 0) {\n",
       "                        \n",
       "                        // At first context on page, halt\n",
       "                        if (this.cxid == 0) {\n",
       "                            this.cid = 0;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to previous context\n",
       "                        } else {\n",
       "                            inc += this.cid + 1;\n",
       "                            this.cxid -= 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = M - 1;\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "\n",
       "                // Move within current context\n",
       "                this.cid += inc;\n",
       "            }\n",
       "            this.markCurrentCandidate(true);\n",
       "\n",
       "            // Push this new cid to the model\n",
       "            this.model.set('_selected_cid', this.cids[this.pid][this.cxid][this.cid]);\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Switch through contexts\n",
       "        switchContext: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "\n",
       "            // Iterate context on this page\n",
       "            var M = this.cids[this.pid].length;\n",
       "            if (this.cxid + inc < 0) {\n",
       "                this.cxid = 0;\n",
       "            } else if (this.cxid + inc >= M) {\n",
       "                this.cxid = M - 1;\n",
       "            } else {\n",
       "                this.cxid += inc;\n",
       "            }\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Switch through pages\n",
       "        switchPage: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "            this.$el.find(\".viewer-page\").hide();\n",
       "            if (this.pid + inc < 0) {\n",
       "                this.pid = 0;\n",
       "            } else if (this.pid + inc > this.nPages - 1) {\n",
       "                this.pid = this.nPages - 1;\n",
       "            } else {\n",
       "                this.pid += inc;\n",
       "            }\n",
       "            this.$el.find(\"#viewer-page-\"+this.pid).show();\n",
       "\n",
       "            // Show pagination\n",
       "            this.$el.find(\"#page\").html(this.pid);\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.cxid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Label currently-selected candidate\n",
       "        labelCandidate: function(label, highlighted) {\n",
       "            var c    = this.getCandidate();\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var cl   = String(label) + \"-candidate\";\n",
       "            var clh  = String(label) + \"-candidate-h\";\n",
       "            var cln  = String(!label) + \"-candidate\";\n",
       "            var clnh = String(!label) + \"-candidate-h\";\n",
       "\n",
       "            // Toggle label highlighting\n",
       "            if (c.hasClass(cl) || c.hasClass(clh)) {\n",
       "                c.removeClass(cl);\n",
       "                c.removeClass(clh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(\"candidate-h\");\n",
       "                }\n",
       "                this.labels[cid] = null;\n",
       "                this.send({event: 'delete_label', cid: cid});\n",
       "            } else {\n",
       "                c.removeClass(cln);\n",
       "                c.removeClass(clnh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(clh);\n",
       "                } else {\n",
       "                    c.addClass(cl);\n",
       "                }\n",
       "                this.labels[cid] = label;\n",
       "                this.send({event: 'set_label', cid: cid, value: label});\n",
       "            }\n",
       "\n",
       "            // Set the label and pass back to the model\n",
       "            this.model.set('_labels_serialized', this.serializeDict(this.labels));\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Serialization of hash maps, because traitlets Dict doesn't seem to work...\n",
       "        serializeDict: function(d) {\n",
       "            var s = [];\n",
       "            for (var key in d) {\n",
       "                s.push(key+\"~~\"+d[key]);\n",
       "            }\n",
       "            return s.join();\n",
       "        },\n",
       "\n",
       "        // Deserialization of hash maps\n",
       "        deserializeDict: function(s) {\n",
       "            var d = {};\n",
       "            var entries = s.split(/,/);\n",
       "            var kv;\n",
       "            for (var i in entries) {\n",
       "                kv = entries[i].split(/~~/);\n",
       "                if (kv[1] == \"true\") {\n",
       "                    d[kv[0]] = true;\n",
       "                } else if (kv[1] == \"false\") {\n",
       "                    d[kv[0]] = false;\n",
       "                }\n",
       "            }\n",
       "            return d;\n",
       "        },\n",
       "    });\n",
       "\n",
       "    return {\n",
       "        ViewerView: ViewerView\n",
       "    };\n",
       "});\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from snorkel.viewer import SentenceNgramViewer\n",
    "\n",
    "# NOTE: This if-then statement is only to avoid opening the viewer during automated testing of this notebook\n",
    "# You should ignore this!\n",
    "import os\n",
    "if 'CI' not in os.environ:\n",
    "    sv = SentenceNgramViewer(fn, session)\n",
    "else:\n",
    "    sv = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can save the best model from the hyperparameter search with a custom name so that we can reload it later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Spouse(Span(\"Kim\", sentence=40895, chars=[0,2], words=[0,0]), Span(\"Kanye\", sentence=40895, chars=[8,12], words=[2,2]))"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = sv.get_selected() if sv else list(fp.union(fn))[0]\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily see the labels that the LFs gave to this candidate using simple ORM-enabled syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Label (LF_no_spouse_in_sentence = -1)]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also now explore some of the additional functionalities of the `lf_stats` method for our dev set LF labels, `L_dev`: we can plug in the gold labels that we have, and the accuracies that our generative model has learned:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>j</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Overlaps</th>\n",
       "      <th>Conflicts</th>\n",
       "      <th>TP</th>\n",
       "      <th>FP</th>\n",
       "      <th>FN</th>\n",
       "      <th>TN</th>\n",
       "      <th>Empirical Acc.</th>\n",
       "      <th>Learned Acc.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LF_distant_supervision</th>\n",
       "      <td>0</td>\n",
       "      <td>0.001073</td>\n",
       "      <td>0.001073</td>\n",
       "      <td>0.000715</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.542395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_distant_supervision_last_names</th>\n",
       "      <td>1</td>\n",
       "      <td>0.006438</td>\n",
       "      <td>0.006080</td>\n",
       "      <td>0.004292</td>\n",
       "      <td>7</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.388889</td>\n",
       "      <td>0.530454</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_husband_wife</th>\n",
       "      <td>2</td>\n",
       "      <td>0.091917</td>\n",
       "      <td>0.061874</td>\n",
       "      <td>0.018956</td>\n",
       "      <td>97</td>\n",
       "      <td>160</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.377432</td>\n",
       "      <td>0.568175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_husband_wife_left_window</th>\n",
       "      <td>3</td>\n",
       "      <td>0.070100</td>\n",
       "      <td>0.055436</td>\n",
       "      <td>0.013948</td>\n",
       "      <td>86</td>\n",
       "      <td>110</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.438776</td>\n",
       "      <td>0.567417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_same_last_name</th>\n",
       "      <td>4</td>\n",
       "      <td>0.024320</td>\n",
       "      <td>0.020386</td>\n",
       "      <td>0.016452</td>\n",
       "      <td>19</td>\n",
       "      <td>49</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.279412</td>\n",
       "      <td>0.530790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_no_spouse_in_sentence</th>\n",
       "      <td>5</td>\n",
       "      <td>0.612303</td>\n",
       "      <td>0.095136</td>\n",
       "      <td>0.016452</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>57</td>\n",
       "      <td>1655</td>\n",
       "      <td>0.966706</td>\n",
       "      <td>0.805315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_and_married</th>\n",
       "      <td>6</td>\n",
       "      <td>0.002146</td>\n",
       "      <td>0.001431</td>\n",
       "      <td>0.001431</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.537142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_familial_relationship</th>\n",
       "      <td>7</td>\n",
       "      <td>0.117668</td>\n",
       "      <td>0.103004</td>\n",
       "      <td>0.029685</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>314</td>\n",
       "      <td>0.954407</td>\n",
       "      <td>0.585114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_family_left_window</th>\n",
       "      <td>8</td>\n",
       "      <td>0.074750</td>\n",
       "      <td>0.065808</td>\n",
       "      <td>0.012876</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>208</td>\n",
       "      <td>0.995215</td>\n",
       "      <td>0.569323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_other_relationship</th>\n",
       "      <td>9</td>\n",
       "      <td>0.008584</td>\n",
       "      <td>0.007868</td>\n",
       "      <td>0.002504</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>20</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>0.542507</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   j  Coverage  Overlaps  Conflicts  TP   FP  \\\n",
       "LF_distant_supervision             0  0.001073  0.001073   0.000715   2    1   \n",
       "LF_distant_supervision_last_names  1  0.006438  0.006080   0.004292   7   11   \n",
       "LF_husband_wife                    2  0.091917  0.061874   0.018956  97  160   \n",
       "LF_husband_wife_left_window        3  0.070100  0.055436   0.013948  86  110   \n",
       "LF_same_last_name                  4  0.024320  0.020386   0.016452  19   49   \n",
       "LF_no_spouse_in_sentence           5  0.612303  0.095136   0.016452   0    0   \n",
       "LF_and_married                     6  0.002146  0.001431   0.001431   1    5   \n",
       "LF_familial_relationship           7  0.117668  0.103004   0.029685   0    0   \n",
       "LF_family_left_window              8  0.074750  0.065808   0.012876   0    0   \n",
       "LF_other_relationship              9  0.008584  0.007868   0.002504   0    0   \n",
       "\n",
       "                                   FN    TN  Empirical Acc.  Learned Acc.  \n",
       "LF_distant_supervision              0     0        0.666667      0.542395  \n",
       "LF_distant_supervision_last_names   0     0        0.388889      0.530454  \n",
       "LF_husband_wife                     0     0        0.377432      0.568175  \n",
       "LF_husband_wife_left_window         0     0        0.438776      0.567417  \n",
       "LF_same_last_name                   0     0        0.279412      0.530790  \n",
       "LF_no_spouse_in_sentence           57  1655        0.966706      0.805315  \n",
       "LF_and_married                      0     0        0.166667      0.537142  \n",
       "LF_familial_relationship           15   314        0.954407      0.585114  \n",
       "LF_family_left_window               1   208        0.995215      0.569323  \n",
       "LF_other_relationship               4    20        0.833333      0.542507  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_dev.lf_stats(session, L_gold_dev, gen_model.learned_lf_stats()['Accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that for labeling functions with low coverage, our learned accuracies are closer to our prior of 70% accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving our training labels\n",
    "\n",
    "Finally, we'll save the `training_marginals`, which are our **probabilistic training labels**, so that we can use them in the next tutorial to train our end extraction model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved 22195 marginals\n",
      "CPU times: user 11.8 s, sys: 24.6 ms, total: 11.8 s\n",
      "Wall time: 11.8 s\n"
     ]
    }
   ],
   "source": [
    "from snorkel.annotations import save_marginals\n",
    "%time save_marginals(session, L_train, train_marginals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, in Part III, we'll use these probabilistic training labels to train a deep neural network."
   ]
  }
 ],
 "metadata": {
  "anaconda-butt": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  },
  "widgets": {
   "state": {
    "5c30a7935aa84f14b282753d32a7c2a8": {
     "views": [
      {
       "cell_index": 49
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}