tutorials/advanced/Categorical_Classes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Categorical Variables in Snorkel\n",
    "\n",
    "This is a short tutorial on how to use categorical variables (i.e. more values than binary) in Snorkel.  We'll use a completely toy scenario with three sentences and two LFs just to demonstrate the mechanics. Please see the main tutorial for a more comprehensive intro!\n",
    "\n",
    "We'll **highlight in bold all parts focusing on the categorical aspect.**\n",
    "\n",
    "### Notes on Current Categorical Support:\n",
    "* The `Viewer` works in the categorical setting, _but labeling `Candidate`s in the `Viewer` does not._\n",
    "    - Instead can import test / dev set labels from e.g. BRAT\n",
    "* The `LogisticRegression` and `SparseLogisticRegression` end models have been extended to the categorical setting, but other end models in `contrib` may not have been\n",
    "    - _Note: It's simple to make this change, so feel free to post an issue with requests for other end models!_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Preprocessing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.parser import TSVDocPreprocessor, CorpusParser\n",
    "\n",
    "doc_preprocessor = TSVDocPreprocessor('data/categorical_example.tsv') \n",
    "corpus_parser = CorpusParser()\n",
    "%time corpus_parser.apply(doc_preprocessor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Defining candidates\n",
    "\n",
    "We'll define candidate relations between person mentions **that now can take on one of three values:**\n",
    "```python\n",
    "['Married', 'Employs', False]\n",
    "```\n",
    "Note the importance of including a value for \"not a relation of interest\"- here we've used `False`, but any value could do.\n",
    "Also note that `None` is a protected value -- denoting a labeling function abstaining -- so this cannot be used as a value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.models import candidate_subclass\n",
    "Relationship = candidate_subclass('Relationship', ['person1', 'person2'], values=['Married', 'Employs', False])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we extract candidates the same as in the Intro Tutorial (simplified here slightly):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.candidates import Ngrams, CandidateExtractor\n",
    "from snorkel.matchers import PersonMatcher\n",
    "from snorkel.models import Sentence\n",
    "\n",
    "# Define a Person-Person candidate extractor\n",
    "ngrams = Ngrams(n_max=3)\n",
    "person_matcher = PersonMatcher(longest_match_only=True)\n",
    "cand_extractor = CandidateExtractor(\n",
    "    Relationship, \n",
    "    [ngrams, ngrams],\n",
    "    [person_matcher, person_matcher],\n",
    "    symmetric_relations=False\n",
    ")\n",
    "\n",
    "# Apply to all (three) of the sentences for this simple example\n",
    "sents = session.query(Sentence).all()\n",
    "\n",
    "# Run the candidate extractor\n",
    "%time cand_extractor.apply(sents, split=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_cands = session.query(Relationship).filter(Relationship.split == 0).all()\n",
    "print \"Number of candidates:\", len(train_cands)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.viewer import SentenceNgramViewer\n",
    "\n",
    "# NOTE: This if-then statement is only to avoid opening the viewer during automated testing of this notebook\n",
    "# You should ignore this!\n",
    "import os\n",
    "if 'CI' not in os.environ:\n",
    "    sv = SentenceNgramViewer(train_cands, session)\n",
    "else:\n",
    "    sv = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Writing Labeling Functions\n",
    "\n",
    "**The _categorical_ labeling functions (LFs) we now write can output the following values:**\n",
    "\n",
    "* Abstain: `None` OR 0\n",
    "* Categorical values: The literal values in `Relationship.values` OR their integer indices.\n",
    "\n",
    "We'll write two simple LFs to illustrate.\n",
    "\n",
    "*Tip: we can get a random candidate (see below), or the example highlighted in the viewer above via `sv.get_selected()`, and then use this to test as we write the LFs!*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import re\n",
    "from snorkel.lf_helpers import get_between_tokens\n",
    "\n",
    "# Getting an example candidate from the Viewer\n",
    "c = train_cands[0]\n",
    "\n",
    "# Traversing the context hierarchy...\n",
    "print c.get_contexts()[0].get_parent().text\n",
    "\n",
    "# Using a helper function\n",
    "get_between_tokens(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def LF_married(c):\n",
    "    return 'Married' if 'married' in get_between_tokens(c) else None\n",
    "\n",
    "WORKPLACE_RGX = r'employ|boss|company'\n",
    "def LF_workplace(c):\n",
    "    sent = c.get_contexts()[0].get_parent()\n",
    "    matches = re.search(WORKPLACE_RGX, sent.text)\n",
    "    return 'Employs' if matches else None\n",
    "\n",
    "LFs = [\n",
    "    LF_married,\n",
    "    LF_workplace\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we apply the LFs to the candidates to produce our label matrix $L$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import LabelAnnotator\n",
    "\n",
    "labeler = LabelAnnotator(lfs=LFs)\n",
    "%time L_train = labeler.apply(split=0)\n",
    "L_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "L_train.todense()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Training the Generative Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.learning import GenerativeModel\n",
    "\n",
    "gen_model = GenerativeModel()\n",
    "\n",
    "# Note: We pass cardinality explicitly here to be safe\n",
    "# Can usually be inferred, except we have no labels with value=3\n",
    "gen_model.train(L_train, cardinality=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_marginals = gen_model.marginals(L_train)\n",
    "\n",
    "assert np.all(train_marginals.sum(axis=1) - np.ones(3) < 1e-10)\n",
    "train_marginals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can save the training marginals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import save_marginals, load_marginals\n",
    "\n",
    "save_marginals(session, L_train, train_marginals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then reload (e.g. in another notebook):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "load_marginals(session, L_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Training the End Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create features and train the `LogisticRegression` model, which now will use a softmax to handle the categorical setting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import FeatureAnnotator\n",
    "featurizer = FeatureAnnotator()\n",
    "\n",
    "%time F_train = featurizer.apply(split=0)\n",
    "F_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.learning import SparseLogisticRegression\n",
    "disc_model_sparse = SparseLogisticRegression(cardinality=Relationship.cardinality)\n",
    "\n",
    "disc_model_sparse.train(F_train, train_marginals, n_epochs=100, print_freq=10, lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_labels = [1, 2, 1]\n",
    "correct, incorrect = disc_model_sparse.error_analysis(session, F_train, train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print \"Accuracy:\", disc_model_sparse.score(F_train, train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_marginals = disc_model_sparse.marginals(F_train)\n",
    "test_marginals"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}