C3_W1_Lab_3_logistic_regression_model_interpretation__M.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression Model Interpretation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome to this exercise! You'll review how to interpret the coefficients in a logistic regression model. \n",
    " - The logistic regression is considered a **Generalized Linear Model**. \n",
    " - In general, you would employ one of these models to interpret the relationship between variables. \n",
    " - The logistic regression can be interpreted in terms of the **odds** and **OR** (Odds Ratio), which you'll learn about here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries that you will use in this notebook\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>obstruct</th>\n",
       "      <th>outcome</th>\n",
       "      <th>TRTMT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>68</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>72</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>66</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>69</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sex  age  obstruct  outcome  TRTMT\n",
       "1    0   57         0        1   True\n",
       "2    1   68         0        0  False\n",
       "3    0   72         0        0   True\n",
       "4    0   66         1        1   True\n",
       "5    1   69         0        1  False"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in the data\n",
    "data = pd.read_csv(\"data/dummy_data.csv\", index_col=0)\n",
    "\n",
    "# View a few rows of the data\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a description of all the fields:\n",
    "\n",
    "- `sex (binary): 1 if Male, 0 if Female`\n",
    "- `age (int): age of patient at the beginning of the study`\n",
    "- `obstruct (binary): obstruction of colon by tumor`\n",
    "- `outcome (binary): 1 if patient died within 5 years`\n",
    "- `TRTMT (binary): if patient was treated`\n",
    "\n",
    "You'll want to pay close attention to the `TRTMT` and `outcome` columns. \n",
    "- `TRTMT`: Whether a treatment was given or not.\n",
    "- `outcome`: To measure the effective of treatment, you'll have the 5-year survival rate.  This is stored in the `outcome` variable, which is a binary variable with two possible values.  1 indicates that the patient died, and 0 indicates that the patient did not die during the 5-year period."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "\n",
    "The formula for computing a logistic regression has the following form:\n",
    "\n",
    "$$\\sigma(\\theta^T x^{(i)}) = \\frac{1}{1 + e^{\\left(-\\theta^T x^{(i)}\\right)}},$$\n",
    "\n",
    "$x^{(i)}$ refers to example 'i' (a particular patient, or generally, a single row in a data table).\n",
    "\n",
    "$\\theta^T x^{(i)} = \\sum_{j} \\theta_j x^{(i)}_j$ is the linear combination of the features $x_1^{(i)}$, $x_2^{(i)}$, $x_2^{(i)}$ etc., weighted by the coefficients $\\theta_1$, $\\theta_2$, $\\theta_3$ etc.\n",
    "\n",
    "So for this example, $\\theta^T x^{(i)} = \\theta_{TRTMT} x^{(i)}_{TRTMT} + \\theta_{AGE}x_{AGE}^{(i)} + \\theta_{SEX}x^{(i)}_{SEX}$\n",
    "\n",
    "Also, $\\sigma$ is the sigmoid function, defined as $\\sigma(a) = \\frac{1}{1 + e^{(-a)}}$ for some variable $a$.  The output of the sigmoid function ranges from 0 to 1, so it's useful. in representing probabilities (whose values also range from 0 to 1).\n",
    "\n",
    "If $x^{(i)}$ is the input vector and $OUTCOME$ is the target variable, then $\\sigma(\\theta^T x^{(i)})$ models the probability of death within 5 years.\n",
    "\n",
    "For example, if the data has three features, $TRTMT$, $AGE$, and  $SEX$, then the patient's probability of death is estimated by: \n",
    "\n",
    "$$Prob(OUTCOME=1) = \\sigma(\\theta^T x^{(i)}) = \\frac{1}{1 + e^{\\left(-\\theta_{TRTMT} x^{(i)}_{TRTMT} - \\theta_{AGE}x_{AGE}^{(i)} - \\theta_{SEX}x^{(i)}_{SEX}\\right)}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit the Model\n",
    "\n",
    "Let's separate the data into the target variable and the features and fit a logistic regression to it. Notice that in this case you are **not separating the data into train and test sets** because you're interested in the **interpretation of the model**, not its predictive capabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the labels\n",
    "y = data.outcome\n",
    "\n",
    "# Get the features (exclude the label)\n",
    "X = data.drop('outcome', axis=1)\n",
    "\n",
    "# Fit the logistic regression on the features and labels\n",
    "classifier = LogisticRegression(solver='lbfgs').fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Odds\n",
    "Looking at the underlying equation, you can't interpret the model in the same way as with a regular linear regression.\n",
    "- With a linear regression such as $y = 2x$ if the $x$ increases by 1 unit, then $y$ increases by 2 units.\n",
    "- How do you interpret the coefficient of a logistic regression model now that there is a sigmoid function?\n",
    "\n",
    "Let's introduce the concept of **odds**, and you'll see how this helps with the interpretation of the logistic regression.\n",
    "\n",
    "If an outcome is binary (either an event happens or the event doesn't happen):\n",
    "- Let $p$ represent the probability of the event (such as death).\n",
    "- Let $1-p$ represent the probability that the event doesn't happen (no death).\n",
    "- The odds are the probability of the event divided by 1 minus the probability of the event:\n",
    "\n",
    "$$\\mathrm{odds} = \\frac{p}{1-p}$$\n",
    "\n",
    "Going back to the logistic regression, recall that the sigmoid function $\\sigma$ ranges between 0 and 1, and so it's a useful function for representing a probability.\n",
    "- So, let $p$, the probability of event, be estimated by $\\sigma(\\theta^T x^{(i)})$.\n",
    "\n",
    "The **odds** defined in terms of the probability of an event $p$ are:\n",
    "\n",
    "$$\\mathrm{odds} = \\frac{p}{1 - p}$$\n",
    "\n",
    "Substitute $p = \\sigma(\\theta^T x^{(i)})$ to get:\n",
    "$$\\mathrm{odds} = \\frac{\\sigma(\\theta^T x)}{1 - \\sigma(\\theta^T x)}$$\n",
    "\n",
    "Substitute for the definition of sigmoid: $\\sigma(\\theta^T x^{(i)}) = \\frac{1}{1 + e^{(-\\theta^T x)}}$\n",
    "$$\\mathrm{odds} = \\frac{\\frac{1}{1 + e^{(-\\theta^T x)}}}{1 - \\frac{1}{1 + e^{(-\\theta^T x)}}} $$\n",
    "\n",
    "Multiply top and bottom by $1 + e^{(-\\theta^T x)}$ and simplify to get:\n",
    "$$\\mathrm{odds} = \\frac{1}{\\left ( 1 + e^{(-\\theta^T x)} \\right)  - (1)}$$\n",
    "\n",
    "Do some more cleanup to get:\n",
    "$$\\mathrm{odds} = e^{\\left(\\theta^T x^{(i)}\\right)}$$\n",
    "\n",
    "So what is this saying?\n",
    "- The odds (probability of death divided by probability of not death) can be estimated using the features and their coefficients if you take the dot product of the coefficients and features, then exponentiate that dot product (take e to the power of the dot product).\n",
    "\n",
    "Since working with the exponential of something isn't necessarily easier to think about, you can take one additional transformation to get rid of the exponential, coming up next."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logit\n",
    "\n",
    "Note that the inverse function of exponentiation is the natural log\n",
    "- $\\mathrm{log}(e^{a}) = a$\n",
    "- $e^{(\\mathrm{log}(a))} = a$\n",
    "\n",
    "So if you want to \"remove\" the exponential $e$, you can apply the natural log function, which we'll write as $\\mathrm{log}$.  You may have seen natural log written as $\\mathrm{ln}$ as well, but we'll use $\\mathrm{log}$ because Python functions usually name natural log functions as `log`.\n",
    "\n",
    "Note that the log of odds is defined as the **logit** function:\n",
    "$$\\mathrm{logit}(a) = \\mathrm{log}\\frac{a}{1-a}$$\n",
    "\n",
    "Apply the $\\mathrm{log}()$ to the odds:\n",
    "\n",
    "$$\\text{logit} = \\log(\\text{odds}) = \\log\\left(\\frac{p}{1 - p}\\right)= \\log\\left( e^{\\left(\\theta^T x^{(i)}\\right)}\\right) = \\theta^T x^{(i)}$$\n",
    "\n",
    "So, what's nice about this?\n",
    "- The right side of this equation is now a weighted sum of the features in $x^{(i)}$, weighted by coefficients in $\\theta^T$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpreting the Coefficient's Effect on the Logit\n",
    "\n",
    "This is an improvement in the interpretability of your model. \n",
    "- Now you can interpret a single coefficient $\\theta_j$ in a similar way that you interpret the coefficient in regular linear regression.\n",
    "\n",
    "For a small example, let's say the coefficient for age is 0.2, patient A has age=40, and the logit for patient A is 3.\n",
    "$$\\text{logit} = \\theta_{age} \\times x_{age} + \\cdots $$\n",
    "\n",
    "Patient A (now)  \n",
    "$$ 3 = \\theta_{age} \\times 40 + \\cdots $$\n",
    "\n",
    "If you increase patient A's age by 1 year, then this increases the logit by 0.2 (which is the coefficient for age).  \n",
    "Patient (A one year older):\n",
    "$$ 3 + 0.2 = 0.2 \\times (40 + 1) + \\cdots $$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The range of possible values for the logit\n",
    "A nice feature of the logit (log odds) is the range of possible values it can have. The $\\mathrm{logit}$ function can be any real number between $-\\infty$ and $+\\infty$.\n",
    "\n",
    "One way to see this is to look at the ranges of values for the sigmoid, the odds, and then logit.\n",
    "- The sigmoid $\\sigma(a)$ ranges from 0 to 1 for a variable $a$.  Recall that we're letting $p = \\sigma(a)$\n",
    "- The odds $\\frac{p}{1-p}$ can be as small as 0 (when $p=0$) and as large as $+\\infty$ (when $p \\rightarrow 1$).  So the odds range from 0 to $+\\infty$.\n",
    "- $\\mathrm{log}(\\mathrm{odds})$ can range from $-\\infty$ (when the odds are 0), to $+\\infty$ (when the odds approach $+\\infty$).  \n",
    "- So the range of the log odds is $-\\infty$ to $+\\infty$\n",
    "\n",
    "To check the coefficients of the model, you can use the model's `coef_ attribute`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.21704833,  0.0460642 ,  0.37798496, -0.418984  ]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the coefficients (the thetas, or weights for each feature)\n",
    "thetas = classifier.coef_\n",
    "thetas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will return a numpy array containing the coefficient for each feature variable. Let's print it in a nicer way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature sex      : coefficient = -0.217048 \n",
      "Feature age      : coefficient = 0.046064  \n",
      "Feature obstruct : coefficient = 0.377985  \n",
      "Feature TRTMT    : coefficient = -0.418984 \n"
     ]
    }
   ],
   "source": [
    "# Print the name of the feature and the coefficient for each feature\n",
    "for i in range(len(X.columns)):\n",
    "    print(\"Feature {:<9s}: coefficient = {:<10f}\".format(X.columns[i], thetas[0, i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The coefficient for age is `0.046`.  This means that when the `age` variable increases by one, the logit will increase in `0.046`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Odds Ratio\n",
    "\n",
    "In order to fully leverage the information that the odds provide, there's one more very useful concept:\n",
    "the **\"Odds Ratio\"**, which we will write as OR for short.\n",
    "\n",
    "The OR allows you to compare the odds of one situation versus another (by dividing one odds by another odds).\n",
    "\n",
    "When computing the OR for binary variables, it's defined as the odds when the variable is 1 divided by the odds when the variable is 0. For example:\n",
    "\n",
    "$$OR_{TRTMT} = \\frac{\\text{odds}(TRTMT=1)}{\\text{odds}(TRTMT=0)}$$\n",
    "\n",
    "In contrast, when computing the OR for continuous variables, it's defined as the ratio between the odds of the variable plus one unit and the odds of the variable. For example:\n",
    "\n",
    "$$OR_{age} = \\frac{\\text{odds}(age+1)}{\\text{odds}(age)}$$\n",
    "\n",
    "For both cases, after applying the appropriate algebra to the formula, you should find that the **OR** for a variable is equal to $e$ to the power of the coefficient associated with it.\n",
    "\n",
    "$$OR_{x_j} = e^{\\theta_j}$$\n",
    "\n",
    "Let's try it for the variable $AGE$:\n",
    "\n",
    "$$OR_{age} = \\frac{odds(age+1)}{odds(age)} = \\frac{e^\\left(\\theta_{INTERCEPT} + \\theta_{SEX}x^{(i)}_{SEX} + \\theta_{AGE}(1+x_{AGE}^{(i)}) + \\theta_{OBSTRUCT}x_{OBSTRUCT}^{(i)} + \\theta_{TRTMT} x^{(i)}_{TRTMT}\\right)}{e^\\left(\\theta_{INTERCEPT} + \\theta_{SEX}x^{(i)}_{SEX} + \\theta_{AGE}x_{AGE}^{(i)} + \\theta_{OBSTRUCT}x_{OBSTRUCT}^{(i)} + \\theta_{TRTMT} x^{(i)}_{TRTMT}\\right)} = e^{\\theta_{AGE}}$$\n",
    "\n",
    "<br>\n",
    "\n",
    "The case for binary variables is quite similar. You can see the derivation for the binary variable `TRTMT` in this week's graded assignment.\n",
    "\n",
    "Now, let's compute the ORs for the feature variables:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute the Odds Ratios"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.80489107, 1.04714164, 1.45934099, 0.65771472]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute Odds Ratios for each feature\n",
    "odds_ratios = np.exp(thetas)\n",
    "odds_ratios"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature sex       : coefficient = -0.217048  // OR = 0.80\n",
      "Feature age       : coefficient = 0.046064   // OR = 1.05\n",
      "Feature obstruct  : coefficient = 0.377985   // OR = 1.46\n",
      "Feature TRTMT     : coefficient = -0.418984  // OR = 0.66\n"
     ]
    }
   ],
   "source": [
    "# Display the coefficient and odds ratio for each feature\n",
    "for i in range(len(X.columns)):\n",
    "    print(\"Feature {:<10s}: coefficient = {:<10f} // OR = {:.2f}\".format(X.columns[i], thetas[0, i], odds_ratios[0, i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpret Odds Ratio\n",
    "\n",
    "- The features with a negative coefficient has an Odds Ratio that is smaller than 1 \n",
    "- and features with non-negative coefficients have ORs greater than 1.\n",
    "\n",
    "A negative coefficient (like treatment) indicates that the feature `TRTMT` reduces the outcome (reduces risk of death), but it's not easy to interpret how much the coefficient value of `-0.419` actually reduces the risk of death.\n",
    "\n",
    "However, now that you have calculated the Odds Ratio of `0.66` for `TRTMT`, you can interpret this value:\n",
    "- If the patient does not receive treatment, and let's assume that their odds of dying is some value like 2.\n",
    "- If the patient **does** receive treatment, then the patient's the odds of dying is $0.66 \\times 2$.\n",
    "- In other words, the odds of dying if given treatment is 0.66 times the odds of dying if not receiving treatment.\n",
    "\n",
    "As you can see from the Odds Ratios, the give data indicates that the following reduce risk of death:\n",
    "- Receiving treatment\n",
    "- Not presenting a colon obstruction\n",
    "- Being of younger age\n",
    "- Being male."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations on finishing this notebook!** By now, you should have a clearer understanding of how to interpret a Logistic Regression model as well as a better intuition for odds and odd ratios. **Great job!**\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}