01_clean.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Political Alignment Case Study\n",
    "\n",
    "Allen Downey\n",
    "\n",
    "[MIT License](https://en.wikipedia.org/wiki/MIT_License)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This is the first in a series of notebooks that make up a case study in exploratory data analysis.\n",
    "\n",
    "In this notebook, we \n",
    "\n",
    "1. Read data from the General Social Survey (GSS),\n",
    "\n",
    "2. Clean the data, particularly dealing with special codes that indicate missing data,\n",
    "\n",
    "3. Validate the data by comparing the values in the dataset with values documented in the codebook.\n",
    "\n",
    "4. Generate \"resampled\" datasets that correct for deliberate oversampling in the dataset, and\n",
    "\n",
    "5. Store the resampled data in a binary format (HDF5) that makes it easier to work with in the notebooks that follow this one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell loads the packages we need.  If everything works, there should be no error messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading the data\n",
    "\n",
    "The data we'll use is from the General Social Survey (GSS).  Using the [GSS Data Explorer](https://gssdataexplorer.norc.org/projects/52787), I selected a subset of the variables in the GSS and made it available along with this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the data file\n",
    "\n",
    "import os\n",
    "\n",
    "if not os.path.exists('gss_eda.tar.gz'):\n",
    "    !wget https://github.com/AllenDowney/PoliticalAlignmentCaseStudy/raw/master/gss_eda.tar.gz\n",
    "    !tar -xzf gss_eda.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, Pandas can read data in most standard formats, including CSV, Excel, Stata, and SPSS.  \n",
    "    \n",
    "Unfortunately, the current version of Pandas cannot read the data generated by GSS.\n",
    "\n",
    "As a workaround, I wrote functions to read the Stata dictionary file and use the information there to read the Stata data file using `pd.read_fwf`, which reads fixed-width files.\n",
    "\n",
    "The follow function reads the dictionary file and returns a Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def read_stata_dict(fp, **options):\n",
    "    \"\"\"Reads a Stata dictionary file.\n",
    "\n",
    "    fp: open file pointer\n",
    "    options: dict of options passed to open()\n",
    "\n",
    "    returns: DataFrame\n",
    "    \"\"\"\n",
    "    type_map = dict(byte=int, int=int, long=int, float=float,\n",
    "                    double=float, numeric=float)\n",
    "\n",
    "    var_info = []\n",
    "    for line in fp:\n",
    "        match = re.search(r'_column\\(([^)]*)\\)', line)\n",
    "        if not match:\n",
    "            continue\n",
    "        start = int(match.group(1))\n",
    "        t = line.split()\n",
    "        vtype, name, fstring = t[1:4]\n",
    "        name = name.lower()\n",
    "        if vtype.startswith('str'):\n",
    "            vtype = str\n",
    "        else:\n",
    "            vtype = type_map[vtype]\n",
    "        long_desc = ' '.join(t[4:]).strip('\"')\n",
    "        var_info.append((start, vtype, name, fstring, long_desc))\n",
    "\n",
    "    columns = ['start', 'type', 'name', 'fstring', 'desc']\n",
    "    variables = pd.DataFrame(var_info, columns=columns)\n",
    "\n",
    "    # fill in the end column by shifting the start column\n",
    "    # NOTE: choose a large number for the end of the last column\n",
    "    variables['end'] = variables.start.shift(-1, fill_value=100000)\n",
    "\n",
    "    return variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can read the dictionary file, `GSS.dct`; the result is a DataFrame with information about the variables in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start</th>\n",
       "      <th>type</th>\n",
       "      <th>name</th>\n",
       "      <th>fstring</th>\n",
       "      <th>desc</th>\n",
       "      <th>end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>3281</td>\n",
       "      <td>&lt;class 'float'&gt;</td>\n",
       "      <td>avoidbuy</td>\n",
       "      <td>%20f</td>\n",
       "      <td>Boycotted products for pol reasons</td>\n",
       "      <td>3301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>165</th>\n",
       "      <td>3301</td>\n",
       "      <td>&lt;class 'float'&gt;</td>\n",
       "      <td>income</td>\n",
       "      <td>%20f</td>\n",
       "      <td>Total family income</td>\n",
       "      <td>3321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>166</th>\n",
       "      <td>3321</td>\n",
       "      <td>&lt;class 'float'&gt;</td>\n",
       "      <td>rincome</td>\n",
       "      <td>%20f</td>\n",
       "      <td>Respondents income</td>\n",
       "      <td>3341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>167</th>\n",
       "      <td>3341</td>\n",
       "      <td>&lt;class 'float'&gt;</td>\n",
       "      <td>realrinc</td>\n",
       "      <td>%20f</td>\n",
       "      <td>Rs income in constant $</td>\n",
       "      <td>3361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>168</th>\n",
       "      <td>3361</td>\n",
       "      <td>&lt;class 'float'&gt;</td>\n",
       "      <td>china</td>\n",
       "      <td>%20f</td>\n",
       "      <td>Liking for china</td>\n",
       "      <td>100000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     start             type      name fstring  \\\n",
       "164   3281  <class 'float'>  avoidbuy    %20f   \n",
       "165   3301  <class 'float'>    income    %20f   \n",
       "166   3321  <class 'float'>   rincome    %20f   \n",
       "167   3341  <class 'float'>  realrinc    %20f   \n",
       "168   3361  <class 'float'>     china    %20f   \n",
       "\n",
       "                                    desc     end  \n",
       "164  Boycotted products for pol reasons     3301  \n",
       "165                  Total family income    3321  \n",
       "166                   Respondents income    3341  \n",
       "167              Rs income in constant $    3361  \n",
       "168                     Liking for china  100000  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('GSS.dct') as fp:\n",
    "    variables = read_stata_dict(fp)\n",
    "\n",
    "variables.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to use `pd.read_fwf` to read the data file, so we have to get the variable information into the required format.\n",
    "\n",
    "I'll extract the \"column specifications\", `start` and `end`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>21</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>41</td>\n",
       "      <td>61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>61</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>81</td>\n",
       "      <td>101</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   start  end\n",
       "0      1   21\n",
       "1     21   41\n",
       "2     41   61\n",
       "3     61   81\n",
       "4     81  101"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "colspecs = variables[['start', 'end']]\n",
    "colspecs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>3281</td>\n",
       "      <td>3301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>165</th>\n",
       "      <td>3301</td>\n",
       "      <td>3321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>166</th>\n",
       "      <td>3321</td>\n",
       "      <td>3341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>167</th>\n",
       "      <td>3341</td>\n",
       "      <td>3361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>168</th>\n",
       "      <td>3361</td>\n",
       "      <td>100000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     start     end\n",
       "164   3281    3301\n",
       "165   3301    3321\n",
       "166   3321    3341\n",
       "167   3341    3361\n",
       "168   3361  100000"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "colspecs = variables[['start', 'end']]\n",
    "colspecs.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the names of the variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       year\n",
       "1        id_\n",
       "2     agewed\n",
       "3    divorce\n",
       "4       sibs\n",
       "Name: name, dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names = variables['name']\n",
    "names.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use `pd.read_fwf` to read the data file `GSS.dat`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('GSS.dat') as fp:\n",
    "    gss = pd.read_fwf(fp,\n",
    "                      colspecs=colspecs.values.tolist(),\n",
    "                      names=names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `shape` and `head` to see what the dataset looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(64814, 169)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>id_</th>\n",
       "      <th>agewed</th>\n",
       "      <th>divorce</th>\n",
       "      <th>sibs</th>\n",
       "      <th>childs</th>\n",
       "      <th>age</th>\n",
       "      <th>educ</th>\n",
       "      <th>paeduc</th>\n",
       "      <th>maeduc</th>\n",
       "      <th>...</th>\n",
       "      <th>fejobaff</th>\n",
       "      <th>discaffm</th>\n",
       "      <th>discaffw</th>\n",
       "      <th>fehire</th>\n",
       "      <th>meovrwrk</th>\n",
       "      <th>avoidbuy</th>\n",
       "      <th>income</th>\n",
       "      <th>rincome</th>\n",
       "      <th>realrinc</th>\n",
       "      <th>china</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1972</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>23</td>\n",
       "      <td>16</td>\n",
       "      <td>10</td>\n",
       "      <td>97</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1972</td>\n",
       "      <td>2</td>\n",
       "      <td>21</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>70</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1972</td>\n",
       "      <td>3</td>\n",
       "      <td>20</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>48</td>\n",
       "      <td>12</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1972</td>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>27</td>\n",
       "      <td>17</td>\n",
       "      <td>16</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1972</td>\n",
       "      <td>5</td>\n",
       "      <td>22</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>61</td>\n",
       "      <td>12</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 169 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   year  id_  agewed  divorce  sibs  childs  age  educ  paeduc  maeduc  ...  \\\n",
       "0  1972    1       0        0     3       0   23    16      10      97  ...   \n",
       "1  1972    2      21        2     4       5   70    10       8       8  ...   \n",
       "2  1972    3      20        2     5       4   48    12       8       8  ...   \n",
       "3  1972    4      24        2     5       0   27    17      16      12  ...   \n",
       "4  1972    5      22        2     2       2   61    12       8       8  ...   \n",
       "\n",
       "   fejobaff  discaffm  discaffw  fehire  meovrwrk  avoidbuy  income  rincome  \\\n",
       "0         0         0         0       0         0         0       0        0   \n",
       "1         0         0         0       0         0         0       0        0   \n",
       "2         0         0         0       0         0         0       0        0   \n",
       "3         0         0         0       0         0         0       0        0   \n",
       "4         0         0         0       0         0         0       0        0   \n",
       "\n",
       "   realrinc  china  \n",
       "0       0.0     -1  \n",
       "1       0.0     -1  \n",
       "2       0.0     -1  \n",
       "3       0.0     -1  \n",
       "4       0.0     -1  \n",
       "\n",
       "[5 rows x 169 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(gss.shape)\n",
    "gss.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset has 64814 rows, one for each respondent, and 166 columns, one for each variable.\n",
    "\n",
    "To read this dataset, we had to download an \"archive file\" and use `tar` to unpack the archive.\n",
    "\n",
    "I had to write a function to read the dictionary file; then we could process the results and use Pandas to read the data file.\n",
    "\n",
    "Sometimes you get lucky and loading data is easier than this; sometimes you are unlucky and it's harder.\n",
    "\n",
    "It can take time and persistence to get a dataset ready to work with.  Online resources can help you figure out the details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validation\n",
    "\n",
    "Now that we've got the data loaded, it is important to validate it, which means checking for errors.\n",
    "\n",
    "The kinds of errors you have to check for depend on the nature of the data, the collection process, how the data is stored and transmitted, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this dataset, there are three kinds of validation we'll think about:\n",
    "\n",
    "1) We need to check the **integrity** of the dataset; that is, whether the data were corrupted or changed during transmission, storage, or conversion from one format to another.\n",
    "\n",
    "2) We need to check our **interpretation** of the data; for example, whether the numbers used to encode the data mean what we think they mean.\n",
    "\n",
    "3) We will also keep an eye out for data, or patterns, that might indicate problems with the survey process and the recording of the data.  For example, in a different dataset I worked with, I found a surprising number of respondents whose height was supposedly 62 centimeters.  After investigating, I concluded that they were probably 6 feet, 2 inches, and their heights were recorded incorrectly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Validating data can be a tedious process, but it is important.  If you interpret data incorrectly and publish invalid results, you will be embarrassed in the best case, and in the worst case you might do serious harm.  See [this article](https://www.vox.com/future-perfect/2019/6/4/18650969/married-women-miserable-fake-paul-dolan-happiness) for a recent example.\n",
    "\n",
    "However, I don't expect you to validate every variable in this dataset.  Instead, I will demonstrate the process, and then ask you to validate one additional variable as an exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first variable we'll validate is called `polviews`.  It records responses to the following question:\n",
    "\n",
    ">We hear a lot of talk these days about liberals and conservatives. \n",
    "I'm going to show you a seven-point scale on which the political views that people might hold are arranged from extremely liberal--point 1--to extremely conservative--point 7. Where would you place yourself on this scale?\n",
    "\n",
    "You can [read the documentation of this variable in the GSS codebook](https://gssdataexplorer.norc.org/projects/52787/variables/178/vshow)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The responses are encoded like this:\n",
    "\n",
    "```\n",
    "1\tExtremely liberal\n",
    "2\tLiberal\n",
    "3\tSlightly liberal\n",
    "4\tModerate\n",
    "5\tSlghtly conservative\n",
    "6\tConservative\n",
    "7\tExtremely conservative\n",
    "8\tDon't know\n",
    "9\tNo answer\n",
    "0\tNot applicable\n",
    "```\n",
    "\n",
    "The following function, `values`, takes a Series that represents a single variable and returns the values in the series and their frequencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def values(series):\n",
    "    \"\"\"Count the values and sort.\n",
    "    \n",
    "    series: pd.Series\n",
    "    \n",
    "    returns: series mapping from values to frequencies\n",
    "    \"\"\"\n",
    "    return series.value_counts().sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the values for the variable `polviews`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "column = gss['polviews']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     6777\n",
       "1     1682\n",
       "2     6514\n",
       "3     7010\n",
       "4    21370\n",
       "5     8690\n",
       "6     8230\n",
       "7     1832\n",
       "8     2326\n",
       "9      383\n",
       "Name: polviews, dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "values(column)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check the integrity of the data and confirm that we have loaded it correctly, we'll do a \"spot check\"; that is, we'll pick one year and compare the values we see in the dataset to the values reported in the codebook.\n",
    "\n",
    "We can select values from a single year like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "one_year = (gss['year'] == 1974)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And look at the values and their frequencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1     22\n",
       "2    201\n",
       "3    207\n",
       "4    564\n",
       "5    221\n",
       "6    160\n",
       "7     35\n",
       "8     70\n",
       "9      4\n",
       "Name: polviews, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "values(column[one_year])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you [compare these results to the values in the codebook](https://gssdataexplorer.norc.org/projects/52787/variables/178/vshow), you should see that they agree.\n",
    "\n",
    "**Exercise:** Go back and change 1974 to another year, and compare the results to the codebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing data\n",
    "\n",
    "For many variables, missing values are encoded with numerical codes that we need to replace before we do any analysis.\n",
    "\n",
    "For `polviews`, the values 8, 9, and 0 represent \"Don't know\", \"No answer\", and \"Not applicable\".\n",
    "\n",
    "\"Not applicable\" usually means the respondent was not asked a particular question.\n",
    "\n",
    "To keep things simple, we'll treat all of these values as equivalent, but we lose some information by doing that.  For example, if a respondent refuses to answer a question, that might suggest something about their answer.  If so, treating their response as missing data might bias the results.\n",
    "\n",
    "Fortunately, for most questions the number of respondents who refused to answer is small."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll replace the numeric codes 8, 9, and 0 with `NaN`, which is a special value used to indicate missing data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean = column.replace([0, 8, 9], np.nan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `notna` and `sum` to count the valid responses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55328"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean.notna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we use `isna` to count the missing responses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9486"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can [check these results against the codebook](https://gssdataexplorer.norc.org/projects/52787/variables/178/vshow); at the bottom of that page, it reports the number of \"Valid cases\" and \"Missing cases\".\n",
    "\n",
    "However, in this example, the results don't match.  The codebook reports 53081 valid cases and 9385 missing cases.\n",
    "\n",
    "To figure out what was wrong, I looked at the difference between the values in the codebook and the values I computed from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2247"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean.notna().sum() - 53081"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "101"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean.isna().sum() - 9385"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks like about one year of data, so I guessed that the numbers in the code book might not include the most recent data, from 2018.\n",
    "\n",
    "Here are the numbers from 2018."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2348"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "one_year = (gss['year'] == 2018)\n",
    "one_year.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2247"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean[one_year].notna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "101"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean[one_year].isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like my hypothesis is correct; the summary statistics in the codebook do not include data from 2018.\n",
    "\n",
    "Based on these checks, it looks like the dataset is intact and we have loaded it correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Replacing missing data\n",
    "\n",
    "For the other variables in this dataset, I read through the code book and identified the special values that indicate missing data.\n",
    "\n",
    "I recorded that information in the following function, which is intended to replace special values with `NaN`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gss_replace_invalid(df):\n",
    "    \"\"\"Replace invalid data with NaN.\n",
    "    \n",
    "    df: DataFrame\n",
    "    \"\"\"\n",
    "    df.educ.replace([98, 99], np.nan, inplace=True)\n",
    "    \n",
    "    df.age.replace([98, 99], np.nan, inplace=True)      # 89 means 89 or older\n",
    "    df.cohort.replace([0, 9999], np.nan, inplace=True)\n",
    "    df.marcohrt.replace([0, 9999], np.nan, inplace=True)\n",
    "    \n",
    "    df.cappun.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.gunlaw.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.divlaw.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.pornlaw.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.racopen.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.grass.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.letdie1.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.prayer.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.affrmact.replace([0, 8, 9], np.nan, inplace=True)\n",
    "\n",
    "    df.abany.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.abdefect.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.abnomore.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.abhlth.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.abpoor.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.abrape.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.absingle.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    \n",
    "    df.fepol.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.fejobaff.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.fehire.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.fechld.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.fepresch.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.fefam.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.discaffm.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.discaffw.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.meovrwrk.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    \n",
    "    df.sexeduc.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.premarsx.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.xmarsex.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.teensex.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.homosex.replace([0, 5, 8, 9], np.nan, inplace=True)  # 5 is other\n",
    "\n",
    "    df.racmar.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.spanking.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.racpres.replace([0, 8, 9], np.nan, inplace=True)\n",
    "\n",
    "    df.fear.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.databank.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.happy.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.hapmar.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    \n",
    "    df.natroad.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natsoc.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natmass.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natpark.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natchld.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natsci.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natenrgy.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natspac.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natenvir.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natheal.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natcity.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natcrime.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natcrimy.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natdrug.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.nateduc.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natrace.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natarms.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.nataid.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.natfare.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    \n",
    "    df.health.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.life.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    \n",
    "    df.helpful.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.fair.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.trust.replace([0, 8, 9], np.nan, inplace=True)\n",
    "\n",
    "    df.conclerg.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.coneduc.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.confed.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.conpress.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.conjudge.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.conlegis.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.conarmy.replace([0, 8, 9], np.nan, inplace=True)\n",
    "\n",
    "    df.confinan.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.conbus.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.conlabor.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.conmedic.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.contv.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.consci.replace([0, 8, 9], np.nan, inplace=True)  \n",
    "\n",
    "    df.spkhomo.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.spkath.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.spkmslm.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.spkrac.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.spkcom.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.spkmil.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    \n",
    "    df.colhomo.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.colath.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.colmslm.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.colrac.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.colcom.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.colmil.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    \n",
    "    df.libhomo.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.libath.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.libmslm.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.librac.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.libcom.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.libmil.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    \n",
    "    df.satjob.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.satfin.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.finrela.replace([0, 8, 9], np.nan, inplace=True)\n",
    "\n",
    "    df.union_.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.res16.replace([0, 8, 9], np.nan, inplace=True)\n",
    "\n",
    "    df.fund.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.memchurh.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.fund16.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.reliten.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.pray.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.sprel16.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    \n",
    "    df.bible.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.postlife.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.god.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.reborn.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.savesoul.replace([0, 8, 9], np.nan, inplace=True)\n",
    "\n",
    "    df.relpersn.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.sprtprsn.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.relexp.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.relactiv.replace([0, 98, 89], np.nan, inplace=True)\n",
    "    df.attend.replace([9], np.nan, inplace=True)\n",
    "\n",
    "    df.compuse.replace([0, 8, 9], np.nan, inplace=True)\n",
    "\n",
    "    df.degree.replace([8, 9], np.nan, inplace=True)\n",
    "    df.padeg.replace([8, 9], np.nan, inplace=True)\n",
    "    df.madeg.replace([8, 9], np.nan, inplace=True)\n",
    "    df.spdeg.replace([8, 9], np.nan, inplace=True)\n",
    "    df.partyid.replace([8, 9], np.nan, inplace=True)\n",
    "    df.polviews.replace([0, 8, 9], np.nan, inplace=True)\n",
    "\n",
    "    df.chldidel.replace([-1, 8, 9], np.nan, inplace=True)\n",
    "    df.childs.replace([9], np.nan, inplace=True)\n",
    "    df.adults.replace([9], np.nan, inplace=True)\n",
    "\n",
    "    df.divorce.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.age.replace([0, 98, 99], np.nan, inplace=True)\n",
    "    df.agewed.replace([0, 98, 99], np.nan, inplace=True)\n",
    "    df.relig.replace([0, 98, 99], np.nan, inplace=True)\n",
    "    df.relig16.replace([0, 98, 99], np.nan, inplace=True)\n",
    "    df.income.replace([0, 13, 98, 99], np.nan, inplace=True)\n",
    "    df.rincome.replace([0, 13, 98, 99], np.nan, inplace=True)\n",
    "    df.realinc.replace([0], np.nan, inplace=True)                  \n",
    "    df.realrinc.replace([0], np.nan, inplace=True)                  \n",
    "    \n",
    "    # note: sibs contains some unlikely numbers\n",
    "    df.sibs.replace([-1, 98, 99], np.nan, inplace=True)\n",
    "    df.educ.replace([97, 98, 99], np.nan, inplace=True)\n",
    "    df.maeduc.replace([97, 98, 99], np.nan, inplace=True)\n",
    "    df.paeduc.replace([97, 98, 99], np.nan, inplace=True)\n",
    "    df.speduc.replace([97, 98, 99], np.nan, inplace=True)\n",
    "\n",
    "    df.phone.replace([0, 2, 9], np.nan, inplace=True)\n",
    "    df.owngun.replace([0, 3, 8, 9], np.nan, inplace=True)\n",
    "    df.pistol.replace([0, 3, 8, 9], np.nan, inplace=True)\n",
    "    df.hunt.replace([0, 8, 9], np.nan, inplace=True)\n",
    "\n",
    "    df.class_.replace([0, 5, 8, 9], np.nan, inplace=True)\n",
    "    df.pres04.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.pres08.replace([0, 8, 9], np.nan, inplace=True)\n",
    "    df.pres12.replace([0, 8, 9], np.nan, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "gss_replace_invalid(gss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# try to make the dataset smaller by replacing\n",
    "# 64-bit FP numbers with 32-bit.\n",
    "\n",
    "for varname in gss.columns:\n",
    "    if gss[varname].dtype == np.float64:\n",
    "        gss[varname] = gss[varname].astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, I have only moderate confidence that this code is correct.  I'm not sure I have dealt with every variable in the dataset, and I'm not sure that the special values for every variable are correct.\n",
    "\n",
    "So I will ask for your help."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: In order to validate the other variables, I'd like each person who works with this notebook to validate one variable.\n",
    "\n",
    "If you run the following cell, it will choose one of the columns from the dataset at random.  That's the variable you will check.\n",
    "\n",
    "If you get `year` or `id_`, run the cell again to get a different variable name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'fear'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(None)\n",
    "np.random.choice(gss.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go back through the previous two sections of this notebook and replace `polviews` with your randomly chosen variable.  Then run the cells again and go to [this online survey to report the results](https://forms.gle/tmST8YCu4qLc414F7). \n",
    "\n",
    "Note: Not all questions were asked during all years.  If your variable doesn't have data for 1974 or 2018, you might have to choose different years."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resampling\n",
    "\n",
    "The GSS uses stratified sampling, which means that some groups are deliberately oversampled to help with statistical validity.\n",
    "\n",
    "As a result, each respondent has a sampling weight which is proportional to the number of people in the population they represent.\n",
    "\n",
    "Before running any analysis, we can compensate for stratified sampling by \"resampling\", that is, by drawing a random sample from the dataset, where each respondent's chance of appearing in the sample is proportional to their sampling weight."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def resample_rows_weighted(df, column):\n",
    "    \"\"\"Resamples a DataFrame using probabilities proportional to given column.\n",
    "\n",
    "    df: DataFrame\n",
    "    column: string column name to use as weights\n",
    "\n",
    "    returns: DataFrame\n",
    "    \"\"\"\n",
    "    weights = df[column]\n",
    "    sample = df.sample(n=len(df), replace=True, weights=weights)\n",
    "    return sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def resample_by_year(df, column):\n",
    "    \"\"\"Resample rows within each year.\n",
    "\n",
    "    df: DataFrame\n",
    "    column: string name of weight variable\n",
    "\n",
    "    returns DataFrame\n",
    "    \"\"\"\n",
    "    grouped = df.groupby('year')\n",
    "    samples = [resample_rows_weighted(group, column)\n",
    "               for _, group in grouped]\n",
    "    sample = pd.concat(samples, ignore_index=True)\n",
    "    return sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(19)\n",
    "sample = resample_by_year(gss, 'wtssall')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving the results\n",
    "\n",
    "I'll save the results to an HDF5 file, which is a binary format that makes it much faster to read the data back.\n",
    "\n",
    "First I'll save the original (not resampled) data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An HDF5 file is like a dictionary on disk.  It contains keys and corresponding values.\n",
    "\n",
    "`to_hdf` takes three arguments:\n",
    "\n",
    "* The filename, `gss_eda.hdf5`.\n",
    "\n",
    "* The key, `gss`\n",
    "\n",
    "* The compression level, which controls how hard the algorithm works to compress the file.\n",
    "\n",
    "So this file contains a single key, `gss`, which maps to the DataFrame with the original GSS data.\n",
    "\n",
    "With compression level `3`, it reduces the size of the file by a factor of 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if the file already exists, remove it\n",
    "\n",
    "import os\n",
    "\n",
    "if os.path.isfile('gss_eda.hdf5'):\n",
    "    !rm gss_eda.hdf5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the original\n",
    "\n",
    "gss.to_hdf('gss_eda.hdf5', 'gss', complevel=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 downey downey 6939829 Aug  5 09:58 gss_eda.hdf5\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l gss_eda.hdf5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And I'll create a second file with three random resamplings of the original dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if the file already exists, remove it\n",
    "import os\n",
    "\n",
    "if os.path.isfile('gss_eda.3.hdf5'):\n",
    "    !rm gss_eda.3.hdf5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This file contains three keys, `gss0`, `gss1`, and `gss2`, which map to three DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate and store three resamplings\n",
    "keys = ['gss0', 'gss1', 'gss2']\n",
    "\n",
    "for i in range(3):\n",
    "    np.random.seed(i)\n",
    "    sample = resample_by_year(gss, 'wtssall')\n",
    "\n",
    "    sample.to_hdf('gss_eda.3.hdf5', keys[i], complevel=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 downey downey 20717513 Aug  5 09:58 gss_eda.3.hdf5\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l gss_eda.3.hdf5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the other notebooks in this case study, we'll load this resampled data rather than reading and cleaning the data every time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}