1-03_Data Manipulation.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "skip"
    ]
   },
   "outputs": [],
   "source": [
    "# Options for all cells \n",
    "import pandas as pd\n",
    "# change display setting of pandas\n",
    "pd.set_option('display.notebook_repr_html', False)\n",
    "# Setting the graphics\n",
    "%matplotlib inline\n",
    "# suppress all warnings (since anova gives a warning)\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **pandas** package provides lots of simple functions for reading data from and writing data to various different file formats, such as CSV, Excel, SQL, JSON, and HTML. All the read functions in the package (`pd.read_*` functions) create a **pandas** `DataFrame` object. This object, like a `data.frame` in R, is meant for storing tabular data where columns can potentially contain different data types. The dimension of a data frame are typically labelled with row names and column names in R. Here, the labels are called index and columns, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "tips = pd.read_csv(\"Data/tips.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip**\n",
    "\n",
    "In Python, only the equals sign `=` is used to assign values to variables. Variable names can start with an underscore `_` (unlike in R) and are case sensitive (like in R). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip**\n",
    "\n",
    "Documentation for each function can be accessed via the websites for the package in question, e.g., [http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html), or can be viewed directly from Jupyter notebooks via `?`. It can be placed in front of the function call as in R, e.g., `?pd.read_csv`, or after it. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions, Methods and Attributes\n",
    "\n",
    "R has been characterised well by \"Everything that exists is an object. Everything that happens is a function call.\" The Python equivalent is \"Everything is an object, and all objects can have attributes.\"\n",
    "\n",
    "When working in R, in most cases we use a function on an object, i.e., `function(object, ...)`. This is the case for functions from a package, e.g., to read in data, generic functions like `summary` which do different things depending on the object class, or functions to access attributes of the object such as the dimension (via the `dim` function).\n",
    "\n",
    "In Python, functions from a package such as `pd.read_csv` are utilised like in R, i.e., `function(arguments)`. Beyond that, objects typically have attributes which are references to other Python objects. Those can be data types such as a list to store the dimension of a data frame but also can be functions specific to that object class. Such functions are referred to as object methods, or simply methods. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attributes, including methods, are accessed via the dot operator `.`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`<object_name>.<attribute>`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, attributes of a **pandas** `DataFrame` include the dimension (`.shape`) and the dimension labels (`.index` and `.columns`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['total_bill', 'tip', 'sex', 'smoker', 'day', 'time', 'size'], dtype='object')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Methods require parentheses after the method name, where any argument can be specified."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`<object_name>.<method>(arg=*)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Methods for a `DataFrame` include the `.head` and `.tail` functions for preview the first or last few rows of the data frame as well the `.describe` function which provides a summary of the data frame, similar to R's `summary` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>sex</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>244.000000</td>\n",
       "      <td>244.000000</td>\n",
       "      <td>244</td>\n",
       "      <td>244</td>\n",
       "      <td>244</td>\n",
       "      <td>244</td>\n",
       "      <td>244.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>157</td>\n",
       "      <td>151</td>\n",
       "      <td>87</td>\n",
       "      <td>176</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>19.785943</td>\n",
       "      <td>2.998279</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.569672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>8.902412</td>\n",
       "      <td>1.383638</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.951100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>3.070000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>13.347500</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>17.795000</td>\n",
       "      <td>2.900000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>24.127500</td>\n",
       "      <td>3.562500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>50.810000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        total_bill         tip   sex smoker  day    time        size\n",
       "count   244.000000  244.000000   244    244  244     244  244.000000\n",
       "unique         NaN         NaN     2      2    4       2         NaN\n",
       "top            NaN         NaN  Male     No  Sat  Dinner         NaN\n",
       "freq           NaN         NaN   157    151   87     176         NaN\n",
       "mean     19.785943    2.998279   NaN    NaN  NaN     NaN    2.569672\n",
       "std       8.902412    1.383638   NaN    NaN  NaN     NaN    0.951100\n",
       "min       3.070000    1.000000   NaN    NaN  NaN     NaN    1.000000\n",
       "25%      13.347500    2.000000   NaN    NaN  NaN     NaN    2.000000\n",
       "50%      17.795000    2.900000   NaN    NaN  NaN     NaN    2.000000\n",
       "75%      24.127500    3.562500   NaN    NaN  NaN     NaN    3.000000\n",
       "max      50.810000   10.000000   NaN    NaN  NaN     NaN    6.000000"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips.describe(include=\"all\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip**\n",
    "\n",
    "In Jupyter, we can view all the attributes and methods associated with an object by using tab completion. Once an object has been created (i.e., tips for above), typing `tips.` into a cell and pressing the tab key shows all the attributes and methods that can be used on that object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**\n",
    "\n",
    "1. Import the mtcars data (mtcars.csv), ensuring that the car names are used for the row index.\n",
    "2. What are the dimensions of the imported data?\n",
    "3. Print the top 5 rows of the data.\n",
    "\n",
    "Extension\n",
    "\n",
    "4. Print the last 10 rows of the data.\n",
    "5. Import only the miles per gallon (mpg) and weight (wt) columns from the mtcars data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": [
     "skip"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>wt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Mazda RX4</th>\n",
       "      <td>21.0</td>\n",
       "      <td>2.620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mazda RX4 Wag</th>\n",
       "      <td>21.0</td>\n",
       "      <td>2.875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Datsun 710</th>\n",
       "      <td>22.8</td>\n",
       "      <td>2.320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hornet 4 Drive</th>\n",
       "      <td>21.4</td>\n",
       "      <td>3.215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hornet Sportabout</th>\n",
       "      <td>18.7</td>\n",
       "      <td>3.440</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    mpg     wt\n",
       "Mazda RX4          21.0  2.620\n",
       "Mazda RX4 Wag      21.0  2.875\n",
       "Datsun 710         22.8  2.320\n",
       "Hornet 4 Drive     21.4  3.215\n",
       "Hornet Sportabout  18.7  3.440"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "import pandas as pd\n",
    "mtcars = pd.read_csv(\"Data/mtcars.csv\", index_col=0)\n",
    "mtcars.shape\n",
    "mtcars.head(5)\n",
    "\n",
    "mtcars.tail(10)\n",
    "mtcars2 = pd.read_csv(\"Data/mtcars.csv\", index_col=0, usecols=[0,1,6])\n",
    "mtcars2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Manipulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **pandas** package can be used for more than importing and exporting data. Written originally by Wes McKinney, the **pandas** package has grown to also include data manipulation, statistics and visualisation. Here, we will focus on the functionality for data manipulation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Common data manipulation tasks mostly have corresponding `DataFrame` methods:\n",
    "\n",
    "| Description | Method | \n",
    "| --- | --- | \n",
    "| Filter rows based on values | `.query` | \n",
    "| Sort rows | `.sort_values` | \n",
    "| Rename columns | `.rename` | \n",
    "| Summarise/Aggregate columns | `.agg` |\n",
    "| Group the data | `.groupby` | \n",
    "\n",
    "However, some tasks such as selecting columns, changing or adding new columns, or selecting rows based on position are not done via a corresponding method. We will show how this is done in Python and introduce the concept of dictionaries which are needed for setting arguments to the `.rename` and `.agg` methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting Columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access columns in a data frame by indexing with `[]` and supplying the corresponding column labels. To access a single column, a string with the column label is sufficient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    Dinner\n",
       "1    Dinner\n",
       "2    Dinner\n",
       "3    Dinner\n",
       "4    Dinner\n",
       "Name: time, dtype: object"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips['time'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access multiple columns, we provide the labels in form of a list. Lists are created using square brackets `[]`, with individual items separated by commas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>size</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>16.99</td>\n",
       "      <td>2</td>\n",
       "      <td>Sun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.34</td>\n",
       "      <td>3</td>\n",
       "      <td>Sun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21.01</td>\n",
       "      <td>3</td>\n",
       "      <td>Sun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23.68</td>\n",
       "      <td>2</td>\n",
       "      <td>Sun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>24.59</td>\n",
       "      <td>4</td>\n",
       "      <td>Sun</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   total_bill  size  day\n",
       "0       16.99     2  Sun\n",
       "1       10.34     3  Sun\n",
       "2       21.01     3  Sun\n",
       "3       23.68     2  Sun\n",
       "4       24.59     4  Sun"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_of_names = ['total_bill', 'size', 'day']\n",
    "tips[list_of_names].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip**\n",
    "\n",
    "Lists in Python, like lists in R, are very versatile and can hold any collection of objects with mixing of multiple types allowed. They are often used to specify a collection of elements for use in function argument. In R, we typically would do that via vectors or lists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selecting multiple columns returns another data frame but selecting a single column returns a `Series` object. This object consists of two parts, `values` and `index`, which can be accessed through the corresponding attributes. A `Series` has similar methods as a `DataFrame`, such as `.head` and `.describe`. Other helpful methods include\n",
    "\n",
    "* `.isnull()`/`.notnull()` to perform a boolean test for missing/non-missing values\n",
    "* `.isfinite()` to perform a boolean test for finite values\n",
    "* `.value_counts()` to return counts of unique values\n",
    "* `.idxmax()` to return the index of the row with the largest value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**\n",
    "\n",
    "Operations such as `+` on two `Series` objects are done based on the index, i.e., matching up elements with the same index, regardless of their position in the series. For comparison, addition of two vectors in R (which do not have an index), is carried out element-wise where the matching up is done by position, i.e, the two first elements are added up, the two second elements are added up, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding and Editing Columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To add a new column to a `DataFrame`, we select a new column via the indexing method (`[]` as above), using the new column name for which column is to be selected, and assign the new values to it. To edit a column, we can overwrite it and assign new values to the column in the same way.\n",
    "\n",
    "For example, we can add the tip share to the tips dataset by dividing the tip amount by the total amount of the bill and assign the result to a new column called `tip_share`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>sex</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "      <th>tip_share</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>16.99</td>\n",
       "      <td>1.01</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.059447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.34</td>\n",
       "      <td>1.66</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "      <td>0.160542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21.01</td>\n",
       "      <td>3.50</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "      <td>0.166587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23.68</td>\n",
       "      <td>3.31</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.139780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>24.59</td>\n",
       "      <td>3.61</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "      <td>0.146808</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   total_bill   tip     sex smoker  day    time  size  tip_share\n",
       "0       16.99  1.01  Female     No  Sun  Dinner     2   0.059447\n",
       "1       10.34  1.66    Male     No  Sun  Dinner     3   0.160542\n",
       "2       21.01  3.50    Male     No  Sun  Dinner     3   0.166587\n",
       "3       23.68  3.31    Male     No  Sun  Dinner     2   0.139780\n",
       "4       24.59  3.61  Female     No  Sun  Dinner     4   0.146808"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips['tip_share'] = tips['tip'] / tips['total_bill']\n",
    "tips.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting Rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select rows based on values in specific columns, we can use the `.query` method which takes logical statements to describe the rows which are to be selected. Unlike in for the `subset` and `filter` functions in R (from base R and the **dplyr** package, respectively), the logical statement is here provided in one string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>sex</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "      <th>tip_share</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20.65</td>\n",
       "      <td>3.35</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "      <td>0.162228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>17.92</td>\n",
       "      <td>4.08</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.227679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>20.29</td>\n",
       "      <td>2.75</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.135535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>15.77</td>\n",
       "      <td>2.23</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.141408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>39.42</td>\n",
       "      <td>7.58</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sat</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "      <td>0.192288</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    total_bill   tip     sex smoker  day    time  size  tip_share\n",
       "19       20.65  3.35    Male     No  Sat  Dinner     3   0.162228\n",
       "20       17.92  4.08    Male     No  Sat  Dinner     2   0.227679\n",
       "21       20.29  2.75  Female     No  Sat  Dinner     2   0.135535\n",
       "22       15.77  2.23  Female     No  Sat  Dinner     2   0.141408\n",
       "23       39.42  7.58    Male     No  Sat  Dinner     4   0.192288"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips_2 = tips.query(\"(time == 'Dinner' & day != 'Sun') | size > 4\")\n",
    "tips_2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Slicing Notation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select rows based on their position, we can also use indexing via `[]` but instead of a string or list to select columns, we use numeric arguments to select rows. The basic concept of this so-called slicing notation is `object[start:stop:step]`. If we wanted to view the first 10 variables at even locations we would use `object[0:20:2]`. This works for selecting sections of lists, strings and rows within data frames. The value of start is inclusive, the value for stop is not, i.e., everything up until but not including `stop` will be selected. If the value of start, stop or step is left empty, then the default values will be taken, which are that start and stop are the first and last values in the object, and step is by default 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[45, 12, 62, 78]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Slicing a list\n",
    "my_list = [12, 45, 74, 12, 97, 62, 53, 78]\n",
    "my_list[1::2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>sex</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "      <th>tip_share</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>16.99</td>\n",
       "      <td>1.01</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.059447</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   total_bill   tip     sex smoker  day    time  size  tip_share\n",
       "0       16.99  1.01  Female     No  Sun  Dinner     2   0.059447"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Slicing a DataFrame for the 1st row\n",
    "tips[:1:] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>sex</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "      <th>tip_share</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>16.99</td>\n",
       "      <td>1.01</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.059447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>12.54</td>\n",
       "      <td>2.50</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.199362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>11.35</td>\n",
       "      <td>2.50</td>\n",
       "      <td>Female</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fri</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.220264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>150</th>\n",
       "      <td>14.07</td>\n",
       "      <td>2.50</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>0.177683</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>18.71</td>\n",
       "      <td>4.00</td>\n",
       "      <td>Male</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Thur</td>\n",
       "      <td>Lunch</td>\n",
       "      <td>3</td>\n",
       "      <td>0.213789</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     total_bill   tip     sex smoker   day    time  size  tip_share\n",
       "0         16.99  1.01  Female     No   Sun  Dinner     2   0.059447\n",
       "50        12.54  2.50    Male     No   Sun  Dinner     2   0.199362\n",
       "100       11.35  2.50  Female    Yes   Fri  Dinner     2   0.220264\n",
       "150       14.07  2.50    Male     No   Sun  Dinner     2   0.177683\n",
       "200       18.71  4.00    Male    Yes  Thur   Lunch     3   0.213789"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Slicing a DataFrame to view 1 every 50 rows\n",
    "tips[::50]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**\n",
    "\n",
    "While indexing in R starts at 1, it starts at 0 in Python. For example, use index `2` to select the third element."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**\n",
    "\n",
    "1. Create a subset of the mtcars data which includes cars that have a weight larger than 3 or four cylinders. How many cars are in that dataset?\n",
    "2. Slice the mtcars dataset to show the first 10 even rows, i.e., the second, fourth, etc row.\n",
    "3. Add a new variable which contains the ratio of horse power to weight.\n",
    "\n",
    "Extension\n",
    "\n",
    "4. Read in the tips dataset (tips.csv).\n",
    "5. Calculate the mean tip for for each day.\n",
    "6. Add a new variable which contains the party size as a binary factor (choose your own cutoff). Hint: The `pd.cut` function works similarly to R's `cut` function.\n",
    "7. Ensure that the two categories are \"small/medium\" (four or fewer people) and \"large\" (more than four people) and labeled appropriately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": [
     "skip"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>sex</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "      <th>size_f</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>16.99</td>\n",
       "      <td>1.01</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>small/medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.34</td>\n",
       "      <td>1.66</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "      <td>small/medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21.01</td>\n",
       "      <td>3.50</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "      <td>small/medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23.68</td>\n",
       "      <td>3.31</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>small/medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>24.59</td>\n",
       "      <td>3.61</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "      <td>small/medium</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   total_bill   tip     sex smoker  day    time  size        size_f\n",
       "0       16.99  1.01  Female     No  Sun  Dinner     2  small/medium\n",
       "1       10.34  1.66    Male     No  Sun  Dinner     3  small/medium\n",
       "2       21.01  3.50    Male     No  Sun  Dinner     3  small/medium\n",
       "3       23.68  3.31    Male     No  Sun  Dinner     2  small/medium\n",
       "4       24.59  3.61  Female     No  Sun  Dinner     4  small/medium"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# solution\n",
    "mtcars.query(\"wt > 3 | cyl == 4\").shape\n",
    "mtcars[1:20:2]\n",
    "mtcars[\"hp2wt\"] = mtcars[\"hp\"] / mtcars[\"wt\"]\n",
    "\n",
    "tips = pd.read_csv(\"Data/tips.csv\")\n",
    "import numpy as np\n",
    "tips.groupby(\"day\").agg({\"tip\" : np.mean})\n",
    "tips[\"size_f\"] = pd.cut(tips[\"size\"], bins=2)\n",
    "bins_size = pd.IntervalIndex.from_breaks([0,4,7])\n",
    "tips[\"size_f\"] = pd.cut(tips[\"size\"], bins_size).cat.rename_categories([\"small/medium\", \"large\"])\n",
    "tips.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dictionaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dictionaries are objects which hold an unordered collection of key-value pairs. They are defined with curly braces using the following notation: `{key : value}`, with different key-value pairs being separated with commas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be used to define pairs of old and new names for the `.rename` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>total_bill</th>\n",
       "      <th>tip</th>\n",
       "      <th>sex</th>\n",
       "      <th>smoker</th>\n",
       "      <th>day</th>\n",
       "      <th>time_of_day</th>\n",
       "      <th>party_size</th>\n",
       "      <th>size_f</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>16.99</td>\n",
       "      <td>1.01</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>small/medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.34</td>\n",
       "      <td>1.66</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "      <td>small/medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21.01</td>\n",
       "      <td>3.50</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>3</td>\n",
       "      <td>small/medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23.68</td>\n",
       "      <td>3.31</td>\n",
       "      <td>Male</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>2</td>\n",
       "      <td>small/medium</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>24.59</td>\n",
       "      <td>3.61</td>\n",
       "      <td>Female</td>\n",
       "      <td>No</td>\n",
       "      <td>Sun</td>\n",
       "      <td>Dinner</td>\n",
       "      <td>4</td>\n",
       "      <td>small/medium</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   total_bill   tip     sex smoker  day time_of_day  party_size        size_f\n",
       "0       16.99  1.01  Female     No  Sun      Dinner           2  small/medium\n",
       "1       10.34  1.66    Male     No  Sun      Dinner           3  small/medium\n",
       "2       21.01  3.50    Male     No  Sun      Dinner           3  small/medium\n",
       "3       23.68  3.31    Male     No  Sun      Dinner           2  small/medium\n",
       "4       24.59  3.61  Female     No  Sun      Dinner           4  small/medium"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tips_2 = tips.rename(columns = {\"time\" : \"time_of_day\", \"size\" : \"party_size\"})\n",
    "tips_2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dictionaries are also used to specify which columns should be summarised or aggregated in which way, using the `.agg` method. Dictionaries can also hold lists as values, allowing us to specify several summary functions for a column.\n",
    "\n",
    "Some fundamental summary functions such as `min` and `max` are provided by base Python. More statistical summary functions such as `mean` and `median` are provided by the **NumPy** package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>size</th>\n",
       "      <th>tip</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>NaN</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.998279</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      size        tip\n",
       "max    NaN  10.000000\n",
       "mean   NaN   2.998279\n",
       "min    1.0   1.000000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "tips.agg({\"size\" : min, \"tip\" : [min, max, np.mean]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip**\n",
    "\n",
    "In R, the `%>%` operator from the **magrittr** package allows us to make pipelines of a series of steps on an object, e.g., a data frame. In Python, the `.` can be used to chain multiple methods on an object together into one line of code, without having to reassign each time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Different Data Types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `Series` in a `DataFrame` can hold different types of data such as numeric data, categorical data, datetimes, etc. For most common data types other than numeric, we need so-called accessors to access the type-specific attributes and methods. For example, the accessor for categorical data is `.cat` and the methods include `.cat.rename_categories()`, `.cat.remove_unused_categories()`, and `.cat.reorder_categories()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning**\n",
    "\n",
    "It is, of course, possible to access the categorical data via `.values` and manipulate this directly. However, the resulting object is an array, not a `Series`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}