Lecture 22 - Pandas pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "54c46642",
   "metadata": {},
   "source": [
    "# Pandas pipe function\n",
    "\n",
    "* Organizes multiple preprocessing operations into a single one"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95b7ed04",
   "metadata": {},
   "source": [
    "## Exercise 1 - Creating a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "59fb236e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>1.0</td>\n",
       "      <td>45</td>\n",
       "      <td>1.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100</td>\n",
       "      <td>2.0</td>\n",
       "      <td>56</td>\n",
       "      <td>1.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>101</td>\n",
       "      <td>3.0</td>\n",
       "      <td>48</td>\n",
       "      <td>1.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>102</td>\n",
       "      <td>4.0</td>\n",
       "      <td>47</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>103</td>\n",
       "      <td>5.0</td>\n",
       "      <td>62</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>104</td>\n",
       "      <td>2.0</td>\n",
       "      <td>112</td>\n",
       "      <td>1.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>105</td>\n",
       "      <td>NaN</td>\n",
       "      <td>54</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>106</td>\n",
       "      <td>5.0</td>\n",
       "      <td>49</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id    A    B    C\n",
       "0  100  1.0   45  1.2\n",
       "1  100  2.0   56  1.4\n",
       "2  101  3.0   48  1.1\n",
       "3  102  4.0   47  1.8\n",
       "4  103  5.0   62  NaN\n",
       "5  104  2.0  112  1.4\n",
       "6  105  NaN   54  1.6\n",
       "7  106  5.0   49  1.5"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame(\n",
    "    \n",
    "    {\n",
    "       \"id\": [100, 100, 101, 102, 103, 104, 105, 106],\n",
    "       \"A\": [1, 2, 3, 4, 5, 2, np.nan, 5],\n",
    "       \"B\": [45, 56, 48, 47, 62, 112, 54, 49],\n",
    "       \"C\": [1.2, 1.4, 1.1, 1.8, np.nan, 1.4, 1.6, 1.5]\n",
    "    }\n",
    "    \n",
    ")\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9abe5de",
   "metadata": {},
   "source": [
    "## Exercise 2 - function to handle missing values\n",
    "\n",
    "* Replace the missing values in the numerical columns with the mean value of the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9b0ed39f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def fill_missing_values(df):\n",
    "   for col in df.select_dtypes(include= [\"int\",\"float\"]).columns:\n",
    "      val = df[col].mean()\n",
    "      df[col].fillna(val, inplace=True)\n",
    "   return df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edc81091",
   "metadata": {},
   "source": [
    "## Exercise 3 - function to remove duplicate values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "86915b3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def drop_duplicates(df, column_name):\n",
    "   df = df.drop_duplicates(subset=column_name)\n",
    "   return df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c0471f9",
   "metadata": {},
   "source": [
    "* It eliminates the duplicate values in the given column or columns. \n",
    "* In addition to the DataFrame, this function also takes a column name as an argument. We can pass the additional arguments to the pipe as well."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5b29a52",
   "metadata": {},
   "source": [
    "## Exercise 4 - function to eliminate outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9ea0ed77",
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_outliers(df, column_list):\n",
    "   for col in column_list:\n",
    "      avg = df[col].mean()\n",
    "      std = df[col].std()\n",
    "      low = avg - 2 * std\n",
    "      high = avg + 2 * std\n",
    "      df = df[(df[col] <= high) | (df[col] >= low)]\n",
    "   return df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1b4d220",
   "metadata": {},
   "source": [
    "What this function does is as follows:\n",
    "\n",
    "* It takes a DataFrame and a list of columns as input\n",
    "* For each column in the list, it calculates the mean and standard deviation\n",
    "* It calculates a lower and upper bound using the mean and standard deviation\n",
    "* It removes the values that are outside range defined by the lower and upper bound"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2414e26d",
   "metadata": {},
   "source": [
    "## Exercise 5 - creating a pipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7bb37aff",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_processed = (df.\n",
    "                pipe(fill_missing_values).\n",
    "                pipe(drop_duplicates, \"id\").\n",
    "                pipe(remove_outliers, [\"A\",\"B\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db6a88dd",
   "metadata": {},
   "source": [
    "* This pipe executes the functions in the given order. \n",
    "* We can pass the arguments to the pipe along with the function names."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae435537",
   "metadata": {},
   "source": [
    "## Exercise 6\n",
    "\n",
    "* One thing to mention here is that some functions in the pipe modify the original DataFrame. Thus, using the pipe as indicated above will update df as well.\n",
    "* One option to overcome this issue is to use a copy of the original DataFrame in the pipe. \n",
    "* If you do not care about keeping the original DataFrame as is, you can just use it in the pipe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1f7b52cc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>45</td>\n",
       "      <td>1.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>56</td>\n",
       "      <td>1.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>101</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>48</td>\n",
       "      <td>1.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>102</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>47</td>\n",
       "      <td>1.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>103</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>62</td>\n",
       "      <td>1.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>104</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>112</td>\n",
       "      <td>1.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>105</td>\n",
       "      <td>3.142857</td>\n",
       "      <td>54</td>\n",
       "      <td>1.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>106</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>49</td>\n",
       "      <td>1.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id         A    B         C\n",
       "0  100  1.000000   45  1.200000\n",
       "1  100  2.000000   56  1.400000\n",
       "2  101  3.000000   48  1.100000\n",
       "3  102  4.000000   47  1.800000\n",
       "4  103  5.000000   62  1.428571\n",
       "5  104  2.000000  112  1.400000\n",
       "6  105  3.142857   54  1.600000\n",
       "7  106  5.000000   49  1.500000"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4264a825",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>45</td>\n",
       "      <td>1.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>101</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>48</td>\n",
       "      <td>1.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>102</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>47</td>\n",
       "      <td>1.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>103</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>62</td>\n",
       "      <td>1.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>104</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>112</td>\n",
       "      <td>1.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>105</td>\n",
       "      <td>3.142857</td>\n",
       "      <td>54</td>\n",
       "      <td>1.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>106</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>49</td>\n",
       "      <td>1.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id         A    B         C\n",
       "0  100  1.000000   45  1.200000\n",
       "2  101  3.000000   48  1.100000\n",
       "3  102  4.000000   47  1.800000\n",
       "4  103  5.000000   62  1.428571\n",
       "5  104  2.000000  112  1.400000\n",
       "6  105  3.142857   54  1.600000\n",
       "7  106  5.000000   49  1.500000"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_processed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "762e6874",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(\n",
    "    \n",
    "    {\n",
    "       \"id\": [100, 100, 101, 102, 103, 104, 105, 106],\n",
    "       \"A\": [1, 2, 3, 4, 5, 2, np.nan, 5],\n",
    "       \"B\": [45, 56, 48, 47, 62, 112, 54, 49],\n",
    "       \"C\": [1.2, 1.4, 1.1, 1.8, np.nan, 1.4, 1.6, 1.5]\n",
    "    }\n",
    "    \n",
    ")\n",
    "\n",
    "df_copy = df.copy()\n",
    "\n",
    "df_processed = (df_copy.\n",
    "                pipe(fill_missing_values).\n",
    "                pipe(drop_duplicates, \"id\").\n",
    "                pipe(remove_outliers, [\"A\",\"B\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b216f8cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>45</td>\n",
       "      <td>1.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>101</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>48</td>\n",
       "      <td>1.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>102</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>47</td>\n",
       "      <td>1.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>103</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>62</td>\n",
       "      <td>1.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>104</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>112</td>\n",
       "      <td>1.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>105</td>\n",
       "      <td>3.142857</td>\n",
       "      <td>54</td>\n",
       "      <td>1.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>106</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>49</td>\n",
       "      <td>1.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id         A    B         C\n",
       "0  100  1.000000   45  1.200000\n",
       "2  101  3.000000   48  1.100000\n",
       "3  102  4.000000   47  1.800000\n",
       "4  103  5.000000   62  1.428571\n",
       "5  104  2.000000  112  1.400000\n",
       "6  105  3.142857   54  1.600000\n",
       "7  106  5.000000   49  1.500000"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_processed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "75a384b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>1.0</td>\n",
       "      <td>45</td>\n",
       "      <td>1.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100</td>\n",
       "      <td>2.0</td>\n",
       "      <td>56</td>\n",
       "      <td>1.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>101</td>\n",
       "      <td>3.0</td>\n",
       "      <td>48</td>\n",
       "      <td>1.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>102</td>\n",
       "      <td>4.0</td>\n",
       "      <td>47</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>103</td>\n",
       "      <td>5.0</td>\n",
       "      <td>62</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>104</td>\n",
       "      <td>2.0</td>\n",
       "      <td>112</td>\n",
       "      <td>1.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>105</td>\n",
       "      <td>NaN</td>\n",
       "      <td>54</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>106</td>\n",
       "      <td>5.0</td>\n",
       "      <td>49</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id    A    B    C\n",
       "0  100  1.0   45  1.2\n",
       "1  100  2.0   56  1.4\n",
       "2  101  3.0   48  1.1\n",
       "3  102  4.0   47  1.8\n",
       "4  103  5.0   62  NaN\n",
       "5  104  2.0  112  1.4\n",
       "6  105  NaN   54  1.6\n",
       "7  106  5.0   49  1.5"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0c4c64f",
   "metadata": {},
   "source": [
    "## Exercise 7 - start the pipeline exclusively\n",
    "\n",
    "* We can start the pipeline with an exclusive starting step, which just copies the original DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "89ba00a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def start_pipeline(df):\n",
    "    return df.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "88f4cd57",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_processed = (df.\n",
    "                pipe(start_pipeline).\n",
    "                pipe(fill_missing_values).\n",
    "                pipe(drop_duplicates, \"id\").\n",
    "                pipe(remove_outliers, [\"A\",\"B\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b3806d86",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>1.0</td>\n",
       "      <td>45</td>\n",
       "      <td>1.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100</td>\n",
       "      <td>2.0</td>\n",
       "      <td>56</td>\n",
       "      <td>1.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>101</td>\n",
       "      <td>3.0</td>\n",
       "      <td>48</td>\n",
       "      <td>1.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>102</td>\n",
       "      <td>4.0</td>\n",
       "      <td>47</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>103</td>\n",
       "      <td>5.0</td>\n",
       "      <td>62</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>104</td>\n",
       "      <td>2.0</td>\n",
       "      <td>112</td>\n",
       "      <td>1.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>105</td>\n",
       "      <td>NaN</td>\n",
       "      <td>54</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>106</td>\n",
       "      <td>5.0</td>\n",
       "      <td>49</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id    A    B    C\n",
       "0  100  1.0   45  1.2\n",
       "1  100  2.0   56  1.4\n",
       "2  101  3.0   48  1.1\n",
       "3  102  4.0   47  1.8\n",
       "4  103  5.0   62  NaN\n",
       "5  104  2.0  112  1.4\n",
       "6  105  NaN   54  1.6\n",
       "7  106  5.0   49  1.5"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "390eaacd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>45</td>\n",
       "      <td>1.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>101</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>48</td>\n",
       "      <td>1.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>102</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>47</td>\n",
       "      <td>1.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>103</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>62</td>\n",
       "      <td>1.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>104</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>112</td>\n",
       "      <td>1.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>105</td>\n",
       "      <td>3.142857</td>\n",
       "      <td>54</td>\n",
       "      <td>1.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>106</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>49</td>\n",
       "      <td>1.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id         A    B         C\n",
       "0  100  1.000000   45  1.200000\n",
       "2  101  3.000000   48  1.100000\n",
       "3  102  4.000000   47  1.800000\n",
       "4  103  5.000000   62  1.428571\n",
       "5  104  2.000000  112  1.400000\n",
       "6  105  3.142857   54  1.600000\n",
       "7  106  5.000000   49  1.500000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_processed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e419c915",
   "metadata": {},
   "source": [
    "## Exercise 8 - logging\n",
    "\n",
    "* We have a pipeline that consists of 4 steps. Depending on the raw data and the task at hand, we may need to create pipelines that have several more steps.\n",
    "\n",
    "* In such workflows, it is important to keep track of what happens at each step so it will be easier to debug in case something goes wrong.\n",
    "\n",
    "* We can achieve this by logging some information after each step. In our pipeline, the size of the DataFrame tells us if an unexpected thing happened.\n",
    "\n",
    "* Let’s print the size of the DataFrame after each step is applied in the pipeline. Since the steps are functions, we can use a Python decorator for this task.\n",
    "\n",
    "* A decorator is a function that takes another function and extends its behavior. The base function is not modified. The decorator wraps it and adds additional functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5e9c09ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import wraps\n",
    "\n",
    "def logging(func):\n",
    "    @wraps(func)\n",
    "    def wrapper(*args, **kwargs):\n",
    "        result = func(*args, **kwargs)\n",
    "        print(f\"The size after {func.__name__} is {result.shape}.\")\n",
    "        return result\n",
    "    return wrapper"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "194023db",
   "metadata": {},
   "source": [
    "## Exercise 9 - using the decorator\n",
    "\n",
    "* We will “decorate” the functions used in the pipeline as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bfbdc0de",
   "metadata": {},
   "outputs": [],
   "source": [
    "@logging\n",
    "def start_pipeline(df):\n",
    "    return df.copy()\n",
    "\n",
    "@logging\n",
    "def fill_missing_values(df):\n",
    "   for col in df.select_dtypes(include= [\"int\",\"float\"]).columns:\n",
    "      val = df[col].mean()\n",
    "      df[col].fillna(val, inplace=True)\n",
    "   return df\n",
    "\n",
    "@logging\n",
    "def drop_duplicates(df, column_name):\n",
    "   df = df.drop_duplicates(subset=column_name)\n",
    "   return df\n",
    "\n",
    "@logging\n",
    "def remove_outliers(df, column_list):\n",
    "   for col in column_list:\n",
    "      avg = df[col].mean()\n",
    "      std = df[col].std()\n",
    "      low = avg - 2 * std\n",
    "      high = avg + 2 * std\n",
    "      df = df[(df[col] <= high) | (df[col] >= low)]\n",
    "   return df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d48a7e3",
   "metadata": {},
   "source": [
    "## Exercise 10 - run the pipeline with decorators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "28593046",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The size after start_pipeline is (8, 4).\n",
      "The size after fill_missing_values is (8, 4).\n",
      "The size after drop_duplicates is (7, 4).\n",
      "The size after remove_outliers is (7, 4).\n"
     ]
    }
   ],
   "source": [
    "df_processed = (df.\n",
    "                pipe(start_pipeline).\n",
    "                pipe(fill_missing_values).\n",
    "                pipe(drop_duplicates, \"id\").\n",
    "                pipe(remove_outliers, [\"A\",\"B\"]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}