1-02_The Python Environment for Data Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the major hurdles in getting up and running with Python is the complexity of the installation\n",
    "process, as there are not only multiple versions of Python to consider, but packages must\n",
    "be downloaded and installed to the right location, and any dependencies between them must\n",
    "also be managed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fortunately, several Python distributions have been put together that work as a one stop installation of both Python and many of the extended libraries. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installing the Anaconda Distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Anaconda distribution is cross platform and comes with Python 2.7 or 3.6, as well as other tools and libraries such as Jupyter, Spyder, IPython, NumPy,\n",
    "SciPy, pandas, seaborn and matplotlib already configured."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fastest way to get up and running with Python is to download the Anaconda Python distribution\n",
    "from [https://www.anaconda.com/download](https://www.anaconda.com/download), and follow the installation instructions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default installation path for the Python 3 Anaconda distribution on Windows is:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "    C:\\Users\\<username>\\AppData\\Local\\Continuum\\Anaconda3\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Jupyter Notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several ways of using Python for data analysis. Spyder is a full interactive development environment (IDE), similar to RStudio. Another tool are Jupyter Notebooks which we will be using here. Like R Notebooks and R Markdown, they are designed for an easy integration of text and programing. They are aimed at providing a more interactive workflow for Python programming, analysis and reporting. While an R Notebook is a script that gets rendered into an output file in one step, Jupyter Notebooks contain cells which can all be executed and rendered interactively. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Starting Jupyter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once installed look for and run an application called Jupyter Notebook, which on Windows is located in:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "    Start menu --> All programs --> Anaconda3 --> Jupyter Notebook \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, open up a command line console and type:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "    jupyter notebook\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After about a minute, you should see the Jupyter application appear. The figure shows an\n",
    "example of what Jupyter looks like on Mac. The appearance on Windows and Linux operating\n",
    "systems will be slightly different."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/fig/jupyter1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see three main tabs in Jupyter on start-up:\n",
    "\n",
    "* **Files** Your file directory\n",
    "\n",
    "* **Running** Lists all of the notebooks currently running\n",
    "\n",
    "* **Clusters** For using IPython in parallel with your cluster (beyond the scope of this course)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Opening a new Jupyter Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To open a Jupyter Notebook, click the New drop down menu on the File tab and select \"Python 3\" under the Notebooks heading."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/fig/jupyter2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will open a blank Notebook with an IPython console (using Python 3) running underneath it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/fig/jupyter3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The IPython console is used to input and execute Python code interactively. Outputs, errors\n",
    "and warning messages are directly shown in the same window. A command which has been\n",
    "input into the console is executed by simply pressing `Shift+Enter`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The IPython Kernel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The IPython kernel is the processing back-end that Jupyter notebooks use to execute Python code. The IPython kernel is a python interpreter designed for interactive use, and adds many helpful features such as:\n",
    "\n",
    "* Coloured input and output lines, as well as error messages for added clarity.\n",
    "* Aliases to useful operating system calls for directory navigation e.g. pwd, cd, ls and mkdir.\n",
    "* 'Magic' functions provide many other helpful shortcuts. Type `%quickref` for a full list. A few are covered later in this course.\n",
    "* Tab completion of variable names, imports, class attributes and functions as well as file names in the current working directory.\n",
    "* Object exploration through the use of `?` and `??` (see below).\n",
    "* Input/Output history referencing and searching."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A detailed introduction and overview of these features can also be found by entering a single\n",
    "question mark `?` within IPython. Key features will be introduced gradually throughout this\n",
    "course with examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Cell Based Execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is common when working interactively to only want to run a section of the code, rather than the entire script. To aid in this, Jupyter uses a concept of 'code cells', which can be executed individually in the IPython console. Each block represents a code cell and can be executed by:\n",
    "\n",
    "* Press `Ctrl+Enter` to execute all the code in the current active cell.\n",
    "\n",
    "* Press `Shift+Enter` to execute all the code in the current active cell and advance the cursor to the next cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `Shift+Enter` multiple times to incrementally step through the code and investigate the results in the IPython console is a common workflow when prototyping and doing exploratory work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/fig/jupyter4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Command and Edit Modes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jupyter notebook is a modal editor which means that the keyboard does different things depending on which mode the notebook is in. There are two modes: **edit mode** and **command mode.** To go from command mode to edit mode, press `Enter`. To return to command mode from edit mode, press `Esc`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Edit mode is indicated by a green cell border and left sidebar, and a prompt showing in the editor area:\n",
    "\n",
    "![](img/fig/CellModeEdit.png)\n",
    "\n",
    "When a cell is in edit mode, you can type into the cell, like a normal text editor. Enter edit mode by pressing `Enter` or using the mouse to click on a cell's editor area."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Command mode is indicated by a grey cell border and a blue sidebar:\n",
    "\n",
    "![](img/fig/CellModeCommand.png)\n",
    "\n",
    "When you are in command mode, you are able to edit the notebook as a whole, but not type into individual cells. Most importantly, in command mode, the keyboard is mapped to a set of shortcuts that let you perform notebook and cell actions efficiently. For example, if you are in command mode and you press `c`, you will copy the current cell - no modifier is needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Warning: Cells in Command Mode** \n",
    "\n",
    "Don't try to type into a cell in command mode; unexpected things will happen!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A summary of useful shortcuts is included at the end of this section, however a full list is always available by going to:\n",
    "\n",
    "```\n",
    "    Help --> Keyboard Shortcuts\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip: Keyboard Shortcuts** \n",
    "\n",
    "You can also access the Keyboard Shortcuts list by pressing `h` in command mode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Markdown Text Cells\n",
    "\n",
    "The default cells are IPython console cells though a cell can be changed to include Markdown text. To change the type of cell from code to Markdown go to the top menu bar:\n",
    "\n",
    "```\n",
    "    Cell > Cell Type > Markdown\n",
    "```\n",
    "\n",
    "Or press `m` while in Command Mode and highlighting the cell. \n",
    "\n",
    "Markdown text cells support plain text, Markdown and HTML. For information about Markdown and HTML:\n",
    "\n",
    "* **Markdown** - [https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)\n",
    "* **HTML** - [https://web.stanford.edu/group/csp/cs21/htmlcheatsheet.pdf](https://web.stanford.edu/group/csp/cs21/htmlcheatsheet.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Useful Keyboard Shortcuts\n",
    "\n",
    "All of the cell manipulation commands also have keyboard shortcuts to execute. A list of the commonly used ones are below; however, to view a full list of shortcuts either go to **Help > Keyboard Shortcuts**, or in Command Mode, press **H**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Command Mode (press Esc to enable)\n",
    "\n",
    "* **Enter** enter edit mode\n",
    "* **H** show keyboard shortcuts\n",
    "* **Shift + Enter** run cell, select below\n",
    "* **Ctrl + Enter** run cell\n",
    "* **Alt + Enter** run cell, insert below\n",
    "* **Y** to code\n",
    "* **M** to markdown\n",
    "* **A/B** insert cell above or below\n",
    "* **X** cut selected cell\n",
    "* **C** copy selected cell\n",
    "* **V** paste cell\n",
    "* **D, D** (i.e. **D** then **D** again) delete cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Edit Mode (press Enter to enable)\n",
    "\n",
    "* **Tab** code completion or indent\n",
    "* **Ctrl + Z** undo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modules, Packages, and Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A module in Python is simply a `.py` file that contains additional code to define functions, classes or variables. Modules provide a way to logically group related code, making it easier to understand and use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Python terms, a module is just another Python object with attributes that you can bind and reference, though in this case the attributes are the various functions, classes or variables defined in the `.py` file. To use them within our namespace we use the `import` statement in one of three ways:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    import module_name\n",
    "    import module_name as alias\n",
    "    from module_name import object1, object2, object3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method used to import will affect how the objects imported are used. Some examples are\n",
    "shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8660254037844386"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# method 1\n",
    "import math\n",
    "math.sin(math.pi/3)\n",
    "\n",
    "# method 2\n",
    "import math as m\n",
    "m.sin(m.pi/3)\n",
    "\n",
    "# method 3\n",
    "from math import sin, pi\n",
    "sin(pi/3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also a fourth way of importing things with a * instead of specifying each object individually.\n",
    "This imports all the functions, classes and variables contained within the module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.14384103622589053"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from math import *\n",
    "log(sin(pi/3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, this method of importing objects is frowned upon as it is less clear what is imported and will also override variables already defined if their names clash."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A module is loaded only once, regardless of the number of times it is imported. This prevents\n",
    "the module execution from happening over and over again if multiple imports occur in a single\n",
    "script."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Packages and Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A package is a collection of modules, similar to packages in R. A collection of packages is usually referred to as a library. There is no formal distinction between a package and a library so sometimes package and library are used interchangeably.\n",
    "\n",
    "Packages and libraries can be made available in the same manner as above, simply using the package or library name instead of the `module_name` in the `import` statements.\n",
    "\n",
    "It is also possible to selectively import only certain modules in a package or library. To access a module (or sub-package), we must use the `.` operator. For example, to extract a module named `foo` from a package named `mypackage`, as the alias `mpf`, we use the following code:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import mypackage.foo as mpf\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then to use any functions from this module, we simply call the module alias, the `.`, and then the function name, i.e., for function `func1` we have the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "x = mpf.func1()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Python Standard Library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python comes with many useful modules and packages as standard. Details on using the Python Standard Library can\n",
    "be found in the documentation at [https://docs.python.org/3.6/library/](https://docs.python.org/3.6/library/), with additional examples available at this fantastic site [http://pymotw.com/2/contents.html](http://pymotw.com/2/contents.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**\n",
    "\n",
    "1. Open up a new Jupyter Notebook and print \"Hello Python world!\" on the console.\n",
    "2. Add a markdown cell and include a note that this is the notebook for today's workshop.\n",
    "3. Import the **numpy** library using the alias np.\n",
    "\n",
    "Extension:\n",
    "\n",
    "4. Draw a (pseudo) random number from a uniform distribution on the interval [2, 5]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "skip"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0721064635544777"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"Hello Python world!\"\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "import random\n",
    "random.uniform(2,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing Additional Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have the Anaconda Python distribution, this comes with many third party python packages\n",
    "already installed. However you may come across additional packages that you wish to install."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike R's `install.packages`, Python itself has no built in system to manage packages, but there is a central repository for Python packages known as the Python Package Index or PyPI. Several command line utilities\n",
    "have been developed to look up and install Python packages from PyPI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The recommended tool to use to install additional Python packages is pip. Python >= 3.4 now comes with pip by default."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can search for packages on PyPI using,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "pip search <package name>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Packages are then installed with,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "pip install <package name>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other helpful pip commands are `pip list`, which lists all the packages installed by pip, and\n",
    "`pip uninstall <package name>` to remove an installed package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Help System"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Python, to find out information about a function we can use the online documentation for the package that it is in. However, we can also find out more about functions by using Jupyter's inbuilt help system.\n",
    "\n",
    "The `?` can be used to display help on a function, Jupyter allows the use of the `?` before or after the function in question. For example, to find out information about the `np.floor` function we can use the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    ">>> import numpy as np\n",
    ">>> ?np.floor\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "![](img/fig/HelpFile.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "512px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_position": {
    "height": "603px",
    "left": "0px",
    "right": "1108px",
    "top": "107px",
    "width": "212px"
   },
   "toc_section_display": "block",
   "toc_window_display": true,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}