1 - Reinforcement Learning Framework.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Reinforcement Learning framework"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a Reinforcement Learning framework we have:\n",
    " * an agent that learns or learned to interact with the environment\n",
    " * the assumption that time evolves in descrete time steps\n",
    " * At the initial time step the agent observes the environment's state\n",
    " * After this obervation, the agent selects an apropriate action in response.\n",
    " * Next, the environment presents a new situation (state) to the agent and at the same time it gives the agent a reward. The reward provides an indication to the agent so it knows how appropriate its action was to the previous state.\n",
    " * The process continues, repeating again these steps\n",
    " \n",
    " Let's not assume the environment shows the agent everything he needs to know to make well infromed decisions, but it simplifies the underlying mathematics if we do assume so."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, reviewing the steps again:\n",
    "\n",
    "* Agent receives the state $S_{0}$ at the initial time step.\n",
    "* Agent chooses an action (and takes such action) $A_{0}$\n",
    "* Environment gives to the agent a reward $R_{1}$ along with the next state $S_{1}$\n",
    "* The agent then selects (and take) another action $A_{1}$ in response to the new state\n",
    "* The environment gives again to the agent a reward - this time $R_{2}$ along with the next state $S_{2}$\n",
    "* The agent then selects (and take) the next action - this time $A_{2}$\n",
    "\n",
    "![Reinforcement Learning Framework](images/learningFramework.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, as the agent interacts with the environment there will be a sequence of states actions and rewards that will be transmitted from the environment to the agent (rewards and next state) as the latter modifies such environment by means of the actions selected to be taken.\n",
    "\n",
    "**Reward will always be the most relevant quantity to the agent.**\n",
    "The goal of any RL agent is to maximize the expected cumulative reward or the sum of the rewards that it obtains as it interacts with the invironment (over all time steps). So it must find the best *strategy* for choosing actions with which the cumulative reward is likely to be high."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two kind of reinforcement learning tasks.\n",
    "\n",
    "**Episodic Taks** are those tasks that have a well defined ending point. In this case, a complete sequence of steps (interactions) from start to finish is known as an *episode*. It can be said from episodic tasks:\n",
    "\n",
    "    * are tasks with well defined end points\n",
    "    * bring \"experience\" from one episode to the next one\n",
    "    \n",
    "**Continuous Tasks** are tasks that go on forever without end. Like an RL algorithm that buys and sells stocks in response to the financial market; this kind of agent would be best modeled as an agent in the continuing tasks. In this cases the agent \"lives forever.\"\n",
    "\n",
    "### To remember:\n",
    "\n",
    "* A **task** is an instance of the reinforcement learning (RL) problem.\n",
    "* **Continuing tasks** are tasks that continue forever, without end.\n",
    "* **Episodic tasks** are tasks with a well-defined starting and ending point.\n",
    "    * In this case, we refer to a complete sequence of interaction, from start to finish, as an **episode**\n",
    "    * Episodic tasks come to an end whenever the agent reaches a **terminal state**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Reward Hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Reward Hypothesis sais:\n",
    "\n",
    "   \"All goals can be framed as the maximization of *expected* cumulative reward\".\n",
    "    \n",
    "This allows us formulate an agent's goal along the lines of maximizing the expected cumulative reward.\n",
    "\n",
    "For example, frame the idea of a humanoid learning to walk in the context of reinforcement learning. In this context we would have that:\n",
    "\n",
    "    * the states could be:\n",
    "        + Position and velocities of the joints\n",
    "        + Statistics about the ground\n",
    "        + Foot sensor data\n",
    "    * the actions could be:\n",
    "        + Forces applied to the joints\n",
    "        \n",
    "![Reward Hoypothesis](./images/RewardHypotesisIllust.png)\n",
    "\n",
    "\n",
    "The reward structure for this problem is surprisingly intuitive (from the DeepMind paper):\n",
    "\n",
    "$$ r = min(v_{x}, v_{max}) - 0.005(v_{y}^2 + v_{z}^2) - 0.05y^2 - 0.02||u||^2 + 0.02$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this formula:\n",
    "    \n",
    "    * The first term will influence the behaviour of the humanoid to walk fast\n",
    "    * The first, second and third terms will influence the humanoid to walk forward\n",
    "    * The fourth term, will influence the humanoid to walk smoothly\n",
    "    * The fifth term will add to the cumulated reward for each time step the humanoid does not fall (in the moment the humanoid falls, the episode is over, so with term it makes sense to walk as much as possible without falling).\n",
    "    \n",
    "The specific function of each term of this equation:\n",
    "\n",
    "    * The first term is the proportional to the robots forward velocity\n",
    "    * The second term penalizes deviation from forward direction\n",
    "    * The third term penalizes deviation from center of track\n",
    "    * The fourth term penalizes the torques\n",
    "    * The fifth term is the constant reward for not falling.\n",
    "    \n",
    "![Humanoid Reward](images/HumanoidReward.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cumulative Reward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An RL agent should not focus only on individual time steps and instead, it should to keep all the time steps in mind.\n",
    "Actions have short and long term consequences and the agent needs to gain some understanding of the complex effects its actions have on the environment.\n",
    "Again: the goal of the agent is to maximize the expected *cumulative* reward.\n",
    "\n",
    "**Definition:** The return at time step $t$ is:\n",
    "\n",
    "$$ G_{t} := R_{t+1} + R_{t+2} + R_{t+3} + R_{t+4} + ...$$\n",
    "\n",
    "This returned is denoted as G.\n",
    "\n",
    "The agent seeks to maximize the expected return. It is *expected* because the agent cannot predict with complete certainty what the future reward is likely to be, so it has to rely on a prediction - an estimate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discounted Return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the idea is that we'll maximize a different sum with rewards that are farther along in time and that are multiplied by smaller values.\n",
    "\n",
    "\n",
    "$$G_{t} = R_{t+1} + (0.9)R_{t+2} + (0.81)R_{t+3} + (0.73)R_{t+4} + ...$$\n",
    "\n",
    "The coefficients (i.e. 0.9, 0.81, 0.73...) are the discount rate. $G_{t}$ is the *Discounted Return*.\n",
    "\n",
    "By *discounted* it is meant, that the goal will change in a way that the agent values more immediate rewards rather then rewards that are received further in the future.\n",
    "\n",
    "*How* to choose the coefficients? The discounted rate is normally a real number between 0 and 1 (including 0 and 1).\n",
    "\n",
    "$$ Discounted\\_rate: \\gamma \\in [0, 1]$$\n",
    "\n",
    "Normally the first term is multiplied by $\\gamma$, then the seconds is multiplied by $\\gamma^2$, the third is multiplied by $\\gamma^3$ and so and so on.\n",
    "\n",
    "$\\gamma$ is set by us to refine the goal that we have for the agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}