WebScraping_Review_Lab.ipynb

{"cells":[{"cell_type":"markdown","metadata":{},"source":["\u003cp style=\"text-align:center\"\u003e\n","    \u003ca href=\"https://skills.network/?utm_medium=Exinfluencer\u0026utm_source=Exinfluencer\u0026utm_content=000026UJ\u0026utm_term=10006555\u0026utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0220ENSkillsNetwork900-2022-01-01\" target=\"_blank\"\u003e\n","    \u003cimg src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  /\u003e\n","    \u003c/a\u003e\n","\u003c/p\u003e\n"]},{"cell_type":"markdown","metadata":{},"source":["# **Web Scraping Lab**\n"]},{"cell_type":"markdown","metadata":{},"source":["Estimated time needed: **30** minutes\n"]},{"cell_type":"markdown","metadata":{},"source":["## Objectives\n"]},{"cell_type":"markdown","metadata":{},"source":["After completing this lab you will be able to:\n"]},{"cell_type":"markdown","metadata":{},"source":["\u003ch2\u003eTable of Contents\u003c/h2\u003e\n","\u003cdiv class=\"alert alert-block alert-info\" style=\"margin-top: 20px\"\u003e\n","    \u003cul\u003e\n","        \u003cli\u003e\n","            \u003ca href=\"https://bso/?utm_medium=Exinfluencer\u0026utm_source=Exinfluencer\u0026utm_content=000026UJ\u0026utm_term=10006555\u0026utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0220ENSkillsNetwork23455606-2021-01-01\"\u003eBeautiful Soup Object\u003c/a\u003e\n","            \u003cul\u003e\n","                \u003cli\u003eTag\u003c/li\u003e\n","                \u003cli\u003eChildren, Parents, and Siblings\u003c/li\u003e\n","                \u003cli\u003eHTML Attributes\u003c/li\u003e\n","                \u003cli\u003eNavigable String\u003c/li\u003e\n","            \u003c/ul\u003e\n","        \u003c/li\u003e\n","     \u003c/ul\u003e\n","    \u003cul\u003e\n","        \u003cli\u003e\n","            \u003ca href=\"https://filter/?utm_medium=Exinfluencer\u0026utm_source=Exinfluencer\u0026utm_content=000026UJ\u0026utm_term=10006555\u0026utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0220ENSkillsNetwork23455606-2021-01-01\"\u003eFilter\u003c/a\u003e\n","            \u003cul\u003e\n","                \u003cli\u003efind All\u003c/li\u003e\n","                \u003cli\u003efind \u003c/li\u003e\n","                \u003cli\u003eHTML Attributes\u003c/li\u003e\n","                \u003cli\u003eNavigable String\u003c/li\u003e\n","            \u003c/ul\u003e\n","        \u003c/li\u003e\n","     \u003c/ul\u003e\n","     \u003cul\u003e\n","        \u003cli\u003e\n","            \u003ca href=\"https://dscw/?utm_medium=Exinfluencer\u0026utm_source=Exinfluencer\u0026utm_content=000026UJ\u0026utm_term=10006555\u0026utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0220ENSkillsNetwork23455606-2021-01-01\"\u003eDownloading And Scraping The Contents Of A Web\u003c/a\u003e\n","    \u003c/li\u003e\n","         \u003c/ul\u003e\n","    \u003cp\u003e\n","        Estimated time needed: \u003cstrong\u003e25 min\u003c/strong\u003e\n","    \u003c/p\u003e\n","\n","\u003c/div\u003e\n","\n","\u003chr\u003e\n"]},{"cell_type":"markdown","metadata":{},"source":["For this lab, we are going to be using Python and several Python libraries. Some of these libraries might be installed in your lab environment or in SN Labs. Others may need to be installed by you. The cells below will install these libraries when executed.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!mamba install bs4==4.10.0 -y\n","!pip install lxml==4.6.4\n","!mamba install html5lib==1.1 -y\n","# !pip install requests==2.26.0"]},{"cell_type":"markdown","metadata":{},"source":["Import the required modules and functions\n"]},{"cell_type":"code","execution_count":1,"metadata":{},"outputs":[],"source":["from bs4 import BeautifulSoup # this module helps in web scrapping.\n","import requests  # this module helps us to download a web page"]},{"cell_type":"markdown","metadata":{},"source":["\u003ch2 id=\"BSO\"\u003eBeautiful Soup Objects\u003c/h2\u003e\n"]},{"cell_type":"markdown","metadata":{},"source":["Beautiful Soup is a Python library for pulling data out of HTML and XML files, we will focus on HTML files. This is accomplished by representing the HTML as a set of objects with methods used to parse the HTML.  We can navigate the HTML as a tree and/or filter out what we are looking for.\n","\n","Consider the following HTML:\n"]},{"cell_type":"code","execution_count":2,"metadata":{},"outputs":[{"data":{"text/html":["\u003c!DOCTYPE html\u003e\n","\u003chtml\u003e\n","\u003chead\u003e\n","\u003ctitle\u003ePage Title\u003c/title\u003e\n","\u003c/head\u003e\n","\u003cbody\u003e\n","\u003ch3\u003e\u003cb id='boldest'\u003eLebron James\u003c/b\u003e\u003c/h3\u003e\n","\u003cp\u003e Salary: $ 92,000,000 \u003c/p\u003e\n","\u003ch3\u003e Stephen Curry\u003c/h3\u003e\n","\u003cp\u003e Salary: $85,000, 000 \u003c/p\u003e\n","\u003ch3\u003e Kevin Durant \u003c/h3\u003e\n","\u003cp\u003e Salary: $73,200, 000\u003c/p\u003e\n","\u003c/body\u003e\n","\u003c/html\u003e\n"],"text/plain":["\u003cIPython.core.display.HTML object\u003e"]},"metadata":{},"output_type":"display_data"}],"source":["%%html\n","\u003c!DOCTYPE html\u003e\n","\u003chtml\u003e\n","\u003chead\u003e\n","\u003ctitle\u003ePage Title\u003c/title\u003e\n","\u003c/head\u003e\n","\u003cbody\u003e\n","\u003ch3\u003e\u003cb id='boldest'\u003eLebron James\u003c/b\u003e\u003c/h3\u003e\n","\u003cp\u003e Salary: $ 92,000,000 \u003c/p\u003e\n","\u003ch3\u003e Stephen Curry\u003c/h3\u003e\n","\u003cp\u003e Salary: $85,000, 000 \u003c/p\u003e\n","\u003ch3\u003e Kevin Durant \u003c/h3\u003e\n","\u003cp\u003e Salary: $73,200, 000\u003c/p\u003e\n","\u003c/body\u003e\n","\u003c/html\u003e"]},{"cell_type":"markdown","metadata":{},"source":["We can store it as a string in the variable HTML:\n"]},{"cell_type":"code","execution_count":3,"metadata":{},"outputs":[],"source":["html=\"\u003c!DOCTYPE html\u003e\u003chtml\u003e\u003chead\u003e\u003ctitle\u003ePage Title\u003c/title\u003e\u003c/head\u003e\u003cbody\u003e\u003ch3\u003e\u003cb id='boldest'\u003eLebron James\u003c/b\u003e\u003c/h3\u003e\u003cp\u003e Salary: $ 92,000,000 \u003c/p\u003e\u003ch3\u003e Stephen Curry\u003c/h3\u003e\u003cp\u003e Salary: $85,000, 000 \u003c/p\u003e\u003ch3\u003e Kevin Durant \u003c/h3\u003e\u003cp\u003e Salary: $73,200, 000\u003c/p\u003e\u003c/body\u003e\u003c/html\u003e\""]},{"cell_type":"markdown","metadata":{},"source":["To parse a document, pass it into the \u003ccode\u003eBeautifulSoup\u003c/code\u003e constructor, the \u003ccode\u003eBeautifulSoup\u003c/code\u003e object, which represents the document as a nested data structure:\n"]},{"cell_type":"code","execution_count":4,"metadata":{},"outputs":[],"source":["soup = BeautifulSoup(html, \"html.parser\")"]},{"cell_type":"markdown","metadata":{},"source":["First, the document is converted to Unicode, (similar to ASCII),  and HTML entities are converted to Unicode characters. Beautiful Soup transforms a complex HTML document into a complex tree of Python objects. The \u003ccode\u003eBeautifulSoup\u003c/code\u003e object can create other types of objects. In this lab, we will cover \u003ccode\u003eBeautifulSoup\u003c/code\u003e and \u003ccode\u003eTag\u003c/code\u003e objects that for the purposes of this lab are identical, and \u003ccode\u003eNavigableString\u003c/code\u003e objects.\n"]},{"cell_type":"markdown","metadata":{},"source":["We can use the method \u003ccode\u003eprettify()\u003c/code\u003e to display the HTML in the nested structure:\n"]},{"cell_type":"code","execution_count":5,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["\u003c!DOCTYPE html\u003e\n","\u003chtml\u003e\n"," \u003chead\u003e\n","  \u003ctitle\u003e\n","   Page Title\n","  \u003c/title\u003e\n"," \u003c/head\u003e\n"," \u003cbody\u003e\n","  \u003ch3\u003e\n","   \u003cb id=\"boldest\"\u003e\n","    Lebron James\n","   \u003c/b\u003e\n","  \u003c/h3\u003e\n","  \u003cp\u003e\n","   Salary: $ 92,000,000\n","  \u003c/p\u003e\n","  \u003ch3\u003e\n","   Stephen Curry\n","  \u003c/h3\u003e\n","  \u003cp\u003e\n","   Salary: $85,000, 000\n","  \u003c/p\u003e\n","  \u003ch3\u003e\n","   Kevin Durant\n","  \u003c/h3\u003e\n","  \u003cp\u003e\n","   Salary: $73,200, 000\n","  \u003c/p\u003e\n"," \u003c/body\u003e\n","\u003c/html\u003e\n"]}],"source":["print(soup.prettify())"]},{"cell_type":"markdown","metadata":{},"source":["## Tags\n"]},{"cell_type":"markdown","metadata":{},"source":["Let's say we want the  title of the page and the name of the top paid player we can use the \u003ccode\u003eTag\u003c/code\u003e. The \u003ccode\u003eTag\u003c/code\u003e object corresponds to an HTML tag in the original document, for example, the tag title.\n"]},{"cell_type":"code","execution_count":6,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["tag object: \u003ctitle\u003ePage Title\u003c/title\u003e\n"]}],"source":["tag_object=soup.title\n","print(\"tag object:\",tag_object)"]},{"cell_type":"markdown","metadata":{},"source":["we can see the tag type \u003ccode\u003ebs4.element.Tag\u003c/code\u003e\n"]},{"cell_type":"code","execution_count":7,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["tag object type: \u003cclass 'bs4.element.Tag'\u003e\n"]}],"source":["print(\"tag object type:\",type(tag_object))"]},{"cell_type":"markdown","metadata":{},"source":["If there is more than one \u003ccode\u003eTag\u003c/code\u003e  with the same name, the first element with that \u003ccode\u003eTag\u003c/code\u003e name is called, this corresponds to the most paid player:\n"]},{"cell_type":"code","execution_count":8,"metadata":{},"outputs":[{"data":{"text/plain":["\u003ch3\u003e\u003cb id=\"boldest\"\u003eLebron James\u003c/b\u003e\u003c/h3\u003e"]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["tag_object=soup.h3\n","tag_object"]},{"cell_type":"markdown","metadata":{},"source":["Enclosed in the bold attribute \u003ccode\u003eb\u003c/code\u003e, it helps to use the tree representation. We can navigate down the tree using the child attribute to get the name.\n"]},{"cell_type":"markdown","metadata":{},"source":["### Children, Parents, and Siblings\n"]},{"cell_type":"markdown","metadata":{},"source":["As stated above the \u003ccode\u003eTag\u003c/code\u003e object is a tree of objects we can access the child of the tag or navigate down the branch as follows:\n"]},{"cell_type":"code","execution_count":9,"metadata":{},"outputs":[{"data":{"text/plain":["\u003cb id=\"boldest\"\u003eLebron James\u003c/b\u003e"]},"execution_count":9,"metadata":{},"output_type":"execute_result"}],"source":["tag_child =tag_object.b\n","tag_child"]},{"cell_type":"markdown","metadata":{},"source":["You can access the parent with the \u003ccode\u003e parent\u003c/code\u003e\n"]},{"cell_type":"code","execution_count":10,"metadata":{},"outputs":[{"data":{"text/plain":["\u003ch3\u003e\u003cb id=\"boldest\"\u003eLebron James\u003c/b\u003e\u003c/h3\u003e"]},"execution_count":10,"metadata":{},"output_type":"execute_result"}],"source":["parent_tag=tag_child.parent\n","parent_tag"]},{"cell_type":"markdown","metadata":{},"source":["this is identical to\n"]},{"cell_type":"code","execution_count":11,"metadata":{},"outputs":[{"data":{"text/plain":["\u003ch3\u003e\u003cb id=\"boldest\"\u003eLebron James\u003c/b\u003e\u003c/h3\u003e"]},"execution_count":11,"metadata":{},"output_type":"execute_result"}],"source":["tag_object"]},{"cell_type":"markdown","metadata":{},"source":["\u003ccode\u003etag_object\u003c/code\u003e parent is the \u003ccode\u003ebody\u003c/code\u003e element.\n"]},{"cell_type":"code","execution_count":12,"metadata":{},"outputs":[{"data":{"text/plain":["\u003cbody\u003e\u003ch3\u003e\u003cb id=\"boldest\"\u003eLebron James\u003c/b\u003e\u003c/h3\u003e\u003cp\u003e Salary: $ 92,000,000 \u003c/p\u003e\u003ch3\u003e Stephen Curry\u003c/h3\u003e\u003cp\u003e Salary: $85,000, 000 \u003c/p\u003e\u003ch3\u003e Kevin Durant \u003c/h3\u003e\u003cp\u003e Salary: $73,200, 000\u003c/p\u003e\u003c/body\u003e"]},"execution_count":12,"metadata":{},"output_type":"execute_result"}],"source":["tag_object.parent"]},{"cell_type":"markdown","metadata":{},"source":["\u003ccode\u003etag_object\u003c/code\u003e sibling is the \u003ccode\u003eparagraph\u003c/code\u003e element\n"]},{"cell_type":"code","execution_count":13,"metadata":{},"outputs":[{"data":{"text/plain":["\u003cp\u003e Salary: $ 92,000,000 \u003c/p\u003e"]},"execution_count":13,"metadata":{},"output_type":"execute_result"}],"source":["sibling_1=tag_object.next_sibling\n","sibling_1"]},{"cell_type":"markdown","metadata":{},"source":["`sibling_2` is the `header` element which is also a sibling of both `sibling_1` and `tag_object`\n"]},{"cell_type":"code","execution_count":14,"metadata":{},"outputs":[{"data":{"text/plain":["\u003ch3\u003e Stephen Curry\u003c/h3\u003e"]},"execution_count":14,"metadata":{},"output_type":"execute_result"}],"source":["sibling_2=sibling_1.next_sibling\n","sibling_2"]},{"cell_type":"markdown","metadata":{},"source":["\u003ch3 id=\"first_question\"\u003eExercise: \u003ccode\u003enext_sibling\u003c/code\u003e\u003c/h3\u003e\n"]},{"cell_type":"markdown","metadata":{},"source":["Using the object \u003ccode\u003esibling\\_2\u003c/code\u003e and the property \u003ccode\u003enext_sibling\u003c/code\u003e to find the salary of Stephen Curry:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["\u003cdetails\u003e\u003csummary\u003eClick here for the solution\u003c/summary\u003e\n","\n","```\n","sibling_2.next_sibling\n","\n","```\n","\n","\u003c/details\u003e\n"]},{"cell_type":"markdown","metadata":{},"source":["### HTML Attributes\n"]},{"cell_type":"markdown","metadata":{},"source":["If the tag has attributes, the tag \u003ccode\u003eid=\"boldest\"\u003c/code\u003e has an attribute \u003ccode\u003eid\u003c/code\u003e whose value is \u003ccode\u003eboldest\u003c/code\u003e. You can access a tag’s attributes by treating the tag like a dictionary:\n"]},{"cell_type":"code","execution_count":15,"metadata":{},"outputs":[{"data":{"text/plain":["'boldest'"]},"execution_count":15,"metadata":{},"output_type":"execute_result"}],"source":["tag_child['id']"]},{"cell_type":"markdown","metadata":{},"source":["You can access that dictionary directly as \u003ccode\u003eattrs\u003c/code\u003e:\n"]},{"cell_type":"code","execution_count":16,"metadata":{},"outputs":[{"data":{"text/plain":["{'id': 'boldest'}"]},"execution_count":16,"metadata":{},"output_type":"execute_result"}],"source":["tag_child.attrs"]},{"cell_type":"markdown","metadata":{},"source":["You can also work with Multi-valued attribute check out \u003ca href=\"https://www.crummy.com/software/BeautifulSoup/bs4/doc/?utm_medium=Exinfluencer\u0026utm_source=Exinfluencer\u0026utm_content=000026UJ\u0026utm_term=10006555\u0026utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0220ENSkillsNetwork23455606-2021-01-01\"\u003e\\[1]\u003c/a\u003e for more.\n"]},{"cell_type":"markdown","metadata":{},"source":["We can also obtain the content if the attribute of the \u003ccode\u003etag\u003c/code\u003e using the Python \u003ccode\u003eget()\u003c/code\u003e method.\n"]},{"cell_type":"code","execution_count":17,"metadata":{},"outputs":[{"data":{"text/plain":["'boldest'"]},"execution_count":17,"metadata":{},"output_type":"execute_result"}],"source":["tag_child.get('id')"]},{"cell_type":"markdown","metadata":{},"source":["### Navigable String\n"]},{"cell_type":"markdown","metadata":{},"source":["A string corresponds to a bit of text or content within a tag. Beautiful Soup uses the \u003ccode\u003eNavigableString\u003c/code\u003e class to contain this text. In our HTML we can obtain the name of the first player by extracting the sting of the \u003ccode\u003eTag\u003c/code\u003e object \u003ccode\u003etag_child\u003c/code\u003e as follows:\n"]},{"cell_type":"code","execution_count":18,"metadata":{},"outputs":[{"data":{"text/plain":["'Lebron James'"]},"execution_count":18,"metadata":{},"output_type":"execute_result"}],"source":["tag_string=tag_child.string\n","tag_string"]},{"cell_type":"markdown","metadata":{},"source":["we can verify the type is Navigable String\n"]},{"cell_type":"code","execution_count":19,"metadata":{},"outputs":[{"data":{"text/plain":["bs4.element.NavigableString"]},"execution_count":19,"metadata":{},"output_type":"execute_result"}],"source":["type(tag_string)"]},{"cell_type":"markdown","metadata":{},"source":["A NavigableString is just like a Python string or Unicode string, to be more precise. The main difference is that it also supports some  \u003ccode\u003eBeautifulSoup\u003c/code\u003e features. We can covert it to sting object in Python:\n"]},{"cell_type":"code","execution_count":20,"metadata":{},"outputs":[{"data":{"text/plain":["'Lebron James'"]},"execution_count":20,"metadata":{},"output_type":"execute_result"}],"source":["unicode_string = str(tag_string)\n","unicode_string"]},{"cell_type":"markdown","metadata":{},"source":["\u003ch2 id=\"filter\"\u003eFilter\u003c/h2\u003e\n"]},{"cell_type":"markdown","metadata":{},"source":["Filters allow you to find complex patterns, the simplest filter is a string. In this section we will pass a string to a different filter method and Beautiful Soup will perform a match against that exact string.  Consider the following HTML of rocket launchs:\n"]},{"cell_type":"code","execution_count":21,"metadata":{},"outputs":[{"data":{"text/html":["\u003ctable\u003e\n","  \u003ctr\u003e\n","    \u003ctd id='flight' \u003eFlight No\u003c/td\u003e\n","    \u003ctd\u003eLaunch site\u003c/td\u003e \n","    \u003ctd\u003ePayload mass\u003c/td\u003e\n","   \u003c/tr\u003e\n","  \u003ctr\u003e \n","    \u003ctd\u003e1\u003c/td\u003e\n","    \u003ctd\u003e\u003ca href='https://en.wikipedia.org/wiki/Florida'\u003eFlorida\u003c/a\u003e\u003c/td\u003e\n","    \u003ctd\u003e300 kg\u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003e2\u003c/td\u003e\n","    \u003ctd\u003e\u003ca href='https://en.wikipedia.org/wiki/Texas'\u003eTexas\u003c/a\u003e\u003c/td\u003e\n","    \u003ctd\u003e94 kg\u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003e3\u003c/td\u003e\n","    \u003ctd\u003e\u003ca href='https://en.wikipedia.org/wiki/Florida'\u003eFlorida\u003ca\u003e \u003c/td\u003e\n","    \u003ctd\u003e80 kg\u003c/td\u003e\n","  \u003c/tr\u003e\n","\u003c/table\u003e\n"],"text/plain":["\u003cIPython.core.display.HTML object\u003e"]},"metadata":{},"output_type":"display_data"}],"source":["%%html\n","\u003ctable\u003e\n","  \u003ctr\u003e\n","    \u003ctd id='flight' \u003eFlight No\u003c/td\u003e\n","    \u003ctd\u003eLaunch site\u003c/td\u003e \n","    \u003ctd\u003ePayload mass\u003c/td\u003e\n","   \u003c/tr\u003e\n","  \u003ctr\u003e \n","    \u003ctd\u003e1\u003c/td\u003e\n","    \u003ctd\u003e\u003ca href='https://en.wikipedia.org/wiki/Florida'\u003eFlorida\u003c/a\u003e\u003c/td\u003e\n","    \u003ctd\u003e300 kg\u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003e2\u003c/td\u003e\n","    \u003ctd\u003e\u003ca href='https://en.wikipedia.org/wiki/Texas'\u003eTexas\u003c/a\u003e\u003c/td\u003e\n","    \u003ctd\u003e94 kg\u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003e3\u003c/td\u003e\n","    \u003ctd\u003e\u003ca href='https://en.wikipedia.org/wiki/Florida'\u003eFlorida\u003ca\u003e \u003c/td\u003e\n","    \u003ctd\u003e80 kg\u003c/td\u003e\n","  \u003c/tr\u003e\n","\u003c/table\u003e"]},{"cell_type":"markdown","metadata":{},"source":["We can store it as a string in the variable \u003ccode\u003etable\u003c/code\u003e:\n"]},{"cell_type":"code","execution_count":22,"metadata":{},"outputs":[],"source":["table=\"\u003ctable\u003e\u003ctr\u003e\u003ctd id='flight'\u003eFlight No\u003c/td\u003e\u003ctd\u003eLaunch site\u003c/td\u003e \u003ctd\u003ePayload mass\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e \u003ctd\u003e1\u003c/td\u003e\u003ctd\u003e\u003ca href='https://en.wikipedia.org/wiki/Florida'\u003eFlorida\u003ca\u003e\u003c/td\u003e\u003ctd\u003e300 kg\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003e2\u003c/td\u003e\u003ctd\u003e\u003ca href='https://en.wikipedia.org/wiki/Texas'\u003eTexas\u003c/a\u003e\u003c/td\u003e\u003ctd\u003e94 kg\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003e3\u003c/td\u003e\u003ctd\u003e\u003ca href='https://en.wikipedia.org/wiki/Florida'\u003eFlorida\u003ca\u003e \u003c/td\u003e\u003ctd\u003e80 kg\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\""]},{"cell_type":"code","execution_count":23,"metadata":{},"outputs":[],"source":["table_bs = BeautifulSoup(table, \"html.parser\")"]},{"cell_type":"markdown","metadata":{},"source":["## find All\n"]},{"cell_type":"markdown","metadata":{},"source":["The \u003ccode\u003efind_all()\u003c/code\u003e method looks through a tag’s descendants and retrieves all descendants that match your filters.\n","\n","\u003cp\u003e\n","The Method signature for \u003ccode\u003efind_all(name, attrs, recursive, string, limit, **kwargs)\u003cc/ode\u003e\n","\u003c/p\u003e\n"]},{"cell_type":"markdown","metadata":{},"source":["### Name\n"]},{"cell_type":"markdown","metadata":{},"source":["When we set the \u003ccode\u003ename\u003c/code\u003e parameter to a tag name, the method will extract all the tags with that name and its children.\n"]},{"cell_type":"code","execution_count":24,"metadata":{},"outputs":[{"data":{"text/plain":["[\u003ctr\u003e\u003ctd id=\"flight\"\u003eFlight No\u003c/td\u003e\u003ctd\u003eLaunch site\u003c/td\u003e \u003ctd\u003ePayload mass\u003c/td\u003e\u003c/tr\u003e,\n"," \u003ctr\u003e \u003ctd\u003e1\u003c/td\u003e\u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e\u003c/a\u003e\u003c/a\u003e\u003c/td\u003e\u003ctd\u003e300 kg\u003c/td\u003e\u003c/tr\u003e,\n"," \u003ctr\u003e\u003ctd\u003e2\u003c/td\u003e\u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Texas\"\u003eTexas\u003c/a\u003e\u003c/td\u003e\u003ctd\u003e94 kg\u003c/td\u003e\u003c/tr\u003e,\n"," \u003ctr\u003e\u003ctd\u003e3\u003c/td\u003e\u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e \u003c/a\u003e\u003c/a\u003e\u003c/td\u003e\u003ctd\u003e80 kg\u003c/td\u003e\u003c/tr\u003e]"]},"execution_count":24,"metadata":{},"output_type":"execute_result"}],"source":["table_rows=table_bs.find_all('tr')\n","table_rows"]},{"cell_type":"markdown","metadata":{},"source":["The result is a Python Iterable just like a list, each element is a \u003ccode\u003etag\u003c/code\u003e object:\n"]},{"cell_type":"code","execution_count":25,"metadata":{},"outputs":[{"data":{"text/plain":["\u003ctr\u003e\u003ctd id=\"flight\"\u003eFlight No\u003c/td\u003e\u003ctd\u003eLaunch site\u003c/td\u003e \u003ctd\u003ePayload mass\u003c/td\u003e\u003c/tr\u003e"]},"execution_count":25,"metadata":{},"output_type":"execute_result"}],"source":["first_row =table_rows[0]\n","first_row"]},{"cell_type":"markdown","metadata":{},"source":["The type is \u003ccode\u003etag\u003c/code\u003e\n"]},{"cell_type":"code","execution_count":26,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["\u003cclass 'bs4.element.Tag'\u003e\n"]}],"source":["print(type(first_row))"]},{"cell_type":"markdown","metadata":{},"source":["we can obtain the child\n"]},{"cell_type":"code","execution_count":27,"metadata":{},"outputs":[{"data":{"text/plain":["\u003ctd id=\"flight\"\u003eFlight No\u003c/td\u003e"]},"execution_count":27,"metadata":{},"output_type":"execute_result"}],"source":["first_row.td"]},{"cell_type":"markdown","metadata":{},"source":["If we iterate through the list, each element corresponds to a row in the table:\n"]},{"cell_type":"code","execution_count":28,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["row 0 is \u003ctr\u003e\u003ctd id=\"flight\"\u003eFlight No\u003c/td\u003e\u003ctd\u003eLaunch site\u003c/td\u003e \u003ctd\u003ePayload mass\u003c/td\u003e\u003c/tr\u003e\n","row 1 is \u003ctr\u003e \u003ctd\u003e1\u003c/td\u003e\u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e\u003c/a\u003e\u003c/a\u003e\u003c/td\u003e\u003ctd\u003e300 kg\u003c/td\u003e\u003c/tr\u003e\n","row 2 is \u003ctr\u003e\u003ctd\u003e2\u003c/td\u003e\u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Texas\"\u003eTexas\u003c/a\u003e\u003c/td\u003e\u003ctd\u003e94 kg\u003c/td\u003e\u003c/tr\u003e\n","row 3 is \u003ctr\u003e\u003ctd\u003e3\u003c/td\u003e\u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e \u003c/a\u003e\u003c/a\u003e\u003c/td\u003e\u003ctd\u003e80 kg\u003c/td\u003e\u003c/tr\u003e\n"]}],"source":["for i,row in enumerate(table_rows):\n","    print(\"row\",i,\"is\",row)\n","    "]},{"cell_type":"markdown","metadata":{},"source":["As \u003ccode\u003erow\u003c/code\u003e is a \u003ccode\u003ecell\u003c/code\u003e object, we can apply the method \u003ccode\u003efind_all\u003c/code\u003e to it and extract table cells in the object \u003ccode\u003ecells\u003c/code\u003e using the tag \u003ccode\u003etd\u003c/code\u003e, this is all the children with the name \u003ccode\u003etd\u003c/code\u003e. The result is a list, each element corresponds to a cell and is a \u003ccode\u003eTag\u003c/code\u003e object, we can iterate through this list as well. We can extract the content using the \u003ccode\u003estring\u003c/code\u003e  attribute.\n"]},{"cell_type":"code","execution_count":29,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["row 0\n","colunm 0 cell \u003ctd id=\"flight\"\u003eFlight No\u003c/td\u003e\n","colunm 1 cell \u003ctd\u003eLaunch site\u003c/td\u003e\n","colunm 2 cell \u003ctd\u003ePayload mass\u003c/td\u003e\n","row 1\n","colunm 0 cell \u003ctd\u003e1\u003c/td\u003e\n","colunm 1 cell \u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e\u003c/a\u003e\u003c/a\u003e\u003c/td\u003e\n","colunm 2 cell \u003ctd\u003e300 kg\u003c/td\u003e\n","row 2\n","colunm 0 cell \u003ctd\u003e2\u003c/td\u003e\n","colunm 1 cell \u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Texas\"\u003eTexas\u003c/a\u003e\u003c/td\u003e\n","colunm 2 cell \u003ctd\u003e94 kg\u003c/td\u003e\n","row 3\n","colunm 0 cell \u003ctd\u003e3\u003c/td\u003e\n","colunm 1 cell \u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e \u003c/a\u003e\u003c/a\u003e\u003c/td\u003e\n","colunm 2 cell \u003ctd\u003e80 kg\u003c/td\u003e\n"]}],"source":["for i,row in enumerate(table_rows):\n","    print(\"row\",i)\n","    cells=row.find_all('td')\n","    for j,cell in enumerate(cells):\n","        print('colunm',j,\"cell\",cell)"]},{"cell_type":"markdown","metadata":{},"source":["If we use a list we can match against any item in that list.\n"]},{"cell_type":"code","execution_count":30,"metadata":{},"outputs":[{"data":{"text/plain":["[\u003ctr\u003e\u003ctd id=\"flight\"\u003eFlight No\u003c/td\u003e\u003ctd\u003eLaunch site\u003c/td\u003e \u003ctd\u003ePayload mass\u003c/td\u003e\u003c/tr\u003e,\n"," \u003ctd id=\"flight\"\u003eFlight No\u003c/td\u003e,\n"," \u003ctd\u003eLaunch site\u003c/td\u003e,\n"," \u003ctd\u003ePayload mass\u003c/td\u003e,\n"," \u003ctr\u003e \u003ctd\u003e1\u003c/td\u003e\u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e\u003c/a\u003e\u003c/a\u003e\u003c/td\u003e\u003ctd\u003e300 kg\u003c/td\u003e\u003c/tr\u003e,\n"," \u003ctd\u003e1\u003c/td\u003e,\n"," \u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e\u003c/a\u003e\u003c/a\u003e\u003c/td\u003e,\n"," \u003ctd\u003e300 kg\u003c/td\u003e,\n"," \u003ctr\u003e\u003ctd\u003e2\u003c/td\u003e\u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Texas\"\u003eTexas\u003c/a\u003e\u003c/td\u003e\u003ctd\u003e94 kg\u003c/td\u003e\u003c/tr\u003e,\n"," \u003ctd\u003e2\u003c/td\u003e,\n"," \u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Texas\"\u003eTexas\u003c/a\u003e\u003c/td\u003e,\n"," \u003ctd\u003e94 kg\u003c/td\u003e,\n"," \u003ctr\u003e\u003ctd\u003e3\u003c/td\u003e\u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e \u003c/a\u003e\u003c/a\u003e\u003c/td\u003e\u003ctd\u003e80 kg\u003c/td\u003e\u003c/tr\u003e,\n"," \u003ctd\u003e3\u003c/td\u003e,\n"," \u003ctd\u003e\u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e \u003c/a\u003e\u003c/a\u003e\u003c/td\u003e,\n"," \u003ctd\u003e80 kg\u003c/td\u003e]"]},"execution_count":30,"metadata":{},"output_type":"execute_result"}],"source":["list_input=table_bs .find_all(name=[\"tr\", \"td\"])\n","list_input"]},{"cell_type":"markdown","metadata":{},"source":["## Attributes\n"]},{"cell_type":"markdown","metadata":{},"source":["If the argument is not recognized it will be turned into a filter on the tag’s attributes. For example the \u003ccode\u003eid\u003c/code\u003e  argument, Beautiful Soup will filter against each tag’s \u003ccode\u003eid\u003c/code\u003e attribute. For example, the first \u003ccode\u003etd\u003c/code\u003e elements have a value of \u003ccode\u003eid\u003c/code\u003e of \u003ccode\u003eflight\u003c/code\u003e, therefore we can filter based on that \u003ccode\u003eid\u003c/code\u003e value.\n"]},{"cell_type":"code","execution_count":31,"metadata":{},"outputs":[{"data":{"text/plain":["[\u003ctd id=\"flight\"\u003eFlight No\u003c/td\u003e]"]},"execution_count":31,"metadata":{},"output_type":"execute_result"}],"source":["table_bs.find_all(id=\"flight\")"]},{"cell_type":"markdown","metadata":{},"source":["We can find all the elements that have links to the Florida Wikipedia page:\n"]},{"cell_type":"code","execution_count":32,"metadata":{},"outputs":[{"data":{"text/plain":["[\u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e\u003c/a\u003e\u003c/a\u003e,\n"," \u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e \u003c/a\u003e\u003c/a\u003e]"]},"execution_count":32,"metadata":{},"output_type":"execute_result"}],"source":["list_input=table_bs.find_all(href=\"https://en.wikipedia.org/wiki/Florida\")\n","list_input"]},{"cell_type":"markdown","metadata":{},"source":["If we set the  \u003ccode\u003ehref\u003c/code\u003e attribute to True, regardless of what the value is, the code finds all tags with \u003ccode\u003ehref\u003c/code\u003e value:\n"]},{"cell_type":"code","execution_count":33,"metadata":{},"outputs":[{"data":{"text/plain":["[\u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e\u003c/a\u003e\u003c/a\u003e,\n"," \u003ca href=\"https://en.wikipedia.org/wiki/Texas\"\u003eTexas\u003c/a\u003e,\n"," \u003ca href=\"https://en.wikipedia.org/wiki/Florida\"\u003eFlorida\u003ca\u003e \u003c/a\u003e\u003c/a\u003e]"]},"execution_count":33,"metadata":{},"output_type":"execute_result"}],"source":["table_bs.find_all(href=True)"]},{"cell_type":"markdown","metadata":{},"source":["There are other methods for dealing with attributes and other related methods; Check out the following \u003ca href='https://www.crummy.com/software/BeautifulSoup/bs4/doc/?utm_medium=Exinfluencer\u0026utm_source=Exinfluencer\u0026utm_content=000026UJ\u0026utm_term=10006555\u0026utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0220ENSkillsNetwork23455606-2021-01-01#css-selectors'\u003elink\u003c/a\u003e\n"]},{"cell_type":"markdown","metadata":{},"source":["\u003ch3 id=\"exer_type\"\u003eExercise: \u003ccode\u003efind_all\u003c/code\u003e\u003c/h3\u003e\n"]},{"cell_type":"markdown","metadata":{},"source":["Using the logic above, find all the elements without \u003ccode\u003ehref\u003c/code\u003e value\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["\u003cdetails\u003e\u003csummary\u003eClick here for the solution\u003c/summary\u003e\n","\n","```\n","table_bs.find_all(href=False)\n","\n","```\n","\n","\u003c/details\u003e\n"]},{"cell_type":"markdown","metadata":{},"source":["Using the soup object \u003ccode\u003esoup\u003c/code\u003e, find the element with the \u003ccode\u003eid\u003c/code\u003e attribute content set to \u003ccode\u003e\"boldest\"\u003c/code\u003e.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["\u003cdetails\u003e\u003csummary\u003eClick here for the solution\u003c/summary\u003e\n","\n","```\n","soup.find_all(id=\"boldest\")\n","\n","```\n","\n","\u003c/details\u003e\n"]},{"cell_type":"markdown","metadata":{},"source":["### string\n"]},{"cell_type":"markdown","metadata":{},"source":["With string you can search for strings instead of tags, where we find all the elments with Florida:\n"]},{"cell_type":"code","execution_count":34,"metadata":{},"outputs":[{"data":{"text/plain":["['Florida', 'Florida']"]},"execution_count":34,"metadata":{},"output_type":"execute_result"}],"source":["table_bs.find_all(string=\"Florida\")"]},{"cell_type":"markdown","metadata":{},"source":["## find\n"]},{"cell_type":"markdown","metadata":{},"source":["The \u003ccode\u003efind_all()\u003c/code\u003e method scans the entire document looking for results, it’s if you are looking for one element you can use the \u003ccode\u003efind()\u003c/code\u003e method to find the first element in the document. Consider the following two table:\n"]},{"cell_type":"code","execution_count":35,"metadata":{},"outputs":[{"data":{"text/html":["\u003ch3\u003eRocket Launch \u003c/h3\u003e\n","\n","\u003cp\u003e\n","\u003ctable class='rocket'\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003eFlight No\u003c/td\u003e\n","    \u003ctd\u003eLaunch site\u003c/td\u003e \n","    \u003ctd\u003ePayload mass\u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003e1\u003c/td\u003e\n","    \u003ctd\u003eFlorida\u003c/td\u003e\n","    \u003ctd\u003e300 kg\u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003e2\u003c/td\u003e\n","    \u003ctd\u003eTexas\u003c/td\u003e\n","    \u003ctd\u003e94 kg\u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003e3\u003c/td\u003e\n","    \u003ctd\u003eFlorida \u003c/td\u003e\n","    \u003ctd\u003e80 kg\u003c/td\u003e\n","  \u003c/tr\u003e\n","\u003c/table\u003e\n","\u003c/p\u003e\n","\u003cp\u003e\n","\n","\u003ch3\u003ePizza Party  \u003c/h3\u003e\n","  \n","    \n","\u003ctable class='pizza'\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003ePizza Place\u003c/td\u003e\n","    \u003ctd\u003eOrders\u003c/td\u003e \n","    \u003ctd\u003eSlices \u003c/td\u003e\n","   \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003eDomino's Pizza\u003c/td\u003e\n","    \u003ctd\u003e10\u003c/td\u003e\n","    \u003ctd\u003e100\u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003eLittle Caesars\u003c/td\u003e\n","    \u003ctd\u003e12\u003c/td\u003e\n","    \u003ctd \u003e144 \u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003ePapa John's \u003c/td\u003e\n","    \u003ctd\u003e15 \u003c/td\u003e\n","    \u003ctd\u003e165\u003c/td\u003e\n","  \u003c/tr\u003e\n"],"text/plain":["\u003cIPython.core.display.HTML object\u003e"]},"metadata":{},"output_type":"display_data"}],"source":["%%html\n","\u003ch3\u003eRocket Launch \u003c/h3\u003e\n","\n","\u003cp\u003e\n","\u003ctable class='rocket'\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003eFlight No\u003c/td\u003e\n","    \u003ctd\u003eLaunch site\u003c/td\u003e \n","    \u003ctd\u003ePayload mass\u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003e1\u003c/td\u003e\n","    \u003ctd\u003eFlorida\u003c/td\u003e\n","    \u003ctd\u003e300 kg\u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003e2\u003c/td\u003e\n","    \u003ctd\u003eTexas\u003c/td\u003e\n","    \u003ctd\u003e94 kg\u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003e3\u003c/td\u003e\n","    \u003ctd\u003eFlorida \u003c/td\u003e\n","    \u003ctd\u003e80 kg\u003c/td\u003e\n","  \u003c/tr\u003e\n","\u003c/table\u003e\n","\u003c/p\u003e\n","\u003cp\u003e\n","\n","\u003ch3\u003ePizza Party  \u003c/h3\u003e\n","  \n","    \n","\u003ctable class='pizza'\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003ePizza Place\u003c/td\u003e\n","    \u003ctd\u003eOrders\u003c/td\u003e \n","    \u003ctd\u003eSlices \u003c/td\u003e\n","   \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003eDomino's Pizza\u003c/td\u003e\n","    \u003ctd\u003e10\u003c/td\u003e\n","    \u003ctd\u003e100\u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003eLittle Caesars\u003c/td\u003e\n","    \u003ctd\u003e12\u003c/td\u003e\n","    \u003ctd \u003e144 \u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","    \u003ctd\u003ePapa John's \u003c/td\u003e\n","    \u003ctd\u003e15 \u003c/td\u003e\n","    \u003ctd\u003e165\u003c/td\u003e\n","  \u003c/tr\u003e\n"]},{"cell_type":"markdown","metadata":{},"source":["We store the HTML as a Python string and assign \u003ccode\u003etwo_tables\u003c/code\u003e:\n"]},{"cell_type":"code","execution_count":36,"metadata":{},"outputs":[],"source":["two_tables=\"\u003ch3\u003eRocket Launch \u003c/h3\u003e\u003cp\u003e\u003ctable class='rocket'\u003e\u003ctr\u003e\u003ctd\u003eFlight No\u003c/td\u003e\u003ctd\u003eLaunch site\u003c/td\u003e \u003ctd\u003ePayload mass\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003e1\u003c/td\u003e\u003ctd\u003eFlorida\u003c/td\u003e\u003ctd\u003e300 kg\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003e2\u003c/td\u003e\u003ctd\u003eTexas\u003c/td\u003e\u003ctd\u003e94 kg\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003e3\u003c/td\u003e\u003ctd\u003eFlorida \u003c/td\u003e\u003ctd\u003e80 kg\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\u003c/p\u003e\u003cp\u003e\u003ch3\u003ePizza Party  \u003c/h3\u003e\u003ctable class='pizza'\u003e\u003ctr\u003e\u003ctd\u003ePizza Place\u003c/td\u003e\u003ctd\u003eOrders\u003c/td\u003e \u003ctd\u003eSlices \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eDomino's Pizza\u003c/td\u003e\u003ctd\u003e10\u003c/td\u003e\u003ctd\u003e100\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eLittle Caesars\u003c/td\u003e\u003ctd\u003e12\u003c/td\u003e\u003ctd \u003e144 \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003ePapa John's \u003c/td\u003e\u003ctd\u003e15 \u003c/td\u003e\u003ctd\u003e165\u003c/td\u003e\u003c/tr\u003e\""]},{"cell_type":"markdown","metadata":{},"source":["We create a \u003ccode\u003eBeautifulSoup\u003c/code\u003e object  \u003ccode\u003etwo_tables_bs\u003c/code\u003e\n"]},{"cell_type":"code","execution_count":37,"metadata":{},"outputs":[],"source":["two_tables_bs= BeautifulSoup(two_tables, 'html.parser')"]},{"cell_type":"markdown","metadata":{},"source":["We can find the first table using the tag name table\n"]},{"cell_type":"code","execution_count":38,"metadata":{},"outputs":[{"data":{"text/plain":["\u003ctable class=\"rocket\"\u003e\u003ctr\u003e\u003ctd\u003eFlight No\u003c/td\u003e\u003ctd\u003eLaunch site\u003c/td\u003e \u003ctd\u003ePayload mass\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003e1\u003c/td\u003e\u003ctd\u003eFlorida\u003c/td\u003e\u003ctd\u003e300 kg\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003e2\u003c/td\u003e\u003ctd\u003eTexas\u003c/td\u003e\u003ctd\u003e94 kg\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003e3\u003c/td\u003e\u003ctd\u003eFlorida \u003c/td\u003e\u003ctd\u003e80 kg\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e"]},"execution_count":38,"metadata":{},"output_type":"execute_result"}],"source":["two_tables_bs.find(\"table\")"]},{"cell_type":"markdown","metadata":{},"source":["We can filter on the class attribute to find the second table, but because class is a keyword in Python, we add an underscore.\n"]},{"cell_type":"code","execution_count":39,"metadata":{},"outputs":[{"data":{"text/plain":["\u003ctable class=\"pizza\"\u003e\u003ctr\u003e\u003ctd\u003ePizza Place\u003c/td\u003e\u003ctd\u003eOrders\u003c/td\u003e \u003ctd\u003eSlices \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eDomino's Pizza\u003c/td\u003e\u003ctd\u003e10\u003c/td\u003e\u003ctd\u003e100\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eLittle Caesars\u003c/td\u003e\u003ctd\u003e12\u003c/td\u003e\u003ctd\u003e144 \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003ePapa John's \u003c/td\u003e\u003ctd\u003e15 \u003c/td\u003e\u003ctd\u003e165\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e"]},"execution_count":39,"metadata":{},"output_type":"execute_result"}],"source":["two_tables_bs.find(\"table\",class_='pizza')"]},{"cell_type":"markdown","metadata":{},"source":["\u003ch2 id=\"DSCW\"\u003eDownloading And Scraping The Contents Of A Web Page\u003c/h2\u003e \n"]},{"cell_type":"markdown","metadata":{},"source":["We Download the contents of the web page:\n"]},{"cell_type":"code","execution_count":40,"metadata":{},"outputs":[],"source":["url = \"http://www.ibm.com\""]},{"cell_type":"markdown","metadata":{},"source":["We use \u003ccode\u003eget\u003c/code\u003e to download the contents of the webpage in text format and store in a variable called \u003ccode\u003edata\u003c/code\u003e:\n"]},{"cell_type":"code","execution_count":41,"metadata":{},"outputs":[],"source":["data  = requests.get(url).text "]},{"cell_type":"markdown","metadata":{},"source":["We create a \u003ccode\u003eBeautifulSoup\u003c/code\u003e object using the \u003ccode\u003eBeautifulSoup\u003c/code\u003e constructor\n"]},{"cell_type":"code","execution_count":42,"metadata":{},"outputs":[],"source":["soup = BeautifulSoup(data,\"html.parser\")  # create a soup object using the variable 'data'"]},{"cell_type":"markdown","metadata":{},"source":["Scrape all links\n"]},{"cell_type":"code","execution_count":43,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["#main-content\n","http://www.ibm.com\n","https://www.ibm.com/cloud/paks?lnk=ushpv18l1\n","https://www.ibm.com/security/executive-order-cybersecurity?lnk=ushpv18f1\n","https://www.ibm.com/consulting/technology/?lnk=ushpv18f2\n","https://www.ibm.com/training/credentials?lnk=ushpv18f3\n","https://www.ibm.com/blogs/blockchain/2021/09/dont-let-the-shipping-container-crisis-ruin-your-holidays-this-year/?lnk=ushpv18f4\n","https://www.ibm.com/products/offers-and-discounts?link=ushpv18t5\u0026lnk2=trial_mktpl_MPDISC\n","https://www.ibm.com/cloud/cloud-pak-for-automation?lnk=ushpv18t1\u0026lnk2=trial_CloudPakAtm\u0026psrc=none\u0026pexp=def\n","https://www.ibm.com/cloud/watson-studio?lnk=ushpv18t2\u0026lnk2=trial_WatStudio\u0026psrc=none\u0026pexp=def\n","https://www.ibm.com/cloud/aspera?lnk=ushpv18t3\u0026lnk2=trial_AsperaCloud\u0026psrc=none\u0026pexp=def\n","https://www.ibm.com/security/identity-access-management/cloud-identity?lnk=ushpv18t4\u0026lnk2=trial_Verify\u0026psrc=none\u0026pexp=def\n","https://www.ibm.com/search?lnk=ushpv18srch\u0026locale=en-us\u0026q=\n","https://www.ibm.com/products?lnk=ushpv18p1\u0026lnk2=trial_mktpl\u0026psrc=none\u0026pexp=def\n","https://www.ibm.com/cloud/hybrid?lnk=ushpv18pt14\n","https://www.ibm.com/watson?lnk=ushpv18pt17\n","https://www.ibm.com/it-infrastructure?lnk=ushpv18pt19\n","https://www.ibm.com/us-en/products/categories?technologyTopics%5B0%5D%5B0%5D=cat.topic:Blockchain\u0026isIBMOffering%5B0%5D=true\u0026lnk=ushpv18pt4\n","https://www.ibm.com/us-en/products/category/technology/security?lnk=ushpv18pt9\n","https://www.ibm.com/us-en/products/category/technology/analytics?lnk=ushpv18pt1\n","https://www.ibm.com/cloud/automation?lnk=ushpv18ct21\n","https://www.ibm.com/quantum-computing?lnk=ushpv18pt16\n","https://www.ibm.com/mysupport/s/?language=en_US\u0026lnk=ushpv18ct11\n","https://www.ibm.com/training/?lnk=ushpv18ct15\n","https://developer.ibm.com/?lnk=ushpv18ct9\n","https://www.ibm.com/garage?lnk=ushpv18pt18\n","https://www.ibm.com/docs/en?lnk=ushpv18ct14\n","https://www.redbooks.ibm.com/?lnk=ushpv18ct10\n","https://www-03.ibm.com/employment/technicaltalent/developer/?lnk=ushpv18ct2\n","https://www.ibm.com/case-studies/verizon-business/?lnk=ushpv18vn1\n","https://www.ibm.com/case-studies/verizon-business/?lnk=ushpv18vn1\n","https://www.ibm.com/\n"]}],"source":["for link in soup.find_all('a',href=True):  # in html anchor/link is represented by the tag \u003ca\u003e\n","\n","    print(link.get('href'))\n"]},{"cell_type":"markdown","metadata":{},"source":["## Scrape  all images  Tags\n"]},{"cell_type":"code","execution_count":44,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["\u003cimg alt=\"\" aria-hidden=\"true\" role=\"presentation\" src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMTA1NSIgaGVpZ2h0PSI1MjcuNSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\" style=\"max-width:100%;display:block;margin:0;border:none;padding:0\"/\u003e\n","data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMTA1NSIgaGVpZ2h0PSI1MjcuNSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\n","\u003cimg alt=\"leadspace mobile image\" class=\"ibm-resize\" decoding=\"async\" src=\"https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/2f/bc/20211115-ls-cloud-paks-26257-720x360.jpg\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/2f/bc/20211115-ls-cloud-paks-26257-720x360.jpg\n","\u003cimg alt=\"\" aria-hidden=\"true\" role=\"presentation\" src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjMyMCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\" style=\"max-width:100%;display:block;margin:0;border:none;padding:0\"/\u003e\n","data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjMyMCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\n","\u003cimg alt=\"U.S. Executive Order 14028\" class=\"ibm-resize ibm-ab-image featured-image\" decoding=\"async\" src=\"https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/80/38/20211107-26227%20X-Force-executive-order-444x320.jpg\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/80/38/20211107-26227%20X-Force-executive-order-444x320.jpg\n","\u003cimg alt=\"\" aria-hidden=\"true\" role=\"presentation\" src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjMyMCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\" style=\"max-width:100%;display:block;margin:0;border:none;padding:0\"/\u003e\n","data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjMyMCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\n","\u003cimg alt=\"IBM Consulting for technology\" class=\"ibm-resize ibm-ab-image featured-image\" decoding=\"async\" src=\"https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/57/06/20211101-f-consulting-technology-26225.jpg\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/57/06/20211101-f-consulting-technology-26225.jpg\n","\u003cimg alt=\"\" aria-hidden=\"true\" role=\"presentation\" src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjMyMCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\" style=\"max-width:100%;display:block;margin:0;border:none;padding:0\"/\u003e\n","data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjMyMCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\n","\u003cimg alt=\"To accelerate your career, start\u0026amp;nbsp;here\" class=\"ibm-resize ibm-ab-image featured-image\" decoding=\"async\" src=\"https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/fd/39/20211107-26176-credential-experience-444x320.jpg\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/fd/39/20211107-26176-credential-experience-444x320.jpg\n","\u003cimg alt=\"\" aria-hidden=\"true\" role=\"presentation\" src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjMyMCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\" style=\"max-width:100%;display:block;margin:0;border:none;padding:0\"/\u003e\n","data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjMyMCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\n","\u003cimg alt=\"Demand is up but your supply isn\u0026amp;#8217;t?\" class=\"ibm-resize ibm-ab-image featured-image\" decoding=\"async\" src=\"https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/37/67/20211107-26238-supply-chain-crisis-444x320.jpg\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/37/67/20211107-26238-supply-chain-crisis-444x320.jpg\n","\u003cimg alt=\"\" aria-hidden=\"true\" role=\"presentation\" src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjI2MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\" style=\"max-width:100%;display:block;margin:0;border:none;padding:0\"/\u003e\n","data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjI2MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\n","\u003cimg alt=\"IBM Cloud Pak for Automation\" class=\"ibm-resize ibm-ab-image trials-image\" decoding=\"async\" src=\"https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/ab/f7/Cloud-pak-for-automation-444x260.png\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/ab/f7/Cloud-pak-for-automation-444x260.png\n","\u003cimg alt=\"IBM Cloud Pak for Automation\" class=\"ibm-resize ibm-ab-image trials-image\" decoding=\"async\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7\n","\u003cimg alt=\"\" aria-hidden=\"true\" role=\"presentation\" src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjI2MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\" style=\"max-width:100%;display:block;margin:0;border:none;padding:0\"/\u003e\n","data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjI2MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\n","\u003cimg alt=\"IBM Watson Studio\" class=\"ibm-resize ibm-ab-image trials-image\" decoding=\"async\" src=\"https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/58/44/Watson-Studio-Desktop-21039-700x420.png\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/58/44/Watson-Studio-Desktop-21039-700x420.png\n","\u003cimg alt=\"IBM Watson Studio\" class=\"ibm-resize ibm-ab-image trials-image\" decoding=\"async\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7\n","\u003cimg alt=\"\" aria-hidden=\"true\" role=\"presentation\" src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjI2MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\" style=\"max-width:100%;display:block;margin:0;border:none;padding:0\"/\u003e\n","data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjI2MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\n","\u003cimg alt=\"IBM Aspera on Cloud\" class=\"ibm-resize ibm-ab-image trials-image\" decoding=\"async\" src=\"https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/e5/32/Aspera-on-Cloud-19783-700x420.png\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/e5/32/Aspera-on-Cloud-19783-700x420.png\n","\u003cimg alt=\"IBM Aspera on Cloud\" class=\"ibm-resize ibm-ab-image trials-image\" decoding=\"async\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7\n","\u003cimg alt=\"\" aria-hidden=\"true\" role=\"presentation\" src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjI2MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\" style=\"max-width:100%;display:block;margin:0;border:none;padding:0\"/\u003e\n","data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNDQwIiBoZWlnaHQ9IjI2MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2ZXJzaW9uPSIxLjEiLz4=\n","\u003cimg alt=\"IBM Security Verify\" class=\"ibm-resize ibm-ab-image trials-image\" decoding=\"async\" src=\"https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/15/16/cloud-transformation-trial-700x420.png\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","https://1.dam.s81c.com/public/content/dam/worldwide-content/homepage/ul/g/15/16/cloud-transformation-trial-700x420.png\n","\u003cimg alt=\"IBM Security Verify\" class=\"ibm-resize ibm-ab-image trials-image\" decoding=\"async\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7\" style=\"position:absolute;top:0;left:0;bottom:0;right:0;box-sizing:border-box;padding:0;border:none;margin:auto;display:block;width:0;height:0;min-width:100%;max-width:100%;min-height:100%;max-height:100%\"/\u003e\n","data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7\n"]}],"source":["for link in soup.find_all('img'):# in html image is represented by the tag \u003cimg\u003e\n","    print(link)\n","    print(link.get('src'))"]},{"cell_type":"markdown","metadata":{},"source":["## Scrape data from HTML tables\n"]},{"cell_type":"code","execution_count":45,"metadata":{},"outputs":[],"source":["#The below url contains an html table with data about colors and color codes.\n","url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/datasets/HTMLColorCodes.html\""]},{"cell_type":"markdown","metadata":{},"source":["Before proceeding to scrape a web site, you need to examine the contents, and the way data is organized on the website. Open the above url in your browser and check how many rows and columns are there in the color table.\n"]},{"cell_type":"code","execution_count":46,"metadata":{},"outputs":[],"source":["# get the contents of the webpage in text format and store in a variable called data\n","data  = requests.get(url).text"]},{"cell_type":"code","execution_count":47,"metadata":{},"outputs":[],"source":["soup = BeautifulSoup(data,\"html.parser\")"]},{"cell_type":"code","execution_count":48,"metadata":{},"outputs":[],"source":["#find a html table in the web page\n","table = soup.find('table') # in html table is represented by the tag \u003ctable\u003e"]},{"cell_type":"code","execution_count":49,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["Color Name---\u003eNone\n","lightsalmon---\u003e#FFA07A\n","salmon---\u003e#FA8072\n","darksalmon---\u003e#E9967A\n","lightcoral---\u003e#F08080\n","coral---\u003e#FF7F50\n","tomato---\u003e#FF6347\n","orangered---\u003e#FF4500\n","gold---\u003e#FFD700\n","orange---\u003e#FFA500\n","darkorange---\u003e#FF8C00\n","lightyellow---\u003e#FFFFE0\n","lemonchiffon---\u003e#FFFACD\n","papayawhip---\u003e#FFEFD5\n","moccasin---\u003e#FFE4B5\n","peachpuff---\u003e#FFDAB9\n","palegoldenrod---\u003e#EEE8AA\n","khaki---\u003e#F0E68C\n","darkkhaki---\u003e#BDB76B\n","yellow---\u003e#FFFF00\n","lawngreen---\u003e#7CFC00\n","chartreuse---\u003e#7FFF00\n","limegreen---\u003e#32CD32\n","lime---\u003e#00FF00\n","forestgreen---\u003e#228B22\n","green---\u003e#008000\n","powderblue---\u003e#B0E0E6\n","lightblue---\u003e#ADD8E6\n","lightskyblue---\u003e#87CEFA\n","skyblue---\u003e#87CEEB\n","deepskyblue---\u003e#00BFFF\n","lightsteelblue---\u003e#B0C4DE\n","dodgerblue---\u003e#1E90FF\n"]}],"source":["#Get all rows from the table\n","for row in table.find_all('tr'): # in html table row is represented by the tag \u003ctr\u003e\n","    # Get all columns in each row.\n","    cols = row.find_all('td') # in html a column is represented by the tag \u003ctd\u003e\n","    color_name = cols[2].string # store the value in column 3 as color_name\n","    color_code = cols[3].string # store the value in column 4 as color_code\n","    print(\"{}---\u003e{}\".format(color_name,color_code))"]},{"cell_type":"markdown","metadata":{},"source":["## Scrape data from HTML tables into a DataFrame using BeautifulSoup and Pandas\n"]},{"cell_type":"code","execution_count":50,"metadata":{},"outputs":[],"source":["import pandas as pd"]},{"cell_type":"code","execution_count":51,"metadata":{},"outputs":[],"source":["#The below url contains html tables with data about world population.\n","url = \"https://en.wikipedia.org/wiki/World_population\""]},{"cell_type":"markdown","metadata":{},"source":["Before proceeding to scrape a web site, you need to examine the contents, and the way data is organized on the website. Open the above url in your browser and check the tables on the webpage.\n"]},{"cell_type":"code","execution_count":52,"metadata":{},"outputs":[],"source":["# get the contents of the webpage in text format and store in a variable called data\n","data  = requests.get(url).text"]},{"cell_type":"code","execution_count":53,"metadata":{},"outputs":[],"source":["soup = BeautifulSoup(data,\"html.parser\")"]},{"cell_type":"code","execution_count":54,"metadata":{},"outputs":[],"source":["#find all html tables in the web page\n","tables = soup.find_all('table') # in html table is represented by the tag \u003ctable\u003e"]},{"cell_type":"code","execution_count":55,"metadata":{},"outputs":[{"data":{"text/plain":["26"]},"execution_count":55,"metadata":{},"output_type":"execute_result"}],"source":["# we can see how many tables were found by checking the length of the tables list\n","len(tables)"]},{"cell_type":"markdown","metadata":{},"source":["Assume that we are looking for the `10 most densly populated countries` table, we can look through the tables list and find the right one we are look for based on the data in each table or we can search for the table name if it is in the table but this option might not always work.\n"]},{"cell_type":"code","execution_count":56,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["5\n"]}],"source":["for index,table in enumerate(tables):\n","    if (\"10 most densely populated countries\" in str(table)):\n","        table_index = index\n","print(table_index)"]},{"cell_type":"markdown","metadata":{},"source":["See if you can locate the table name of the table, `10 most densly populated countries`, below.\n"]},{"cell_type":"code","execution_count":57,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["\u003ctable class=\"wikitable sortable\" style=\"text-align:right\"\u003e\n"," \u003ccaption\u003e\n","  10 most densely populated countries\n","  \u003csmall\u003e\n","   (with population above 5 million)\n","  \u003c/small\u003e\n"," \u003c/caption\u003e\n"," \u003ctbody\u003e\n","  \u003ctr\u003e\n","   \u003cth\u003e\n","    Rank\n","   \u003c/th\u003e\n","   \u003cth\u003e\n","    Country\n","   \u003c/th\u003e\n","   \u003cth\u003e\n","    Population\n","   \u003c/th\u003e\n","   \u003cth\u003e\n","    Area\n","    \u003cbr/\u003e\n","    \u003csmall\u003e\n","     (km\n","     \u003csup\u003e\n","      2\n","     \u003c/sup\u003e\n","     )\n","    \u003c/small\u003e\n","   \u003c/th\u003e\n","   \u003cth\u003e\n","    Density\n","    \u003cbr/\u003e\n","    \u003csmall\u003e\n","     (pop/km\n","     \u003csup\u003e\n","      2\n","     \u003c/sup\u003e\n","     )\n","    \u003c/small\u003e\n","   \u003c/th\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","   \u003ctd\u003e\n","    1\n","   \u003c/td\u003e\n","   \u003ctd align=\"left\"\u003e\n","    \u003cspan class=\"flagicon\"\u003e\n","     \u003cimg alt=\"\" class=\"thumbborder\" data-file-height=\"600\" data-file-width=\"900\" decoding=\"async\" height=\"15\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/4/48/Flag_of_Singapore.svg/23px-Flag_of_Singapore.svg.png\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/4/48/Flag_of_Singapore.svg/35px-Flag_of_Singapore.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/4/48/Flag_of_Singapore.svg/45px-Flag_of_Singapore.svg.png 2x\" width=\"23\"/\u003e\n","    \u003c/span\u003e\n","    \u003ca href=\"/wiki/Singapore\" title=\"Singapore\"\u003e\n","     Singapore\n","    \u003c/a\u003e\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    5,704,000\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    710\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    8,033\n","   \u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","   \u003ctd\u003e\n","    2\n","   \u003c/td\u003e\n","   \u003ctd align=\"left\"\u003e\n","    \u003cspan class=\"flagicon\"\u003e\n","     \u003cimg alt=\"\" class=\"thumbborder\" data-file-height=\"600\" data-file-width=\"1000\" decoding=\"async\" height=\"14\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Flag_of_Bangladesh.svg/23px-Flag_of_Bangladesh.svg.png\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Flag_of_Bangladesh.svg/35px-Flag_of_Bangladesh.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Flag_of_Bangladesh.svg/46px-Flag_of_Bangladesh.svg.png 2x\" width=\"23\"/\u003e\n","    \u003c/span\u003e\n","    \u003ca href=\"/wiki/Bangladesh\" title=\"Bangladesh\"\u003e\n","     Bangladesh\n","    \u003c/a\u003e\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    171,670,000\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    143,998\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    1,192\n","   \u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","   \u003ctd\u003e\n","    3\n","   \u003c/td\u003e\n","   \u003ctd align=\"left\"\u003e\n","    \u003cp\u003e\n","     \u003cspan class=\"flagicon\"\u003e\n","      \u003cimg alt=\"\" class=\"thumbborder\" data-file-height=\"600\" data-file-width=\"1200\" decoding=\"async\" height=\"12\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/0/00/Flag_of_Palestine.svg/23px-Flag_of_Palestine.svg.png\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/0/00/Flag_of_Palestine.svg/35px-Flag_of_Palestine.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/0/00/Flag_of_Palestine.svg/46px-Flag_of_Palestine.svg.png 2x\" width=\"23\"/\u003e\n","     \u003c/span\u003e\n","     \u003ca href=\"/wiki/State_of_Palestine\" title=\"State of Palestine\"\u003e\n","      Palestine\n","     \u003c/a\u003e\n","    \u003c/p\u003e\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    5,266,785\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    6,020\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    847\n","   \u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","   \u003ctd\u003e\n","    4\n","   \u003c/td\u003e\n","   \u003ctd align=\"left\"\u003e\n","    \u003cspan class=\"flagicon\"\u003e\n","     \u003cimg alt=\"\" class=\"thumbborder\" data-file-height=\"600\" data-file-width=\"900\" decoding=\"async\" height=\"15\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/5/59/Flag_of_Lebanon.svg/23px-Flag_of_Lebanon.svg.png\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/5/59/Flag_of_Lebanon.svg/35px-Flag_of_Lebanon.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/5/59/Flag_of_Lebanon.svg/45px-Flag_of_Lebanon.svg.png 2x\" width=\"23\"/\u003e\n","    \u003c/span\u003e\n","    \u003ca href=\"/wiki/Lebanon\" title=\"Lebanon\"\u003e\n","     Lebanon\n","    \u003c/a\u003e\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    6,856,000\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    10,452\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    656\n","   \u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","   \u003ctd\u003e\n","    5\n","   \u003c/td\u003e\n","   \u003ctd align=\"left\"\u003e\n","    \u003cspan class=\"flagicon\"\u003e\n","     \u003cimg alt=\"\" class=\"thumbborder\" data-file-height=\"600\" data-file-width=\"900\" decoding=\"async\" height=\"15\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/7/72/Flag_of_the_Republic_of_China.svg/23px-Flag_of_the_Republic_of_China.svg.png\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/7/72/Flag_of_the_Republic_of_China.svg/35px-Flag_of_the_Republic_of_China.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/7/72/Flag_of_the_Republic_of_China.svg/45px-Flag_of_the_Republic_of_China.svg.png 2x\" width=\"23\"/\u003e\n","    \u003c/span\u003e\n","    \u003ca href=\"/wiki/Taiwan\" title=\"Taiwan\"\u003e\n","     Taiwan\n","    \u003c/a\u003e\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    23,604,000\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    36,193\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    652\n","   \u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","   \u003ctd\u003e\n","    6\n","   \u003c/td\u003e\n","   \u003ctd align=\"left\"\u003e\n","    \u003cspan class=\"flagicon\"\u003e\n","     \u003cimg alt=\"\" class=\"thumbborder\" data-file-height=\"600\" data-file-width=\"900\" decoding=\"async\" height=\"15\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/0/09/Flag_of_South_Korea.svg/23px-Flag_of_South_Korea.svg.png\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/0/09/Flag_of_South_Korea.svg/35px-Flag_of_South_Korea.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/0/09/Flag_of_South_Korea.svg/45px-Flag_of_South_Korea.svg.png 2x\" width=\"23\"/\u003e\n","    \u003c/span\u003e\n","    \u003ca href=\"/wiki/South_Korea\" title=\"South Korea\"\u003e\n","     South Korea\n","    \u003c/a\u003e\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    51,781,000\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    99,538\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    520\n","   \u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","   \u003ctd\u003e\n","    7\n","   \u003c/td\u003e\n","   \u003ctd align=\"left\"\u003e\n","    \u003cspan class=\"flagicon\"\u003e\n","     \u003cimg alt=\"\" class=\"thumbborder\" data-file-height=\"720\" data-file-width=\"1080\" decoding=\"async\" height=\"15\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/1/17/Flag_of_Rwanda.svg/23px-Flag_of_Rwanda.svg.png\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/1/17/Flag_of_Rwanda.svg/35px-Flag_of_Rwanda.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/1/17/Flag_of_Rwanda.svg/45px-Flag_of_Rwanda.svg.png 2x\" width=\"23\"/\u003e\n","    \u003c/span\u003e\n","    \u003ca href=\"/wiki/Rwanda\" title=\"Rwanda\"\u003e\n","     Rwanda\n","    \u003c/a\u003e\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    12,374,000\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    26,338\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    470\n","   \u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","   \u003ctd\u003e\n","    8\n","   \u003c/td\u003e\n","   \u003ctd align=\"left\"\u003e\n","    \u003cspan class=\"flagicon\"\u003e\n","     \u003cimg alt=\"\" class=\"thumbborder\" data-file-height=\"600\" data-file-width=\"1000\" decoding=\"async\" height=\"14\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/5/56/Flag_of_Haiti.svg/23px-Flag_of_Haiti.svg.png\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/5/56/Flag_of_Haiti.svg/35px-Flag_of_Haiti.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/5/56/Flag_of_Haiti.svg/46px-Flag_of_Haiti.svg.png 2x\" width=\"23\"/\u003e\n","    \u003c/span\u003e\n","    \u003ca href=\"/wiki/Haiti\" title=\"Haiti\"\u003e\n","     Haiti\n","    \u003c/a\u003e\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    11,578,000\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    27,065\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    428\n","   \u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","   \u003ctd\u003e\n","    9\n","   \u003c/td\u003e\n","   \u003ctd align=\"left\"\u003e\n","    \u003cspan class=\"flagicon\"\u003e\n","     \u003cimg alt=\"\" class=\"thumbborder\" data-file-height=\"600\" data-file-width=\"900\" decoding=\"async\" height=\"15\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/2/20/Flag_of_the_Netherlands.svg/23px-Flag_of_the_Netherlands.svg.png\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/2/20/Flag_of_the_Netherlands.svg/35px-Flag_of_the_Netherlands.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/2/20/Flag_of_the_Netherlands.svg/45px-Flag_of_the_Netherlands.svg.png 2x\" width=\"23\"/\u003e\n","    \u003c/span\u003e\n","    \u003ca href=\"/wiki/Netherlands\" title=\"Netherlands\"\u003e\n","     Netherlands\n","    \u003c/a\u003e\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    17,660,000\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    41,526\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    425\n","   \u003c/td\u003e\n","  \u003c/tr\u003e\n","  \u003ctr\u003e\n","   \u003ctd\u003e\n","    10\n","   \u003c/td\u003e\n","   \u003ctd align=\"left\"\u003e\n","    \u003cspan class=\"flagicon\"\u003e\n","     \u003cimg alt=\"\" class=\"thumbborder\" data-file-height=\"800\" data-file-width=\"1100\" decoding=\"async\" height=\"15\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/d/d4/Flag_of_Israel.svg/21px-Flag_of_Israel.svg.png\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/d/d4/Flag_of_Israel.svg/32px-Flag_of_Israel.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/d/d4/Flag_of_Israel.svg/41px-Flag_of_Israel.svg.png 2x\" width=\"21\"/\u003e\n","    \u003c/span\u003e\n","    \u003ca href=\"/wiki/Israel\" title=\"Israel\"\u003e\n","     Israel\n","    \u003c/a\u003e\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    9,430,000\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    22,072\n","   \u003c/td\u003e\n","   \u003ctd\u003e\n","    427\n","   \u003c/td\u003e\n","  \u003c/tr\u003e\n"," \u003c/tbody\u003e\n","\u003c/table\u003e\n","\n"]}],"source":["print(tables[table_index].prettify())"]},{"cell_type":"code","execution_count":58,"metadata":{},"outputs":[{"data":{"text/html":["\u003cdiv\u003e\n","\u003cstyle scoped\u003e\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","\u003c/style\u003e\n","\u003ctable border=\"1\" class=\"dataframe\"\u003e\n","  \u003cthead\u003e\n","    \u003ctr style=\"text-align: right;\"\u003e\n","      \u003cth\u003e\u003c/th\u003e\n","      \u003cth\u003eRank\u003c/th\u003e\n","      \u003cth\u003eCountry\u003c/th\u003e\n","      \u003cth\u003ePopulation\u003c/th\u003e\n","      \u003cth\u003eArea\u003c/th\u003e\n","      \u003cth\u003eDensity\u003c/th\u003e\n","    \u003c/tr\u003e\n","  \u003c/thead\u003e\n","  \u003ctbody\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e0\u003c/th\u003e\n","      \u003ctd\u003e1\u003c/td\u003e\n","      \u003ctd\u003eSingapore\u003c/td\u003e\n","      \u003ctd\u003e5,704,000\u003c/td\u003e\n","      \u003ctd\u003e710\u003c/td\u003e\n","      \u003ctd\u003e8,033\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e1\u003c/th\u003e\n","      \u003ctd\u003e2\u003c/td\u003e\n","      \u003ctd\u003eBangladesh\u003c/td\u003e\n","      \u003ctd\u003e171,670,000\u003c/td\u003e\n","      \u003ctd\u003e143,998\u003c/td\u003e\n","      \u003ctd\u003e1,192\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e2\u003c/th\u003e\n","      \u003ctd\u003e3\u003c/td\u003e\n","      \u003ctd\u003e\\n Palestine\\n\\n\u003c/td\u003e\n","      \u003ctd\u003e5,266,785\u003c/td\u003e\n","      \u003ctd\u003e6,020\u003c/td\u003e\n","      \u003ctd\u003e847\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e3\u003c/th\u003e\n","      \u003ctd\u003e4\u003c/td\u003e\n","      \u003ctd\u003eLebanon\u003c/td\u003e\n","      \u003ctd\u003e6,856,000\u003c/td\u003e\n","      \u003ctd\u003e10,452\u003c/td\u003e\n","      \u003ctd\u003e656\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e4\u003c/th\u003e\n","      \u003ctd\u003e5\u003c/td\u003e\n","      \u003ctd\u003eTaiwan\u003c/td\u003e\n","      \u003ctd\u003e23,604,000\u003c/td\u003e\n","      \u003ctd\u003e36,193\u003c/td\u003e\n","      \u003ctd\u003e652\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e5\u003c/th\u003e\n","      \u003ctd\u003e6\u003c/td\u003e\n","      \u003ctd\u003eSouth Korea\u003c/td\u003e\n","      \u003ctd\u003e51,781,000\u003c/td\u003e\n","      \u003ctd\u003e99,538\u003c/td\u003e\n","      \u003ctd\u003e520\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e6\u003c/th\u003e\n","      \u003ctd\u003e7\u003c/td\u003e\n","      \u003ctd\u003eRwanda\u003c/td\u003e\n","      \u003ctd\u003e12,374,000\u003c/td\u003e\n","      \u003ctd\u003e26,338\u003c/td\u003e\n","      \u003ctd\u003e470\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e7\u003c/th\u003e\n","      \u003ctd\u003e8\u003c/td\u003e\n","      \u003ctd\u003eHaiti\u003c/td\u003e\n","      \u003ctd\u003e11,578,000\u003c/td\u003e\n","      \u003ctd\u003e27,065\u003c/td\u003e\n","      \u003ctd\u003e428\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e8\u003c/th\u003e\n","      \u003ctd\u003e9\u003c/td\u003e\n","      \u003ctd\u003eNetherlands\u003c/td\u003e\n","      \u003ctd\u003e17,660,000\u003c/td\u003e\n","      \u003ctd\u003e41,526\u003c/td\u003e\n","      \u003ctd\u003e425\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e9\u003c/th\u003e\n","      \u003ctd\u003e10\u003c/td\u003e\n","      \u003ctd\u003eIsrael\u003c/td\u003e\n","      \u003ctd\u003e9,430,000\u003c/td\u003e\n","      \u003ctd\u003e22,072\u003c/td\u003e\n","      \u003ctd\u003e427\u003c/td\u003e\n","    \u003c/tr\u003e\n","  \u003c/tbody\u003e\n","\u003c/table\u003e\n","\u003c/div\u003e"],"text/plain":["  Rank           Country   Population     Area Density\n","0    1         Singapore    5,704,000      710   8,033\n","1    2        Bangladesh  171,670,000  143,998   1,192\n","2    3  \\n Palestine\\n\\n    5,266,785    6,020     847\n","3    4           Lebanon    6,856,000   10,452     656\n","4    5            Taiwan   23,604,000   36,193     652\n","5    6       South Korea   51,781,000   99,538     520\n","6    7            Rwanda   12,374,000   26,338     470\n","7    8             Haiti   11,578,000   27,065     428\n","8    9       Netherlands   17,660,000   41,526     425\n","9   10            Israel    9,430,000   22,072     427"]},"execution_count":58,"metadata":{},"output_type":"execute_result"}],"source":["population_data = pd.DataFrame(columns=[\"Rank\", \"Country\", \"Population\", \"Area\", \"Density\"])\n","\n","for row in tables[table_index].tbody.find_all(\"tr\"):\n","    col = row.find_all(\"td\")\n","    if (col != []):\n","        rank = col[0].text\n","        country = col[1].text\n","        population = col[2].text.strip()\n","        area = col[3].text.strip()\n","        density = col[4].text.strip()\n","        population_data = population_data.append({\"Rank\":rank, \"Country\":country, \"Population\":population, \"Area\":area, \"Density\":density}, ignore_index=True)\n","\n","population_data"]},{"cell_type":"markdown","metadata":{},"source":["## Scrape data from HTML tables into a DataFrame using BeautifulSoup and read_html\n"]},{"cell_type":"markdown","metadata":{},"source":["Using the same `url`, `data`, `soup`, and `tables` object as in the last section we can use the `read_html` function to create a DataFrame.\n","\n","Remember the table we need is located in `tables[table_index]`\n","\n","We can now use the `pandas` function `read_html` and give it the string version of the table as well as the `flavor` which is the parsing engine `bs4`.\n"]},{"cell_type":"code","execution_count":59,"metadata":{},"outputs":[{"data":{"text/plain":["[   Rank      Country  Population  Area(km2)  Density(pop/km2)\n"," 0     1    Singapore     5704000        710              8033\n"," 1     2   Bangladesh   171670000     143998              1192\n"," 2     3    Palestine     5266785       6020               847\n"," 3     4      Lebanon     6856000      10452               656\n"," 4     5       Taiwan    23604000      36193               652\n"," 5     6  South Korea    51781000      99538               520\n"," 6     7       Rwanda    12374000      26338               470\n"," 7     8        Haiti    11578000      27065               428\n"," 8     9  Netherlands    17660000      41526               425\n"," 9    10       Israel     9430000      22072               427]"]},"execution_count":59,"metadata":{},"output_type":"execute_result"}],"source":["pd.read_html(str(tables[5]), flavor='bs4')"]},{"cell_type":"markdown","metadata":{},"source":["The function `read_html` always returns a list of DataFrames so we must pick the one we want out of the list.\n"]},{"cell_type":"code","execution_count":60,"metadata":{},"outputs":[{"data":{"text/html":["\u003cdiv\u003e\n","\u003cstyle scoped\u003e\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","\u003c/style\u003e\n","\u003ctable border=\"1\" class=\"dataframe\"\u003e\n","  \u003cthead\u003e\n","    \u003ctr style=\"text-align: right;\"\u003e\n","      \u003cth\u003e\u003c/th\u003e\n","      \u003cth\u003eRank\u003c/th\u003e\n","      \u003cth\u003eCountry\u003c/th\u003e\n","      \u003cth\u003ePopulation\u003c/th\u003e\n","      \u003cth\u003eArea(km2)\u003c/th\u003e\n","      \u003cth\u003eDensity(pop/km2)\u003c/th\u003e\n","    \u003c/tr\u003e\n","  \u003c/thead\u003e\n","  \u003ctbody\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e0\u003c/th\u003e\n","      \u003ctd\u003e1\u003c/td\u003e\n","      \u003ctd\u003eSingapore\u003c/td\u003e\n","      \u003ctd\u003e5704000\u003c/td\u003e\n","      \u003ctd\u003e710\u003c/td\u003e\n","      \u003ctd\u003e8033\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e1\u003c/th\u003e\n","      \u003ctd\u003e2\u003c/td\u003e\n","      \u003ctd\u003eBangladesh\u003c/td\u003e\n","      \u003ctd\u003e171670000\u003c/td\u003e\n","      \u003ctd\u003e143998\u003c/td\u003e\n","      \u003ctd\u003e1192\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e2\u003c/th\u003e\n","      \u003ctd\u003e3\u003c/td\u003e\n","      \u003ctd\u003ePalestine\u003c/td\u003e\n","      \u003ctd\u003e5266785\u003c/td\u003e\n","      \u003ctd\u003e6020\u003c/td\u003e\n","      \u003ctd\u003e847\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e3\u003c/th\u003e\n","      \u003ctd\u003e4\u003c/td\u003e\n","      \u003ctd\u003eLebanon\u003c/td\u003e\n","      \u003ctd\u003e6856000\u003c/td\u003e\n","      \u003ctd\u003e10452\u003c/td\u003e\n","      \u003ctd\u003e656\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e4\u003c/th\u003e\n","      \u003ctd\u003e5\u003c/td\u003e\n","      \u003ctd\u003eTaiwan\u003c/td\u003e\n","      \u003ctd\u003e23604000\u003c/td\u003e\n","      \u003ctd\u003e36193\u003c/td\u003e\n","      \u003ctd\u003e652\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e5\u003c/th\u003e\n","      \u003ctd\u003e6\u003c/td\u003e\n","      \u003ctd\u003eSouth Korea\u003c/td\u003e\n","      \u003ctd\u003e51781000\u003c/td\u003e\n","      \u003ctd\u003e99538\u003c/td\u003e\n","      \u003ctd\u003e520\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e6\u003c/th\u003e\n","      \u003ctd\u003e7\u003c/td\u003e\n","      \u003ctd\u003eRwanda\u003c/td\u003e\n","      \u003ctd\u003e12374000\u003c/td\u003e\n","      \u003ctd\u003e26338\u003c/td\u003e\n","      \u003ctd\u003e470\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e7\u003c/th\u003e\n","      \u003ctd\u003e8\u003c/td\u003e\n","      \u003ctd\u003eHaiti\u003c/td\u003e\n","      \u003ctd\u003e11578000\u003c/td\u003e\n","      \u003ctd\u003e27065\u003c/td\u003e\n","      \u003ctd\u003e428\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e8\u003c/th\u003e\n","      \u003ctd\u003e9\u003c/td\u003e\n","      \u003ctd\u003eNetherlands\u003c/td\u003e\n","      \u003ctd\u003e17660000\u003c/td\u003e\n","      \u003ctd\u003e41526\u003c/td\u003e\n","      \u003ctd\u003e425\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e9\u003c/th\u003e\n","      \u003ctd\u003e10\u003c/td\u003e\n","      \u003ctd\u003eIsrael\u003c/td\u003e\n","      \u003ctd\u003e9430000\u003c/td\u003e\n","      \u003ctd\u003e22072\u003c/td\u003e\n","      \u003ctd\u003e427\u003c/td\u003e\n","    \u003c/tr\u003e\n","  \u003c/tbody\u003e\n","\u003c/table\u003e\n","\u003c/div\u003e"],"text/plain":["   Rank      Country  Population  Area(km2)  Density(pop/km2)\n","0     1    Singapore     5704000        710              8033\n","1     2   Bangladesh   171670000     143998              1192\n","2     3    Palestine     5266785       6020               847\n","3     4      Lebanon     6856000      10452               656\n","4     5       Taiwan    23604000      36193               652\n","5     6  South Korea    51781000      99538               520\n","6     7       Rwanda    12374000      26338               470\n","7     8        Haiti    11578000      27065               428\n","8     9  Netherlands    17660000      41526               425\n","9    10       Israel     9430000      22072               427"]},"execution_count":60,"metadata":{},"output_type":"execute_result"}],"source":["population_data_read_html = pd.read_html(str(tables[5]), flavor='bs4')[0]\n","\n","population_data_read_html"]},{"cell_type":"markdown","metadata":{},"source":["## Scrape data from HTML tables into a DataFrame using read_html\n"]},{"cell_type":"markdown","metadata":{},"source":["We can also use the `read_html` function to directly get DataFrames from a `url`.\n"]},{"cell_type":"code","execution_count":61,"metadata":{},"outputs":[],"source":["dataframe_list = pd.read_html(url, flavor='bs4')"]},{"cell_type":"markdown","metadata":{},"source":["We can see there are 25 DataFrames just like when we used `find_all` on the `soup` object.\n"]},{"cell_type":"code","execution_count":62,"metadata":{},"outputs":[{"data":{"text/plain":["26"]},"execution_count":62,"metadata":{},"output_type":"execute_result"}],"source":["len(dataframe_list)"]},{"cell_type":"markdown","metadata":{},"source":["Finally we can pick the DataFrame we need out of the list.\n"]},{"cell_type":"code","execution_count":63,"metadata":{},"outputs":[{"data":{"text/html":["\u003cdiv\u003e\n","\u003cstyle scoped\u003e\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","\u003c/style\u003e\n","\u003ctable border=\"1\" class=\"dataframe\"\u003e\n","  \u003cthead\u003e\n","    \u003ctr style=\"text-align: right;\"\u003e\n","      \u003cth\u003e\u003c/th\u003e\n","      \u003cth\u003eRank\u003c/th\u003e\n","      \u003cth\u003eCountry\u003c/th\u003e\n","      \u003cth\u003ePopulation\u003c/th\u003e\n","      \u003cth\u003eArea(km2)\u003c/th\u003e\n","      \u003cth\u003eDensity(pop/km2)\u003c/th\u003e\n","    \u003c/tr\u003e\n","  \u003c/thead\u003e\n","  \u003ctbody\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e0\u003c/th\u003e\n","      \u003ctd\u003e1\u003c/td\u003e\n","      \u003ctd\u003eSingapore\u003c/td\u003e\n","      \u003ctd\u003e5704000\u003c/td\u003e\n","      \u003ctd\u003e710\u003c/td\u003e\n","      \u003ctd\u003e8033\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e1\u003c/th\u003e\n","      \u003ctd\u003e2\u003c/td\u003e\n","      \u003ctd\u003eBangladesh\u003c/td\u003e\n","      \u003ctd\u003e171670000\u003c/td\u003e\n","      \u003ctd\u003e143998\u003c/td\u003e\n","      \u003ctd\u003e1192\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e2\u003c/th\u003e\n","      \u003ctd\u003e3\u003c/td\u003e\n","      \u003ctd\u003ePalestine\u003c/td\u003e\n","      \u003ctd\u003e5266785\u003c/td\u003e\n","      \u003ctd\u003e6020\u003c/td\u003e\n","      \u003ctd\u003e847\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e3\u003c/th\u003e\n","      \u003ctd\u003e4\u003c/td\u003e\n","      \u003ctd\u003eLebanon\u003c/td\u003e\n","      \u003ctd\u003e6856000\u003c/td\u003e\n","      \u003ctd\u003e10452\u003c/td\u003e\n","      \u003ctd\u003e656\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e4\u003c/th\u003e\n","      \u003ctd\u003e5\u003c/td\u003e\n","      \u003ctd\u003eTaiwan\u003c/td\u003e\n","      \u003ctd\u003e23604000\u003c/td\u003e\n","      \u003ctd\u003e36193\u003c/td\u003e\n","      \u003ctd\u003e652\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e5\u003c/th\u003e\n","      \u003ctd\u003e6\u003c/td\u003e\n","      \u003ctd\u003eSouth Korea\u003c/td\u003e\n","      \u003ctd\u003e51781000\u003c/td\u003e\n","      \u003ctd\u003e99538\u003c/td\u003e\n","      \u003ctd\u003e520\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e6\u003c/th\u003e\n","      \u003ctd\u003e7\u003c/td\u003e\n","      \u003ctd\u003eRwanda\u003c/td\u003e\n","      \u003ctd\u003e12374000\u003c/td\u003e\n","      \u003ctd\u003e26338\u003c/td\u003e\n","      \u003ctd\u003e470\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e7\u003c/th\u003e\n","      \u003ctd\u003e8\u003c/td\u003e\n","      \u003ctd\u003eHaiti\u003c/td\u003e\n","      \u003ctd\u003e11578000\u003c/td\u003e\n","      \u003ctd\u003e27065\u003c/td\u003e\n","      \u003ctd\u003e428\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e8\u003c/th\u003e\n","      \u003ctd\u003e9\u003c/td\u003e\n","      \u003ctd\u003eNetherlands\u003c/td\u003e\n","      \u003ctd\u003e17660000\u003c/td\u003e\n","      \u003ctd\u003e41526\u003c/td\u003e\n","      \u003ctd\u003e425\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e9\u003c/th\u003e\n","      \u003ctd\u003e10\u003c/td\u003e\n","      \u003ctd\u003eIsrael\u003c/td\u003e\n","      \u003ctd\u003e9430000\u003c/td\u003e\n","      \u003ctd\u003e22072\u003c/td\u003e\n","      \u003ctd\u003e427\u003c/td\u003e\n","    \u003c/tr\u003e\n","  \u003c/tbody\u003e\n","\u003c/table\u003e\n","\u003c/div\u003e"],"text/plain":["   Rank      Country  Population  Area(km2)  Density(pop/km2)\n","0     1    Singapore     5704000        710              8033\n","1     2   Bangladesh   171670000     143998              1192\n","2     3    Palestine     5266785       6020               847\n","3     4      Lebanon     6856000      10452               656\n","4     5       Taiwan    23604000      36193               652\n","5     6  South Korea    51781000      99538               520\n","6     7       Rwanda    12374000      26338               470\n","7     8        Haiti    11578000      27065               428\n","8     9  Netherlands    17660000      41526               425\n","9    10       Israel     9430000      22072               427"]},"execution_count":63,"metadata":{},"output_type":"execute_result"}],"source":["dataframe_list[5]"]},{"cell_type":"markdown","metadata":{},"source":["We can also use the `match` parameter to select the specific table we want. If the table contains a string matching the text it will be read.\n"]},{"cell_type":"code","execution_count":64,"metadata":{},"outputs":[{"data":{"text/html":["\u003cdiv\u003e\n","\u003cstyle scoped\u003e\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","\u003c/style\u003e\n","\u003ctable border=\"1\" class=\"dataframe\"\u003e\n","  \u003cthead\u003e\n","    \u003ctr style=\"text-align: right;\"\u003e\n","      \u003cth\u003e\u003c/th\u003e\n","      \u003cth\u003eRank\u003c/th\u003e\n","      \u003cth\u003eCountry\u003c/th\u003e\n","      \u003cth\u003ePopulation\u003c/th\u003e\n","      \u003cth\u003eArea(km2)\u003c/th\u003e\n","      \u003cth\u003eDensity(pop/km2)\u003c/th\u003e\n","    \u003c/tr\u003e\n","  \u003c/thead\u003e\n","  \u003ctbody\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e0\u003c/th\u003e\n","      \u003ctd\u003e1\u003c/td\u003e\n","      \u003ctd\u003eSingapore\u003c/td\u003e\n","      \u003ctd\u003e5704000\u003c/td\u003e\n","      \u003ctd\u003e710\u003c/td\u003e\n","      \u003ctd\u003e8033\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e1\u003c/th\u003e\n","      \u003ctd\u003e2\u003c/td\u003e\n","      \u003ctd\u003eBangladesh\u003c/td\u003e\n","      \u003ctd\u003e171670000\u003c/td\u003e\n","      \u003ctd\u003e143998\u003c/td\u003e\n","      \u003ctd\u003e1192\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e2\u003c/th\u003e\n","      \u003ctd\u003e3\u003c/td\u003e\n","      \u003ctd\u003ePalestine\u003c/td\u003e\n","      \u003ctd\u003e5266785\u003c/td\u003e\n","      \u003ctd\u003e6020\u003c/td\u003e\n","      \u003ctd\u003e847\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e3\u003c/th\u003e\n","      \u003ctd\u003e4\u003c/td\u003e\n","      \u003ctd\u003eLebanon\u003c/td\u003e\n","      \u003ctd\u003e6856000\u003c/td\u003e\n","      \u003ctd\u003e10452\u003c/td\u003e\n","      \u003ctd\u003e656\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e4\u003c/th\u003e\n","      \u003ctd\u003e5\u003c/td\u003e\n","      \u003ctd\u003eTaiwan\u003c/td\u003e\n","      \u003ctd\u003e23604000\u003c/td\u003e\n","      \u003ctd\u003e36193\u003c/td\u003e\n","      \u003ctd\u003e652\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e5\u003c/th\u003e\n","      \u003ctd\u003e6\u003c/td\u003e\n","      \u003ctd\u003eSouth Korea\u003c/td\u003e\n","      \u003ctd\u003e51781000\u003c/td\u003e\n","      \u003ctd\u003e99538\u003c/td\u003e\n","      \u003ctd\u003e520\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e6\u003c/th\u003e\n","      \u003ctd\u003e7\u003c/td\u003e\n","      \u003ctd\u003eRwanda\u003c/td\u003e\n","      \u003ctd\u003e12374000\u003c/td\u003e\n","      \u003ctd\u003e26338\u003c/td\u003e\n","      \u003ctd\u003e470\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e7\u003c/th\u003e\n","      \u003ctd\u003e8\u003c/td\u003e\n","      \u003ctd\u003eHaiti\u003c/td\u003e\n","      \u003ctd\u003e11578000\u003c/td\u003e\n","      \u003ctd\u003e27065\u003c/td\u003e\n","      \u003ctd\u003e428\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e8\u003c/th\u003e\n","      \u003ctd\u003e9\u003c/td\u003e\n","      \u003ctd\u003eNetherlands\u003c/td\u003e\n","      \u003ctd\u003e17660000\u003c/td\u003e\n","      \u003ctd\u003e41526\u003c/td\u003e\n","      \u003ctd\u003e425\u003c/td\u003e\n","    \u003c/tr\u003e\n","    \u003ctr\u003e\n","      \u003cth\u003e9\u003c/th\u003e\n","      \u003ctd\u003e10\u003c/td\u003e\n","      \u003ctd\u003eIsrael\u003c/td\u003e\n","      \u003ctd\u003e9430000\u003c/td\u003e\n","      \u003ctd\u003e22072\u003c/td\u003e\n","      \u003ctd\u003e427\u003c/td\u003e\n","    \u003c/tr\u003e\n","  \u003c/tbody\u003e\n","\u003c/table\u003e\n","\u003c/div\u003e"],"text/plain":["   Rank      Country  Population  Area(km2)  Density(pop/km2)\n","0     1    Singapore     5704000        710              8033\n","1     2   Bangladesh   171670000     143998              1192\n","2     3    Palestine     5266785       6020               847\n","3     4      Lebanon     6856000      10452               656\n","4     5       Taiwan    23604000      36193               652\n","5     6  South Korea    51781000      99538               520\n","6     7       Rwanda    12374000      26338               470\n","7     8        Haiti    11578000      27065               428\n","8     9  Netherlands    17660000      41526               425\n","9    10       Israel     9430000      22072               427"]},"execution_count":64,"metadata":{},"output_type":"execute_result"}],"source":["pd.read_html(url, match=\"10 most densely populated countries\", flavor='bs4')[0]"]},{"cell_type":"markdown","metadata":{},"source":["## Authors\n"]},{"cell_type":"markdown","metadata":{},"source":["Ramesh Sannareddy\n"]},{"cell_type":"markdown","metadata":{},"source":["### Other Contributors\n"]},{"cell_type":"markdown","metadata":{},"source":["Rav Ahuja\n"]},{"cell_type":"markdown","metadata":{},"source":["## Change Log\n"]},{"cell_type":"markdown","metadata":{},"source":["| Date (YYYY-MM-DD) | Version | Changed By                                               | Change Description |\n","| ----------------- | ------- | -------------------------------------------------------- | ------------------ |\n","| 2021-08-04        | 0.2     | Made changes to markdown of nextsibling                  |                    |\n","| 2020-10-17        | 0.1     | Joseph Santarcangelo  Created initial version of the lab |                    |\n"]},{"cell_type":"markdown","metadata":{},"source":["Copyright © 2020 IBM Corporation. This notebook and its source code are released under the terms of the [MIT License](https://cognitiveclass.ai/mit-license?utm_medium=Exinfluencer\u0026utm_source=Exinfluencer\u0026utm_content=000026UJ\u0026utm_term=10006555\u0026utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0220ENSkillsNetwork23455606-2021-01-01\u0026cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBMDeveloperSkillsNetwork-PY0101EN-SkillsNetwork-19487395\u0026cm_mmca1=000026UJ\u0026cm_mmca2=10006555\u0026cm_mmca3=M12345678\u0026cvosrc=email.Newsletter.M12345678\u0026cvo_campaign=000026UJ).\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":[]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":[]}],"metadata":{"kernelspec":{"display_name":"Python","language":"python","name":"conda-env-python-py"},"language_info":{"name":""}},"nbformat":4,"nbformat_minor":4}