Parsing HTML using Python

Parsing HTML is one of the most common task done today to collect information from the websites and mine it for various purposes, like to establish price performance of a product over time, reviews of a book on a website and much more. There exist many libraries like BeautifulSoup in Python which abstracts away so many painful points in parsing HTML but it is worth knowing how those libraries actually work beneath that layer of abstraction.

In this lesson, that is what we intend to do. We will find out how values of different HTML tags can be extracted and also override the default functionality of this module to add some logic of our own. We will do this using the HTMLParser class in Python in html.parser module. Let’s see the code in action.

Looking at HTMLParser class

To parse HTML text in Python, we can make use of HTMLParser class in html.parser module. Let’s look at the class dfinition for the HTMLParser class:

The convert_charrefs field, if set to True will make all the character references converted to their Unicode equivalents. Only the script/style elements aren’t converted. Now, we will try to understand each function for this class as well to better understand what each function does.

• handle_startendtag This is the first function which is triggered when HTML string is passed to the class instance. Once the text reaches here, the control is passed to other functions in the class which narrows down to other tags in the String. This is also clear in the definition for this function:
  def handle_startendtag(self, tag, attrs):
  self.handle_starttag(tag, attrs)
  self.handle_endtag(tag)
• handle_starttag: This method manages the start tag for the data it receives. Its definition is as shown below:
  def handle_starttag(self, tag, attrs):
  pass
• handle_endtag: This method manages the end tag for the data it receives:
  def handle_endtag(self, tag):
  pass
• handle_charref: This method manages the character references in the data it receives. Its definition is as shown below:
  def handle_charref(self, name):
  pass
• handle_entityref: This function handles the entity references in the HTML passed to it:
  def handle_entityref(self, name):
  pass
• handle_data:This is the function where real work is done to extract values from the HTML tags and is passed the data related to each tag. Its definition is as shown below:
  def handle_data(self, data):
  pass
• handle_comment: Using this function, we can also get comments attached to an HTML source:
  def handle_comment(self, data):
  pass
• handle_pi: As HTML can also have processing instructions, this is the function where these Its definition is as shown below:
  def handle_pi(self, data):
  pass
• handle_decl: This method handles the declarations in the HTML, its definition is provided as:
  def handle_decl(self, decl):
  pass

Subclassing the HTMLParser class

In this section, we will sub-class the HTMLParser class and will take a look at some of the functions being called when HTML data is passed to class instance. Let’s write a simple script which do all of this:

Here is what we get back with this command:

HTMLParser functions

In this section, we will work with various functions of the HTMLParser class and look at functionality of each of those:

With various calls, let us feed separate HTML data to this instance and see what output these calls generate. We will start with a simple DOCTYPE string:

Here is what we get back with this call:

Let us now try an image tag and see what data it extracts:

Here is what we get back with this call:

Next, let’s try how script tag behaves with Python functions:

Here is what we get back with this call:

Finally, we pass comments to the HTMLParser section as well:

Here is what we get back with this call:

Conclusion

In this lesson, we looked at how we can parse HTML using Python own HTMLParser class without any other library. We can easily modify the code to change the source of the HTML data to an HTTP client.

Read more Python based posts here.