Web scraping with Python for fun and profit

Web is everywhere, we know. It is also used more and more to present information to a wide audience. Sadly, it is commonly the only way data is presented...

That said, we need to get that info; the process of extracting information from web pages is knows as web scraping, and note that's is a very fragile process: every time the webpage changes, it's likely you'll have to modify the code that parses it.

The probably most famous Python module to do web scraping is BeautifulSoup. While it might be nice for simple webpages, I found it really hard to get something done for more complex pages, in particular with those with JavaScript embedded.

Thanks to Ian blogpost, I discovered how nice is to use lxml to do web scraping, in particular in association with Firebug Firefox addon: it's just a simple process of:

1. take the page;
2. generate the lxml tree;
3. with Firebug find the XPath to the element you need;
4. loop / parse / have fun :)
   

If you find in need to web scrape a page, give lxml a try: you'll be surprised and satisfied!