What it is
wxpath is a Python library for declarative web crawling using XPath. It allows you to describe traversal and extraction in a single expression, making it easier to perform recursive or paginated web crawling and extraction without writing imperative crawl loops.
How it works
wxpath introduces the url(...) operator and the /// syntax to enable recursive (or paginated) web crawling and extraction. The library executes the expression concurrently, breadth-first-ish, and streams results as they are discovered. For example, you can fetch a page, extract links, and stream them concurrently with a single expression.
import wxpath
expr = "url('https://quotes.toscrape.com')//a/@href"
for link in wxpath.wxpath_async_blocking_iter(expr):
print(link)By using ///url(...) syntax, wxpath can perform deep crawling up to a specified max_depth:
import wxpath
path_expr = """
url('https://quotes.toscrape.com')
///url(//a/@href)
//a/@href
"""
for item in wxpath.wxpath_async_blocking_iter(path_expr, max_depth=1):
print(item)TWSC experience
Not yet tested by TWSC.
Related
Sources
The Web Scraping Wiki is compiled from the tests and research published on The Web Scraping Club, the newsletter for web scraping professionals. Subscribe to get the next article in your inbox.