What it is

wxpath is a Python library for declarative web crawling using XPath. It allows you to describe traversal and extraction in a single expression, making it easier to perform recursive or paginated web crawling and extraction without writing imperative crawl loops.

How it works

wxpath introduces the url(...) operator and the /// syntax to enable recursive (or paginated) web crawling and extraction. The library executes the expression concurrently, breadth-first-ish, and streams results as they are discovered. For example, you can fetch a page, extract links, and stream them concurrently with a single expression.

import wxpath
 
expr = "url('https://quotes.toscrape.com')//a/@href"
 
for link in wxpath.wxpath_async_blocking_iter(expr):
    print(link)

By using ///url(...) syntax, wxpath can perform deep crawling up to a specified max_depth:

import wxpath
 
path_expr = """
url('https://quotes.toscrape.com')
  ///url(//a/@href)
    //a/@href
"""
 
for item in wxpath.wxpath_async_blocking_iter(path_expr, max_depth=1):
    print(item)

TWSC experience

Not yet tested by TWSC.

Sources


The Web Scraping Wiki is compiled from the tests and research published on The Web Scraping Club, the newsletter for web scraping professionals. Subscribe to get the next article in your inbox.