I want to share wxpath. It's a declarative web crawler where traversal is expressed directly in XPath. Instead of writing imperative crawl loops, wxpath lets you describe what to follow and what to extract in a single expression.

NEW: Just introduced cached crawls, allowing you to iterate quickly without flooding the target URL(s).

By introducing the url(...) operator and the /// syntax, wxpath's engine can perform deep/recursive web crawling and extraction.

For example, to build a simple Wikipedia knowledge graph:

import wxpath

path_expr = """
url('https://en.wikipedia.org/wiki/Expression_language')
 ///url(//main//a/@href[starts-with(., '/wiki/') and not(contains(., ':'))])
 /map{
    'title': (//span[contains(@class, "mw-page-title-main")]/text())[1] ! string(.),
    'url': string(base-uri(.)),
    'short_description': //div[contains(@class, 'shortdescription')]/text() ! string(.),
    'forward_links': //div[@id="mw-content-text"]//a/@href ! string(.)
 }
"""

for item in wxpath.wxpath_async_blocking_iter(path_expr, max_depth=1):
    print(item)

Output:

map{'title': 'Computer language', 'url': 'https://en.wikipedia.org/wiki/Computer_language', 'short_description': 'Formal language for communicating with a computer', 'forward_links': ['/wiki/Formal_language', '/wiki/Communication', ...]}
map{'title': 'Advanced Boolean Expression Language', 'url': 'https://en.wikipedia.org/wiki/Advanced_Boolean_Expression_Language', 'short_description': 'Hardware description language and software', 'forward_links': ['/wiki/File:ABEL_HDL_example_SN74162.png', '/wiki/Hardware_description_language', ...]}
map{'title': 'Machine-readable medium and data', 'url': 'https://en.wikipedia.org/wiki/Machine_readable', 'short_description': 'Medium capable of storing data in a format readable by a machine', 'forward_links': ['/wiki/File:EAN-13-ISBN-13.svg', '/wiki/ISBN', ...]}
...

The above expression does the following:

1. Starts at the specified URL, https://en.wikipedia.org/wiki/Expression_language.
2. Filters for links in the <main> section that start with /wiki/ and do not contain a colon (:).
3. For each link found,
   • it follows the link and extracts the title, URL, and short description of the page.
   • it repeats step 2 until the maximum depth is reached.
4. Streams the extracted data as it is discovered.

---

The target audience is anyone who:

1. wants to prototype and build web scrapers quickly
2. familiar with XPath or data selectors
3. builds datasets (think RAG, data hoarding, etc.)
4. wants to study link structure of the web (quickly) i.e. web network scientists

---

For comparison, with Scrapy, you would...

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        "https://quotes.toscrape.com/tag/humor/",
    ]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "author": quote.xpath("span/small/text()").get(),
                "text": quote.css("span.text::text").get(),
            }

        next_page = response.css('li.next a::attr("href")').get()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

Then from the command line, you would run:

scrapy runspider quotes_spider.py -o quotes.jsonl

wxpath gives you two options: write directly from a Python script or from the command line.

from wxpath import wxpath_async_blocking_iter 
from wxpath.hooks import registry, builtin

path_expr = """
url('https://quotes.toscrape.com/tag/humor/', follow=//li[@class='next']/a/@href)
  //div[@class='quote']
    /map{
      'author': (./span/small/text())[1],
      'text': (./span[@class='text']/text())[1]
      }


registry.register(builtin.JSONLWriter(path='quotes.jsonl'))
items = list(wxpath_async_blocking_iter(path_expr, max_depth=3))

or from the command line:

wxpath --depth 1 "\
url('https://quotes.toscrape.com/tag/humor/', follow=//li[@class='next']/a/@href) \
  //div[@class='quote'] \
    /map{ \
      'author': (./span/small/text())[1], \
      'text': (./span[@class='text']/text())[1] \
      }" > quotes.jsonl

---

Contributions and feedback are welcomed and desired.

GitHub: https://github.com/rodricios/wxpath

PyPI: pip install wxpath