What it is
Tadpole is a language designed for writing declarative, modular scraping code. It abstracts away the complexity of interacting with the browser, making it easier to build scrapers.
How it works
Tadpole allows you to import modules from local files or remote repositories, enabling composable scraping code. The language simplifies the process of building scrapers by abstracting the browser interaction.
Example
tadpole run redfin.kdl --input '{"text": "Seattle, WA"}' --auto --headless --output output.jsonimport "modules/redfin/mod.kdl" repo="github.com/tadpolehq/community"
main {
new_page {
redfin.search text="=text"
wait_until
redfin.extract_from_card extract_to="addresses" {
address {
redfin.extract_address_from_card
}
}
}
}
{
"addresses": [
{
"address": "2011 E James St, Seattle, WA 98122"
},
{
"address": "8020 17th Ave NW, Seattle, WA 98117"
},
{
"address": "4015 SW Donovan St, Seattle, WA 98136"
},
{
"address": "116 13th Ave, Seattle, WA 98122"
}
]
}TWSC Experience
Not yet tested by TWSC.
Related
Sources
The Web Scraping Wiki is compiled from the tests and research published on The Web Scraping Club, the newsletter for web scraping professionals. Subscribe to get the next article in your inbox.