Summary of an article published on The Web Scraping Club on 2024-08-06 (free to read). Read the full article

Idealista is one of the largest real estate marketplaces in Southern Europe, and its listings feed market research, lead generation and competitor analysis. It is also protected by DataDome, which scores every visitor on a trust score built from browser fingerprint, behavior, IP reputation, HTTP headers and CAPTCHA challenges. Automated browsers like Selenium, Puppeteer and Playwright carry fingerprints that this scoring picks up easily.

We wrote this one with our partner ScraperAPI. We start with a plain headless Selenium script on a listing page for Alcala de Henares, Madrid, and extract title, price, bedrooms, area, description and image URL with BeautifulSoup. Then we look at what DataDome returns when it flags the script. We rebuild the same scraper on ScraperAPI with rendering enabled, so proxy rotation, CAPTCHA handling and anti-bot evasion are handled by the service, and we save the results to JSON.

What’s inside

  • How DataDome’s trust score decides between allowing a visitor, blocking it or serving a CAPTCHA
  • Why default headless Chrome and mechanical request timing give the script away
  • What the DataDome block page looks like in the raw response when the Selenium script is caught
  • Why the ScraperAPI request needs JavaScript rendering for the listing data to be present

In the full article: The full Selenium and ScraperAPI scripts, the CSS selectors for each listing field, the DataDome block response and the sample JSON output of the listings.

Read the full article on The Web Scraping Club