Summary of an article published on The Web Scraping Club on 2024-07-23 (free to read). Read the full article

Market analysts scrape e-commerce sites for market research, trend analysis and pricing new products. The practical issue is that every platform has its own page types, and keeping parsers and anti-bot handling alive across them takes time. A managed API that returns parsed data cuts a lot of that work.

We built two small scrapers in Python with Oxylabs’ E-Commerce Scraper API, a partner of the newsletter. The first takes a search query on Amazon and returns the list of products. The second reads a single product page on AliExpress through the universal_ecommerce source. We use Requests to call the API and pandas to hold and export the results. I cover how Amazon is handled through dedicated data sources while other shops need only a product URL, which parameters are mandatory in the payload, and which fields we pull from the parsed response.

What’s inside

  • Amazon is supported through dedicated source keys, one per page type, while other shops go through a single universal source.
  • Only the source and query parameters are mandatory in the Amazon search payload. Domain, start page, number of pages and the parse flag are optional.
  • The Amazon scraper extracts title, ASIN, price and URL into a DataFrame, then exports to CSV and JSON.
  • The AliExpress example reads title, price with currency, and description from the parsed product page.

In the full article: The full payload parameter table, the complete Python code for both scrapers, the list of Amazon data sources and the output screenshots for each run.

Read the full article on The Web Scraping Club