Summary of an article published on The Web Scraping Club on 2022-12-18 (free to read). Read the full article
ChatGPT came out a few weeks ago and the feeling is of talking with an omniscient human. It writes articles, translates, answers questions and produces code. Reading that list, I could think of at least ten professions that should worry: SEO content writers, translators, stock image creators. The question for this newsletter is whether programmers, and web scrapers in particular, should worry too.
I’m not an AI expert, so I let ChatGPT explain GPT models and Dall-E in its own words. Then we tested it on our field. I asked for Scrapy spiders for Amazon.com, first with no constraints, then with XPath selectors and a defined set of fields. I also asked for a spider for thewebscraping.club, a site that did not exist in the 2021 training data. We looked at whether the code is formally correct and whether the selectors match the real HTML. Then I add my view on copyright, Zyte’s Automatic Extraction and anti-bot work.
What’s inside
- How ChatGPT differs from GPT-3, as ChatGPT itself explains it, and why I think its training data was text and not HTML.
- Which fields we asked the Amazon spider to extract: ASIN, title, prices, product and image URLs.
- Why asking for a spider for a site born after 2021 works as a test of where its selectors come from.
- Copyright concerns, from GitHub Copilot and code licenses such as MIT, GPL and Apache to AI images in the style of Banksy or Van Gogh.
In the full article: The screenshots of ChatGPT’s generated spiders for Amazon and thewebscraping.club, its full answers on GPT and Dall-E, and my closing view on why coding is becoming marginal in scraping projects.