Summary of an article published on The Web Scraping Club on 2023-04-23 (free to read). Read the full article

Most of what we cover here is how to get the data. We rarely look at who buys it. The financial sector is one of the hungriest buyers, because a new and reliable dataset can be worth millions. At Re Analytics we work in this industry, so I wanted to start with the basics of alternative data: information about a company published by sources outside the company, from satellite images to credit card transactions to web data.

I explain where web scraping sits among alternative data providers, with examples like review sentiment and e-commerce pricing. Then I go through the compliance side, which is the real barrier for anyone who wants to sell to funds. We look at the four risk areas listed in a Zyte piece and at the Investment Data Standards Organization checklist for web collection. I also compare what fundamental funds and quant funds expect from a dataset, and what this means for freelancers versus established companies.

What’s inside

  • How satellite images of Walmart parking lots were used to estimate quarterly revenues before the public figures came out.
  • The split between providers who rework other people’s data and providers who extract it from public sources, with review sentiment and e-commerce discounting as examples.
  • The four risk categories funds check: exclusivity and insider trading, privacy, copyright, and data acquisition, and why logins and paywalls raise red flags.
  • Why fundamental funds want a ticker-specific dataset, while quants want years of history across many stocks.

In the full article: The full list of IDSO checklist points quoted, the Zyte reference on proving proper collection, the slide of alternative data categories and my closing view on how hard it is for a freelancer to enter this market.

Read the full article on The Web Scraping Club