Summary of an article published on The Web Scraping Club on 2023-07-16 (free to read). Read the full article
TikTok has more than 1 billion monthly users and exposes metrics such as views, followers and engagement rate. That makes it a target for hashtag tracking, engagement analysis, content extraction, follower growth monitoring and trend prediction. The catch is that the first excitement fades quickly: rate limits, geo-blocks and the risk of getting blocked show up, and privacy rules in Europe make collecting user data a legal problem.
I walk through the main scraping use cases on TikTok and a set of best practices for staying under the radar and out of trouble. Then I lay out a workflow, from defining the data you need to analyzing it once collected. On collection, I list three routes: building your own scraper, buying a ready-made one, or applying for the official API. I also cover where proxies fit in and what to check in a provider.
What’s inside
- Why I suggest scraping market trends and post engagement rather than user data, given privacy laws in Europe and elsewhere.
- How to pace requests: imitating human behavior with delays and batching, and tracking success rates, errors and response times.
- What each collection route requires, including the developer account and access application for the official TikTok API.
- How to treat the data afterward: removing irrelevant or redundant records, running NLP for sentiment and topics, and comparing with external sources.
In the full article: Full lists of use cases and best practices, the four-step workflow, and the criteria for choosing a proxy provider.