Summary of an article published on The Web Scraping Club on 2025-06-22 (free to read). Read the full article

LinkedIn has plenty of useful data, but scraping it the traditional way is a pain. An HTTP client with an HTML parser, or Playwright and Selenium, runs into the login wall, page structures that change from one listing to the next, and infinite scroll with client-side rendering. Automating the login also raises legal concerns, so we stay on public data. Hidden API endpoints are a better target: public, and more stable than page markup.

Taking job listings as the target, I show why a Google search lands on a Jobs URL that returns only the login wall, and which URL shows the real page. We then find the endpoint in the DevTools Network tab, using a Wayback Machine snapshot because LinkedIn removed dynamic loading from the public page. I then build a Python script with Requests and Beautiful Soup that calls the endpoint and parses the returned HTML into job data.

What’s inside

  • Why two Jobs URLs that look almost the same behave differently, and the query parameter that seems to bring back the login wall.
  • The endpoint has no official documentation, so the parameters for keyword, location and pagination come from testing, and some in the original URL can be ignored.
  • The endpoint returns raw HTML instead of JSON, with a limited number of listings per page.
  • Job cards carry different or missing fields, so the parsing checks every element before reading it.

In the full article: Python code for the request and parsing functions, the full endpoint with its parameters, the Wayback Machine snapshot to inspect, and the complete working script.

Read the full article on The Web Scraping Club