While I was performing web scraping, I discovered that I needed a specific string of characters to make an HTTP request to retrieve the information I was after. Unfortunately, I couldn’t find the string of characters anywhere except in the header of a specific AJAX request. In this post, I will share my experience in attempting to extract this string programmatically.
If you’ve ever explored the network tab of your browser’s devtools, you’ve probably come across the term HAR. HAR stands for HTTP Archive and is used to record all HTTP traffic exchanged between a web browser and a web server. Essentially, everything that you can access in the network tab is encompassed within an HAR file.
Generating HAR programmatically
First, initialize a new NodeJS project:
$ npm init -y
Then install Playwright as a dependency:
$ npm i playwright
Then, create a new file named index.js:
Then in package.json, add a script command named start: