nicar25-passive-scraping

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

nicar25-passive-scraping

★ 10 Open GitHub ↗

Passive scraping for social media (HAR and WACZ to the rescue)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Passive scraping for social media

*NICAR 2025, Minneapolis*

Jonathan Soma, Columbia University

[jonathan.soma@gmail.com](mailto:jonathan.soma@gmail.com) 🦃 [Twitter](https://twitter.com/dangerscarf) 🦃 [Bluesky](https://bsky.app/profile/dangerscarf.bsky.social) 🦃 [jonathansoma.com](https://jonathansoma.com)

- [Slides](slides.pdf)

## Useful tools

- [Instaloader](https://instaloader.github.io/), for downloading Instagram posts, comments and profile content.
- [yt-dlp](https://github.com/yt-dlp/yt-dlp), which will download any video from any website ever (mostly)
- [Playwright](https://playwright.dev/), my favorite scraping framework (browser automation!)
- [HAR data extractor](https://jonathansoma.com/har2data/)
- [WARC data extractor](https://jonathansoma.com/warc2data/)

## Tutorials

- [Finding Undocumented APIs](https://inspectelement.org/apis.html)

## Pack-ratting

### HAR files

> This is for Chrome, but is vaguely the same for other browsers.

Right-click the page and select **Inspect** to open up the developer tools.

![Right-click, inspect](https://raw.githubusercontent.com/jsoma/nicar25-passive-scraping/main/images/inspect.png)

Find the Network tab – you might need to resize the window a little bit, or click the >> arrow to find it.

![Network tab](https://raw.githubusercontent.com/jsoma/nicar25-passive-scraping/main/images/network-tab.png)

Refresh the page, scroll around, click the download icon.

![Download HAR](https://raw.githubusercontent.com/jsoma/nicar25-passive-scraping/main/images/har-download.png)

**You're done!** Now you can head to the [HAR Data Extractor](https://jonathansoma.com/har2data/)

### WACZ files

WACZ/WARC file support is not built into Chrome, so you'll need to download an extension for it. I recommend [Webrecorder ArchiveWeb.page](https://chromewebstore.google.com/detail/webrecorder-archivewebpag/fpeoodllldobpkbkabpblcfaogecpndd?hl=en).

Click the WACZ icon on your menu bar, create a new archiving session under "Save to..." if you feel like it, then click **Start archiving**.

![Open up WACZ](https://raw.githubusercontent.com/jsoma/nicar25-passive-scraping/main/images/wacz-one.png)

Reload the page to make it start. Scroll around, do whatever. When you're done, click the **Stop archiving** button.

> Supposedly Autopilot will scroll down for you forever, but it doesn't always work.

![Stop archiving with WACZ](https://raw.githubusercontent.com/jsoma/nicar25-passive-scraping/main/images/wacz-stop.png)

To download your WACZ file, click **View Archived Pages**, then **Download** on the bottom left. The WACZ file is fine!

![Download your WACZ file](https://raw.githubusercontent.com/jsoma/nicar25-passive-scraping/main/images/wacz-download.png)

**You're done!** Now you can head to the [WARC Data Extractor](https://jonathansoma.com/warc2data/)

## Slides

nicar25-passive-scraping

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

nicar25-passive-scraping

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README