KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #typescript #claude-code #codex #dsh #dsh-plugin #ai-agent #ai-agents #deepseek-harness #agent-skills #macos #cli

Twitter-Scraper

★ 100 Open GitHub ↗

A python web scraper for Twitter data

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Twitter-Scraper
This is not a perfect scraper, so feel free to add improvements if you find any.

IMPROVEMENTS:
- Improved error handling so that tweets are not rejected if certain fields are null, etc...
- Leveraged the `WebDriverAwait` class to enable better detection of desired load states
- Each record is saved while scraping instead of all at the end; minimizing data loss for a failed session.

NOTES AND THINGS TO THINK ABOUT:
- The `scroll_down_page` function has an argument for `num_seconds_to_load` that represents the num of
seconds that the program will wait until attempting to scroll again. I'm currently making 5 attemps with
a pause between. You could also increase the number of max attempts and decrease the `num_seconds_to_load`.
This could possibly speed up the scraping as you would be more likely to get to a successfull scroll down
quicker.
-  The `collect_all_tweets_from_current_view` function has a `lookback_limit` argument that controls how
many tweets are processed from each scroll. I've written more about this in the function docstring.
- I've implemented `WebDriverWait` in several sections of this updated code. I think this is a much
better solution than a hard-coded `sleep` call because it will only timeout after a certain period of
time if specific conditions are not met. There are many other sections of this code that could be
improved, I'm sure, by leveraging this class.
- Feel free to replace the `save_tweet_data_to_csv` function with any other `io` option you want, such
as a database save via `pyodbc`, `sqlite3`, or whatever you want really.
- I encourage you to explore the "Advanced Search" functionality. Try adding your criteria and see how the url
 is built. You can then leverage this to make your searches more customized... with date ranges, special keywords,
 etc...  --> https://twitter.com/search-advanced?

Twitter-Scraper

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README