reddit-text-extract

KO | EN

gitlite — search

#javascript #python #go #android #machine-learning #linux #java #awesome #kubernetes #awesome-list #pytorch #security #mcp #open-source

reddit-text-extract

(★ 9)

Python code to extract text from subreddits

reddit-text-extract Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# reddit-text-extract

Python code to extract all the text data from posts, urls and comments on various subreddits.

It is useful for collecting text corpus related to different domains when working on language modelling tasks.

## Usage

* `git clone git@github.com:shagunsodhani/reddit-text-extract.git`
* `cd reddit-text-extract`
* `sudo pip3 install -r requirements.txt`
* Add name of subreddits (to crawl data from) in `config/subreddit.txt`
* `cp config/config.cfg.sample config/config.cfg`
* Populate values in `config/config.cfg`
    * `user_agent` = user agent as defined by [praw](https://praw.readthedocs.io/en/stable/) . Can put in any value.
    * `subreddit_list_path` = path to `config/subreddit.txt`
    * `path_to_save` = file path where crawled data is to be saved.
    * `readability_token` = token to use [Readability parser API](https://www.readability.com/developers/api/parser). Can be obtained from [here](https://www.readability.com/developers/api).
* `python3 app/scrape.py`

reddit-text-extract

reddit-text-extract Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

reddit-text-extract

reddit-text-extract Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README