KO
|
EN
gitlite โ search
Search
#python
#javascript
#nodejs
#kubernetes
#react
#typescript
#vue
#xamarin
#npm
#docker
#python3
#flask
RedditLemmyImporter
โ 71
Open GitHub โ
๐ฅ Anti-Reddit Aktion ๐ฅ
Download README (.md)
Explore Similar Repositories
plemmy
:
A Python package for accessing the LemmyHttp API
Lemmy-Swift-Client
:
A Swift client for Lemmy.
lemmony
:
A better "All" browsing experience for Lemmy instances
Cavy
:
Lemmy client for iOS, written in Swift
lemmy-ui-svelte
:
A UI for Lemmy written in Svelte
// repository documentation
Was this content helpful?
โ 0
(0 ratings)
Select Rating:
โ
โ
โ
โ
โ
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
RedditLemmyImporter
?
Download (.md)
This project translates Reddit API responses into a PL/pgSQL script which loads the data into a [Lemmy](https://github.com/LemmyNet/lemmy/) database. In other words, it <img alt="takes" src="https://user-images.githubusercontent.com/95945959/166686199-b78b681c-843f-4d2f-9d51-ec65f8d0d630.png" height="25" /> Reddit posts/comments and <img alt="puts" src="https://user-images.githubusercontent.com/95945959/166686493-be7fb6da-a3ef-4312-8a46-90a650d4e552.png" height="25" /> them into Lemmy. ## Screenshots Here's an example of a backup of the now-banned r/GenZhou up and running on a Lemmy test instance: Community|Post ---|--- | ## Getting input data To get the JSON API response for a single post, you can call [the proper Reddit API](https://www.reddit.com/dev/api/#GET_comments_{article}) (requires an API key), or just append `.json` to the comments URL, like this: ``` HTML: https://www.reddit.com/r/GenZedong/comments/laucjl/china_usa/ https://www.reddit.com/r/GenZedong/comments/laucjl JSON: https://www.reddit.com/r/GenZedong/comments/laucjl/china_usa/.json?limit=10000 https://www.reddit.com/r/GenZedong/comments/laucjl.json?limit=10000 ``` Note that we've also added the `limit` parameter, because otherwise Reddit will pretty aggressively prune the comment tree with "Load more comments" links. The response object contains the data for that one post and any replies. You can feed this directly into RedditLemmyImporter. However, if you want to import multiple posts, you can put multiple responses in the same input file, with each one separated by a newline. For example: ``` ~ $ cat urls https://www.reddit.com/r/GenZedong/comments/tpyft9/why_is_like_half_this_sub_made_of_trans_women/ https://www.reddit.com/r/GenZedong/comments/pet8zc/therapist_trans_stalin_isnt_real_she_cant_hurt/ https://www.reddit.com/r/GenZedong/comments/ttcyok/happy_trans_visibility_day_comrades/ https://www.reddit.com/r/GenZedong/comments/t9kbdm/women_of_genzedong_i_congratulate_you_for_your_day/ ~ $ xargs -I URL curl --silent --user-agent "Subreddit archiver" --cookie "REDACTED" URL.json?limit=10000 < urls > dump.json ``` ## Cloning an entire subreddit If you need a complete scraping solution, check out [this Python script](https://lemmygrad.ml/comment/130292). It pulls posts into a local MongoDB database, which means you can run it on a cron to keep a local clone of posts as they're made. To export your `dump.json` try something like this: ``` mongoexport --uri="mongodb://localhost:27017/subredditArchiveDB" --collection=GenZedong --out=dump-wrapped.json ``` /r/GenZhou was scraped by `@DongFangHong@lemmygrad.ml` using this method. Data is available up to about a week before it was banned: https://mega.nz/file/knBwmTJL#PpqO0I3Jv-xw-o7RBWSi0JSScjSV7-4Eb3JR5HzTc5w Note that the script buries the data we need within a top-level property named `json`. RedditLemmyImporter can handle this directly using the `--json-pointer` option. For example: ``` java -jar redditLemmyImporter-0.3.jar -c genzhouarchive -u archive_bot -o import.sql --json-pointer=/json GenZhouArchive.json ``` ## Generating a SQL script using the release binary Prerequisites: Java 8 or above Download the jar file from the [releases page](https://github.com/rileynull/RedditLemmyImporter/releases) and run it: ``` java -jar redditLemmyImporter-0.3.jar -c genzhouarchive -u archive_bot -o import.sql dump.json ``` In this case we're generating a PL/pgSQL script that will load the data from `dump.json` into the comm `genzhouarchive` under the user `archive_bot`. The script will be written to `import.sql`. Full command usage: ``` Usage: redditLemmyImporter [OPTIONS] dump dump Path to the JSON dump file from the Reddit API. Required. Specify - to read from stdin. -c, --comm=name Target community name. Required. -u, --user=name Target user name. Required. --json-pointer=pointer Locate the Reddit API response somewhere within the top-level object in each input line. See RFC 6901 for the JSON Pointer specification. -o, --output-file=file Output file. Prints to stdout if this option isn't specified. -h, --help Show this help message and exit. -V, --version Print version information and exit. ``` ## Generating a SQL script using the source repository Prerequisites: JDK >=1.8, Maven 3. Clone the repo and cd to the source tree. Run: ``` mvn compile mvn exec:java -Dexec.args="-c genzhouarchive -u archive_bot -o import.sql path/to/dump.json" ``` (This will pull down dependencies from Maven Central so you must be connected to the internet during the compile step.) You could also package a release and then follow the instructions from the previous section: ``` mvn clean package java -jar target/redditLemmyImporter-0.3-SNAPSHOT.jar -c genzhouarchive -u archive_bot -o import.sql dump.json ``` ## Running the SQL script Copy `import.sql` to the server running Postgres and run this: ``` psql --dbname=lemmy --username=lemmy --file=import.sql ``` Note that this uses the default values for the database name and database username. If you've changed them in your [Lemmy configuration](https://join-lemmy.org/docs/en/administration/configuration.html#full-config-with-default-values) then update the values accordingly. **The target comm and target user must already exist in your Lemmy instance or the SQL script will do nothing.** ## Running the SQL script with Dockerized Lemmy Copy `import.sql` to the server running Docker and run this: ``` <import.sql docker exec -i $(docker ps -qf name=postgres) psql --dbname=lemmy --username=lemmy - ``` **The target comm and target user must already exist in your Lemmy instance or the SQL script will do nothing.**
