KO
|
EN
gitlite — search
Search
#python
#java
#python3
#arduino
#golang
#machine-learning
#rust
#html
#flask
#javascript
#seismology
#nodejs
ek-scraper
★ 19
Open GitHub ↗
Scraper for kleinanzeigen.de
Download README (.md)
Explore Similar Repositories
dbRackCsound
:
No description available.
simple_peer_flutter
:
Simple WebRTC for flutter (similar to the simple-peer project)
Astar-Path-Planner-for-2D-grid-map-in-ROS
:
A*-Path-Planner-for-2D-grid-map-in-ROS
awesome-pbbg
:
No description available.
remora-flexi-hal
:
Fork of LinuxCNC Remora project for Flexi-HAL CNC controller.
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
ek-scraper
?
Download (.md)
# ek-scraper Simple scraper for kleinanzeigen.de searches with notifications for new ads. ## Installation Install this package from PyPi in a separate virtual environment using [`uv`](https://docs.astral.sh/uv/). ``` sh uv tool install ek-scraper ``` ## Usage > For the full usage check the `ek-scraper --help` command Create a configuration file using ``` sh ek-scraper create-config <path/to/config.json> ``` The example configuration file will look like this: ```json { "filter": { "exclude_topads": true, "exclude_patterns": [] }, "notifications": { "pushover": { "token": "<your-app-api-token>", "user": "<your-user-api-token>", "device": [] }, "ntfy.sh": { "topic": "<your-private-topic>", "priority": 3 }, "email": { "smtp_host": "<smtp.example.com>", "smtp_port": "587", "username": "<your-account-username>", "password": "<your-account-password>", "sender": "<sender@example.com>", "recipient": ["<receiver@other.example.com>","<receiver@another.example.com>"], "bcc": ["<blindreceiver@third.example.com>","<blindreceiver@fourth.example.com>"] }, }, "searches": [ { "name": "Wohnungen in Hamburg Altona", "url": "https://www.kleinanzeigen.de/s-wohnung-mieten/altona/c203l9497", "recursive": true } ] } ``` See [Configuration](#configuration) for details on all configuration options. * Configure one or more searches in the `searches` section of the configuration, see [Searches](#searches) for more details * Configure notifications in the `notifications` section of the configuration, see [Notifications](#notifications) for details on notification configuration * (Optional) Configure filters in the `filter` section of the configuration, see [Filter](#filter) for more details Run the following command to initialize the data store without sending any notifications: ``` sh ek-scraper run --no-notifications path/to/config.json ``` Afterwards, run ```sh ek-scraper run path/to/config.json ``` to receive notifications according to your `notifications` configuration. ## Development Follow the steps below to set up a development environment for this project. 1. Clone this repository ``` sh git clone git@github.com:jonasehrlich/ek-scraper.git ``` 2. Change directory into the repository ``` sh cd ek-scraper ``` 3. Create a virtual environment using uv ``` sh uv sync ``` 4. (Optional) Install pre-commit environment using `pre-commit` or `prek` ## Configuration ### Searches Searches can be configured in the `searches` array of the configuration file. Each of the searches can be configured with the following parameters. | Name | Description | | ----------- | ---------------------------------------------------------------------------------- | | `name` | Name of the search, use a descriptive one (required) | | `url` | URL of the first page of your search (required) | | `recursive` | Whether to follow all pages of the search result <br/>(optional, defaults to true) | ### Filter Filters can be configured in the `filter` section of the configuration file to exclude specific ads from your scrape results on the client side. The following settings can be configured. | Name | Description | | ---- | ----------- | | `exclude_topads` | Whether to exclude top ads from the results (optional, defaults to true) | | `exclude_patterns` | Case-insensitive regular expression patterns used to exclude ads (optional) | ### Notifications Notifications can be configured in the `notifications` section of the configuration file. #### Push notifications using [Pushover](https://pushover.net/)  `ek-scraper` supports push notifications to your devices using [Pushover](https://pushover.net/). For further information on the service check their terms and conditions. The implementation for _Pushover_ notifications will send a single notification per search, if new ads are discovered. To configure _Pushover_ for notifications from the scraper, first register at the service and create an application (e.g. `ek-scraper`). To use the service in `ek-scraper`, add the `pushover` object to the `notifications` object in your configuration file and fill the API tokens. Selection of the devices which will receive the notifications, is supported using the `device` array. | Name | Description | | -------- | ------------------------------------------------------------------------------------------- | | `token` | API token of the Pushover app (required) | | `user` | API token of the Pushover user (required) | | `device` | List of device names to send the notifications to <br/> (optional, defaults to all devices) | #### Push notifications using [ntfy.sh](https://ntfy.sh/)  `ek-scraper` supports push notifications to your devices using [ntfy.sh](https://ntfy.sh/). For further information on the service check their terms and conditions. The implementation for _ntfy.sh_ notifications will send a single notification per search, if new ads are discovered. To configure _ntfy.sh_ for notifications from the scraper, define a topic and subscribe to it in the mobile app. > Note that topic names are public, so it's wise to choose something that cannot be guessed easily. > This can be done by including a UUID, e.g. by running the following command in your shell: > > ``` sh > echo "ek-scraper-$(uuidgen)" > ``` To use the service in `ek-scraper`, add the `ntfy.sh` object to the `notifications` object in your configuration file and add the topic you previously subscribed to. | Name | Description | | ---------- | ----------------------------------------------------------------- | | `topic` | Topic to publish the notifications to | | `priority` | Priority to send the notifications with (optional, defaults to 3) | #### Push notifications using E-Mail `ek-scraper` supports push notifications to your devices using E-Mail. The implementation for _E-Mail_ notifications will send a single Email containing the new results per search, if new ads are discovered. To configure _E-Mail_ for notifications from the scraper, you need to add your smtp server details and credentials. To use the service in `ek-scraper`, add the `email` object to the `notifications` object in your configuration file. Currently only supporting STARTTLS. | Name | Description | | ------------ | ------------------------------------------------------------------------------------------- | | `smtp_host` | SMTP host address (required) | | `smtp_port` | SMTP host port (default: 587) (required) | | `username` | Email account username (required) | | `password` | Email account password (required) | | `sender` | Outbound email address (required) | | `recipient` | Recipients who receive the email (optional) | | `bcc` | Recipients who the results get sent to as bcc (optional) | ## Running `ek-scraper` regularly > It should be avoided to run the tool too often to avoid getting your IP address blocked by > [kleinanzeigen.de](kleinanzeigen.de) In order to run `ek-scraper` regularly on a Unix-like system, configure it as a cronjob. To configure a cronjob, run ``` sh crontab -e ``` Edit the crontab table to run the command you want to run. A handy tool to check schedule configurations for cronjobs is [crontab.guru](https://crontab.guru/). For more information on configuring cronjobs use your favorite search engine.
