stl-scraper

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #claude-code #codex #cli #developer-tools #react #windows

★ 84 Open GitHub ↗

Python scraper for Airbnb and short-term rental market research.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# stl-scraper - Short-Term Listings Scraper

Scrape short-term listings providers (Airbnb).

Given a search query, e.g. "San Diego, CA" or "Rome, Italy", search Airbnb inventory and collect data on listings. Save
results to a CSV file or Elasticsearch.

## Notice

This project is maintained on a best-effort basis; API changes may require updates.

## Usage

### Usage (basic installation)

```shell
# activate the virtual env
. .venv/bin/activate

# run the script
./stl.py search "Madrid, Spain"
```

### Usage (advanced installation)
```shell
# spin up the containers
docker compose up -d

# ensure dependencies installed
docker compose exec jupyter-scipy-notebook conda install --yes --file work/requirements.txt

# run the script
docker compose exec jupyter-scipy-notebook /opt/conda/bin/python work/stl.py search -v "Madrid, Spain"
```

## Options

```
Short-Term Listings (STL) Scraper

Usage:
    stl.py search <query> [--checkin=<checkin> --checkout=<checkout> 
                  [--priceMin=<priceMin>] [--priceMax=<priceMax>]] 
                  [--roomTypes=<roomTypes>] [--storage=<storage>] [-v|--verbose]
    stl.py calendar (<listingId> | --all)
    stl.py pricing <listingId> --checkin=<checkin> --checkout=<checkout>
    stl.py data <listingId>

Arguments:
    <query>          The query string to search (e.g. "San Diego, CA")
    <listingId>      The listing id

Options:
    --checkin=<checkin>    Check-in date, e.g. "2023-06-01"
    --checkout=<checkout>  Check-out date, e.g. "2023-06-30"
    --priceMin=<priceMin>  Minimum nightly or monthly price
    --priceMax=<priceMax>  Maximum nightly or monthly price
    --all                  Update calendar for all listings (requires Elasticsearch backend)

Global Options:
    --currency=<currency>  "USD", "EUR", etc. [default: USD]
    --source=<source>      Only allows "airbnb" for now. [default: airbnb]
```

## Requirements

- Python >= 3.10, or Docker Compose

## Installation

### Installation via pip (Option 1 - Basic)

This option assumes you have Python >= 3.10 installed, and that you will manage dependencies using the python `venv` 
module with `pip`. You can connect to your own instance of Elasticsearch. However, Elasticsearch is not required.

```shell
# create the config file
cp .env.dist .env

# create the virtual env
python3 -m venv .venv

# activate the virtual env
. .venv/bin/activate

# install dependencies in virtual env
pip install -r requirements.txt
```

### Installation via docker compose (Option 2 - Advanced)

This option uses docker compose to build:

- `jupyter-scipy-notebook`: jupyter scipy notebook, python, conda
- `setup` temporary container that configures elasticsearch & kibana security
- `es01`: elasticsearch container
- `kibana`: kibana container

```shell
# create the config file
cp .env.dist .env

# Create the containers
docker compose up -d

# Install project requirements
docker compose exec jupyter-scipy-notebook conda install --yes --file work/requirements.txt
```

## Using kibana

You can directly view records in Elasticsearch by using Kibana.

1. Scrape some listings using above commands
2. Browse to http://localhost:5601/app/management/kibana/dataViews (u: elastic / p: abc123)
3. Click "Create new data view" on top right
4. Use `short-term-listings` as name and index pattern
5. Click "Save data view to Kibana"
6. Click "Analytics > Discover" on the main menu, selecting the `short-term-listings` data view, and see JSON records

## Troubleshooting

- If using Elasticsearch, you may need to run `sudo sysctl -w vm.max_map_count=262144` on your host machine (see
  https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html)

stl-scraper

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README