ShortVideoCrawl

KO | EN

gitlite — search

#ai #typescript #golang #go #android #rust #deep-learning #linux #nodejs #java #vue #awesome-list #security #open-source

ShortVideoCrawl

(★ 14)

Short video crawler based on scrapy

ShortVideoCrawl Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ShortVideoCrawl

[![GitHub](https://img.shields.io/github/license/dxsooo/ShortVideoCrawl)](https://raw.githubusercontent.com/dxsooo/ShortVideoCrawl/master/LICENSE)
[![CodeFactor](https://www.codefactor.io/repository/github/dxsooo/shortvideocrawl/badge)](https://www.codefactor.io/repository/github/dxsooo/shortvideocrawl)

Short video crawler based on [scrapy](https://github.com/scrapy/scrapy), crawling with search query of the target sites.

Supports:

> \*度小视/全民小视频官网已经下线，但是目前本项目仍可用（2024.6测试）

## Usage

requirements:

- python 3.10+
- poetry

### prepare

```bash
git clone https://github.com/dxsooo/ShortVideoCrawl
cd ShortVideoCrawl
poetry install --only main
poetry shell
```

### run

For example:

```bash
cd shortvideocrawl

# main parameters:
#   query: query word
#   count: target video count

# kuaishou
scrapy crawl kuaishou -a query='蔡徐坤' -a count=50

# xigua, with highest resolution and size smaller than 64 MB, duration smaller than 5 min
# scrapy crawl ixigua -a query='蔡徐坤' -a count=50

# xinpianchang, with highest resolution and size smaller than 64 MB, duration smaller than 5 min, but can only get a fixed number of video
scrapy crawl xinpianchang -a query='蔡徐坤'

# haokan, with highest resolution
# scrapy crawl haokan -a query='蔡徐坤' -a count=50

# quanmin
scrapy crawl quanmin -a query='蔡徐坤' -a count=50
```

videos are saved in `./videos`, named with video id of source platform.

ShortVideoCrawl

ShortVideoCrawl Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

ShortVideoCrawl

ShortVideoCrawl Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README