scdx

KO | EN

gitlite — search

#javascript #python #react #machine-learning #rust #deep-learning #docker #nodejs #windows #awesome #pytorch #ios #openai #open-source

(★ 16)

A simple tool for querying the Common Crawl CDX

scdx 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# `scdx`
## Simple CDX

![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)![Rust](https://img.shields.io/badge/rust-%23000000.svg?style=for-the-badge&logo=rust&logoColor=white)

A tool for querying the Common Crawl CDX Index.  Versions in both Python and Rust are included in this repository.  The command–line syntax is identical in both versions.

### Installation:

1. Clone this repository
2. To run the Rust version, compile and run via:
```
$ cargo build --release
```

```
$ cd target/release
$ chmod +x scdx
```

### Usage:

```bash
$ scdx --sleep 2 --domain commoncrawl.org --crawls CC-MAIN-2021-04 CC-MAIN-2024-10
```

```bash
$ scdx -s 10 -d '*.wikipedia.org' -c CC-MAIN-2023-50
```

```bash
$ scdx -l -d apple.com
```

The program will display a progress bar and output a file with a timestamp (e.g `2024-02-27_18-34-50_output.jsonl`) to the working directory, unless the `-o` or `--output` options are used.

The default sleep time is 2 seconds. Please be polite! Polling multiple times a second will make the index server sad. See the CCF system status [here](https://status.commoncrawl.org).

If no crawls are specified, all crawls will be queried. Use the `-l` or `--latest` flag to only query the latest crawl.

The API used supports two methods of wildcarding, like the (more advanced and mature) [cdx-toolkit](https://github.com/cocrawler/cdx_toolkit) by Greg Lindahl.

- **Prefixed asterisk**

The query `*.example.com`, in CDX jargon sets `matchType='domain'`, and will return captures for `blog.example.com`, `support.example.com`, etc.

- **Appended asterisk**

The query `example.com/*` will return captures for any page on `example.com`.

The Python version uses [`tqdm`](https://tqdm.github.io/) to display a progress bar, and the Rust version uses [`indicatif`](https://docs.rs/indicatif/latest/indicatif/).

### Licence
[MIT License](LICENSE)

### Thanks
- **[Greg Lindahl](https://github.com/wumpus)**
- **[Pedro Ortiz Suarez](https://github.com/pjox)**

scdx

scdx 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

scdx

scdx 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README