warc-crawler

KO | EN

gitlite — search

#python #hacktoberfest #llm #go #android #machine-learning #linux #docker #awesome #cli #awesome-list #openai #ios #mcp

(★ 10)

Process web archives (WARC format) with StormCrawler and index content into OpenSearch

warc-crawler Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Let StormCrawler “Crawl” WARC Files

For more information about processing (and creating) WARC archives using StormCrawler, see
- related StormCrawler issues: [#755](https://github.com/apache/stormcrawler/issues/755)
- StormCrawler's [WARC module README](https://github.com/apache/stormcrawler/blob/master/external/warc/README.md)
- StormCrawler's [WARCSpout](https://github.com/apache/stormcrawler/blob/master/external/warc/src/main/java/org/apache/stormcrawler/warc/WARCSpout.java)

## Build the Project

```sh
mvn clean package
```

## Create List of WARC Files To Process

All topologies expect that WARC files to be processed are listed in text files line by line using
- either a local file system path (ideally absolute, relative paths may not work in distributed mode)
- or a http:// or https:// URL

Text files are expected in the folder `/data/input/`. The input folder is defined in the Flux files. Please change this location at your needs.

### Create Sample Input from Common Crawl

TODO

## Run the Crawler

### Run a Flux Topology

To submit a [Flux](https://storm.apache.org/releases/2.8.8/flux.html) to do the same:

``` sh
storm local target/warc-crawler-*.jar org.apache.storm.flux.Flux topology/warc-crawler-stdout/warc-crawler-stdout.flux
```

This will run the topology in local mode.

The command `storm jar ...` is used to run the topology in distributed mode:
``` sh
storm jar target/warc-crawler-*.jar org.apache.storm.flux.Flux topology/warc-crawler-stdout/warc-crawler-stdout.flux
```

It is best to run the topology in distributed mode to benefit from the Storm UI and logging. In that case, the topology runs continuously, as intended.

Note that in local mode, Flux uses a default TTL for the topology of 60 secs. The command above runs the topology for 24 hours (`24*60*60*1000` milliseconds). In distributed mode, the topology is run forever (until it is killed).

### Run a Java Topology

A Java topology class using the storm command:

``` sh
storm local target/warc-crawler-*.jar --local-ttl 600 -- org.commoncrawl.stormcrawler.CrawlTopology -conf topology/warc-crawler-stdout/warc-crawler-stdout-conf.yaml
```
This will launch the crawl topology in local mode for 10 minutes (600 seconds). Use `storm jar ...` to run the topology in distributed mode. Note: `--` is required to signal that remaining options (here `-conf`) are not consumed by `storm` and passed to the CrawlTopology as arguments.

## Alternative Topologies

Several Flux topologies are provided to test and evaluate crawling of WARC archives. Each Flux file is accompanied by a configuration file which fits the requirements of the topology run on a single host. You need to modify Flux file and configuration if you want to scale up and run the topology on a distributed Storm cluster.

### DevNull Topology

[warc-crawler-dev-null](topology/warc-crawler-dev-null/) runs a single WARCSpout which sends the page captures to [DevNullBolt](https://storm.apache.org/releases/2.4.0/javadocs/org/apache/storm/perf/bolt/DevNullBolt.html) which (you guess it) only ack's and discards each tuple. Useful to measure the performance of the WARCSpout.

### Stdout Topology

[warc-crawler-stdout](topology/warc-crawler-stdout/) reads WARC files, parse the content payload, maps content and metadata fields to index fields and writes fields (shortened) to the log output:
```
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] pagetype      article
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] pageimage     169 chars
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] keywords      Coronavirus
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] keywords      NHS
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] keywords      Politics
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] keywords      Boris Johnson
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] keywords      Matt Hancock
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] keywords      Rishi Sunak
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] keywords      Sunderland
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] capturetime   1601983220000
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] description   114 chars
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] title Coronavirus LIVE updates: Boris Johnson Tory conference ...
2020-10-08 14:30:30.113 STDIO Thread-4-index-executor[2 2] [INFO] publicationdate       2020-10-06T11:05:33Z
```

This topology can be used to test parsers and extractors without the need to setup any indexer backend. The Java topology class ([CrawlTopology](src/main/java/org/commoncrawl/stormcrawler/CrawlTopology.java)) runs an equivalent topology.

### Rewrite WARC files

[warc-crawler-warc-rewrite](topology/warc-crawler-warc-rewrite/) reads WARC files and sends the content to a WARC writer bolt which stores it again in WARC files. Could be extended by additional bolts to filter and/or enrich the WARC records.

### Index into OpenSearch

[warc-crawler-index-opensearch](topology/warc-crawler-index-opensearch/) reads WARC files, parses HTML pages, extracts text and metadata and sends documents into OpenSearch for indexing.

This topology requires that OpenSearch is running:
- Install OpenSearch (and Kibana) 2.9.5 - also higher versions might work
- Start OpenSearch
- Initialize OpenSearch indices by running [OS_IndexInit.sh](topology/warc-crawler-index-opensearch/OS_IndexInit.sh)
- Initialize the dashboards by running [importDashboards.sh](topology/warc-crawler-index-opensearch/dashboards/importDashboards.sh)
- Adapt the [opensearch-conf.yaml](topology/warc-crawler-index-opensearch/opensearch-conf.yaml) file, so that OpenSearch is reachable from the Storm workers – the host name `opensearch-sc` is used in the [Docker setup](#run-topology-on-docker), change the host name to `localhost` when running in local mode with a local OpenSearch installation.
- When the topology is running, visit the status dashboard: <http://localhost:5601/app/dashboards#/view/Crawl-status>

See also the documentation of [StormCrawler's OpenSearch module](https://github.com/apache/stormcrawler/tree/master/external/opensearch).

## Run Topology on Docker and Docker Compose

You can run the topologies which require no indexing backend using the provided Dockerfile:
```
docker build . -t warc-crawler:latest

docker run --rm -ti -v /path/to/warc/data:/data/input warc-crawler:latest /bin/bash

$> storm local warc-crawler.jar org.apache.storm.flux.Flux topology/warc-crawler-stdout/warc-crawler-stdout.flux
```

In addition, a configuration to run the topologies via [docker-compose](https://docs.docker.com/compose/) is provided. The file [docker-compose.yaml](./docker-compose.yaml) puts every component (Storm Nimbus, Supervisor and UI, but also OpenSearch) into its own container. The topology is launched from a separate container which is linked to the container of Storm Nimbus.

WARC input is per default read from the folder `warcdata` in the current directory. Another location can be defined by setting the environment variable `WARCINPUT`:
```sh
WARCINPUT=/my/warc/data/path/
export WARCINPUT
```

First we launch all components:

```
docker compose -f docker-compose.yaml up --build --renew-anon-volumes --remove-orphans
```

Now we can launch the container `storm-crawler`
```
docker compose run --rm storm-crawler
```

and in the running container our topology:
```
$warc-crawler/> storm jar warc-crawler.jar org.apache.storm.flux.Flux topology/warc-crawler-dev-null/warc-crawler-dev-null.flux
```

Let's check whether topology is running:
```
$warc-crawler/> storm list
Topology_name        Status     Num_tasks  Num_workers  Uptime_secs
-------------------------------------------------------------------
warc-crawler-dev-null ACTIVE    6          1            240
```

Also the [Storm UI on localhost](http://localhost:8080/) is available and will provide metrics about the running topology.

To inspect the worker log files we need to attach to the container running Storm Supervisor
```
docker exec -it supervisor /bin/bash
```
then find the log file and read it:
```
$> ls /logs/workers-artifacts/*/*/worker.log
/logs/workers-artifacts/warc-crawler-dev-null-1-1603368933/6700/worker.log

$> more /logs/workers-artifacts/warc-crawler-dev-null-1-1603368933/6700/worker.log
```

If done we kill the topology
```
$warc-crawler/> storm kill warc-crawler-dev-null -w 10
1636 [main] INFO  o.a.s.c.kill-topology - Killed topology: warc-crawler-dev-null
```

... and leave the container (`exit`) and shut down all running containers:
```
docker compose down
```

Of course, the topology could be also launched in a single command:
```
docker compose run --rm storm-crawler storm jar warc-crawler.jar org.apache.storm.flux.Flux topology/warc-crawler-dev-null/warc-crawler-dev-null.flux
```

For additional information, please see the [StormCrawler Docker Compose documentation](https://stormcrawler.apache.org/docs/3.6.0/index.html#_docker_compose_setup).

#### Run OpenSearch Topologies on Docker

First, the OpenSearch indices and dashboards need to be initialized by running [OS_IndexInit.sh](topology/warc-crawler-index-opensearch/OS_IndexInit.sh) and  [importDashboards.sh](topology/warc-crawler-index-opensearch/dashboards/importDashboards.sh).

Then the OpenSearch topology can be launched via
```
docker compose run --rm storm-crawler storm jar warc-crawler.jar org.apache.storm.flux.Flux \
   topology/warc-crawler-index-opensearch/warc-crawler-index-opensearch.flux
```

warc-crawler

warc-crawler Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

warc-crawler

warc-crawler Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README