opendata-content.log-tool

KO | EN

gitlite — search

#javascript #react #typescript #llm #go #android #machine-learning #rust #deep-learning #linux #docker #macos #cli #pytorch

opendata-content.log-tool

(★ 11)

A reference implementation for processing the content.log files found at opendata.dwd.de/weather.

opendata-content.log-tool Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# content.log Tool

A reference implementation for processing the content.log files found at opendata.dwd.de/weather.

Example usage:

```
CONTENT_LOG_URL="https://opendata.dwd.de/weather/nwp/content.log.bz2"
PATTERN="/icon-d2/grib/03/t_2m/.*_icosahedral_"
LAST_RUN_AT=$(date -ud 00:00 -Ihours)

wget $CONTENT_LOG_URL -O content.log.bz2
bzgrep $PATTERN content.log.bz2 > my_content.log
./get_updated_files.py -b $CONTENT_LOG_URL -u $LAST_RUN_AT my_content.log > updated_files.txt
wget -i updated_files.txt
```

Running the program above will download all updated files into the current working directory. The produced file
`updated_files.txt` will hold hyperlinks to files that are updated since the given date-time according
to the file's modification date found in content.log.

Also mind that there are multiple servers behind https://opendata.dwd.de which might not be exactly in sync with each
other regarding file modification timestamps. Look into the code of `get_updated_files.py` for a suggestion on how to
deal with that.

While this program relies on the file modification timestamp dumped into `content.log.bz2`, it might be more feasible
to process the data reference time that is contained in the filenames instead.

```
$ ./get_updated_files.py --help
usage: get_updated_files.py [-h] --updated-since UPDATED_SINCE [--url-base URL_BASE]
                            [--min-delta MIN_DELTA] [--version]
                            [CONTENT_LOG_FILE [CONTENT_LOG_FILE ...]]

Filters paths of a DWD Open Data content.log file for entries that have been updated.

positional arguments:
  CONTENT_LOG_FILE      The decompressed content.log file (default: STDIN)

optional arguments:
  -h, --help            show this help message and exit
  --updated-since UPDATED_SINCE, -u UPDATED_SINCE
                        last time files were checked for updates
  --url-base URL_BASE, -b URL_BASE
                        resolve the paths taken from content.log relative to the given
                        base URL; put the URL of the content.log.bz2 here to end up with
                        correct hyperlinks to DWD's Open Data
  --min-delta MIN_DELTA, -d MIN_DELTA
                        minimum number of seconds a file needs to be younger than
                        UPDATED_SINCE (default: 60)
  --version             show program's version number and exit
```

opendata-content.log-tool

opendata-content.log-tool Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

opendata-content.log-tool

opendata-content.log-tool Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README