crawler-planning

KO | EN

gitlite — search

#python #hacktoberfest #react #golang #rust #linux #java #kubernetes #vue #awesome-list #pytorch #openai #mcp #open-source

crawler-planning

(★ 15)

Task tracking for the crawlers we're working on

crawler-planning Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Welcome to the OpenSanctions crawler planning repository!

This is where we coordinate implementing crawlers - the open source code which takes data from various sources, converts it to our data model, and adds it to our database for use in screening and investigations.

We have a contracted team who helps us keep on top of demand and our data coverage goals. If you're [interested in joining, let us know](https://www.opensanctions.org/docs/company/jobs/crawler-team/)!

We also welcome volunteer open source contributions, but recommend that you first take a look at the project board and [chat with us](https://www.opensanctions.org/contact/) to discuss the most appropriate next crawler to add. We'll then handle the movements on the project board described below on your behalf.

## New data source suggestions

Suggestions for data sources can be added by [submitting an issue](https://github.com/opensanctions/crawler-planning/issues/new/choose) in this repository. But please have a quick little [search](https://github.com/opensanctions/crawler-planning/issues) to see if someone else has already suggested the same source.

We review submissions and queue them for addition if/when they meet our [data inclusion criterea](https://www.opensanctions.org/docs/criteria/).

## Getting started on the team

We will assign your first task on the [project board](https://github.com/orgs/opensanctions/projects/2) which we'll pick to be a nice introductory task.

Your next step is to dive into the [instructions to get started with zavod](https://zavod.opensanctions.org/), our ETL framework, and see if you can get a crawler working.

## Starting off

### Issue Documentation
Record important decisions and reasoning in GitHub issues, such as explaining why a crawler skips an interesting section of the site.

### Branch Naming Conventions
Follow a consistent branch naming strategy to avoid issues with special characters in shell environments. Use **slug-like** names that are descriptive, hyphen-separated, and free of special characters such as `#`, `(`, or `)`:

```bash
git checkout planning-206-office-of-antiboycott-compliance
```

## Workflow

The **Todo column** is ordered in decreasing priority.

When you're ready to start your next task, take the next top card in the **Todo column** and move it to the **In Progress column**. Add yourself as an assignee, if we haven't already done so.

When you feel your new/modified crawler is ready for production,
  - create a pull request to the [opensanctions](https://github.com/opensanctions/opensanctions) repository with your branch
  - [link your pull request to the card issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)
  - move the card to the **In Review column**.

If it's ready
  - we'll add it to the appropriate collection
  - we'll merge it
  - we'll move it to the **Merged column**
  - it will be deployed automatically, and run according to its schedule.

If some changes are needed, we'll comment on the pull request, and move it back to to the **In Progress column** for you to revise.

Please try to finish any work in progress before starting anything new.

## Kinds of crawlers, sources, or data

Crawlers generally bring one type of [target](https://www.opensanctions.org/docs/glossary/#targets) into our database, and tasks are annotated to indicate the type:

- `crime-list` - crime-linked entities
- `sanctions-list` - sanctioned entities
- `pep-list` - Politically exposed persons

Each kind of target should be [annotated according to our data model](https://www.opensanctions.org/docs/topics/).

## Releasing datasets

Our Deploy Team members can release datasets once they have been merged, and completed their first successful run.

Using the dataset name `jp_meti_ru` as example:

- If it's a company register or other enrichment source, we won't add it to `default`, the next step is to configure an enricher and that would be released as follows.
- If the planning card has the `check-release-plan` label, check inside the crawler for any deviation required from the following.
    - This is normally for special datasets that aren't intended to be added published in `default` (perhaps via its children).

1. Check if it has completed a run - visit https://www.opensanctions.org/datasets/jp_meti_ru/
    - check that there aren't any errors, and ideally no warnings.
    - if it hasn't run yet, perhaps trigger it.
2. Check that everything looks good.
    - Check that the title, summary, description, publisher name and description all look sensible rendered on the website
    - Check that any markdown like paragraph breaks, hyperlinks, bulleted lists etc rendered correctly
    - Check once more for spelling and grammar issues
    - Download the data and spot check a couple of entities
      - Check that they have the expected topic
      - Check that they have names
      - These are boring things that have gone wrong and been caught on release
3. Release the dataset
     - Make sure coverage.start is correct - either the date of the first run for dynamically updated data, or the date of extraction or original source publication for static datasets.
     - Add the dataset to the most appropriate collection which is a descendant of `default`
4. Drag the card to Added to collection
5. When the parent collection and `default` exporters have run, and the website cache cleared
    - Check that you can see the entities in search results. Spot check that a couple look good.
    - Perhaps add a comment to the card to highlight something interesting about this dataset to mention in announcements
6. Drag the card to Ready to be announced
    - Once here, it can be announced in the various channels

crawler-planning

crawler-planning Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

crawler-planning

crawler-planning Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README