KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #dsh #dsh-plugin #ai-agent #claude-code #codex #typescript #ai-agents #mcp #cli #rust #deepseek-harness

opendataindex

★ 46 Open GitHub ↗

Open Data Index website

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Deprecated!

This codebase represents the static Index website generator for the Global Open Data Index before 2016. The current Survey and Index codebase are now unified at: https://github.com/okfn/opendatasurvey.

# Open Data Index

This is the code that powers the [Open Data Index](http://index.okfn.org/), an [Open Knowledge](http://okfn.org/) initiative.

The Open Data Index displays a snapshot of data collected in an [Open Data Survey](http://census.okfn.org).

This new Index codebase was written for the [2014 Global Survey](http://global.census.okfn.org), and can be used with any survey powered by the [survey codebase](https://github.com/okfn/opendatacensus).

The code generates a static site using [Pelican](http://docs.getpelican.com), a Python-based static site generator.

## Setup

Getting setup with the code is easy if you have some familiarity with Python and virtualenv.

* Setup a virtualenv for the project
* Install the dependencies: `pip install -r requirements.txt`
* Install the CLI: `cd cli && python setup.py install && cd ../`
* Prepare the data (download from the database): `odi prepare`
* Populate the content sources: `odi populate`
    * Use `odi populate --limited` if you have a large amount of data (like the Global Index), and want a smaller set for local development (see Configuration below)
* `./develop_server.sh start` to run a server that watches and builds

> Use `python cli/debug.py` commad instead of `odi` command in development mode
and to be sure it's a fresh version of CLI.
> It's possible to use `pelican --debug -s config_default.py -o output content && cd output && python -m SimpleHTTPServer 4444` for site building and viewing to avoide Pelican built-in server. Also `python -m SimpleHTTPServer 4444` can be usefull before pushing to gh-pages branch.

[Pelican documentation for more information](http://docs.getpelican.com)

## Configuration

* Add a `config_instance.py` file to the root of the directory (this will not be version-controlled)
    * see `config_instance.example.py` for reference
* All special config for the Open Data Index project is found on the ODI object
* Override any settings you wish in this file
* For example to use local database:

```
from config_default import *
ODI['database'] = {
    'entries': 'http://global.census.okfn.org/api/entries/{year}.cascade.json',
    'places': 'http://global.census.okfn.org/api/places/score/{year}.cascade.json',
    'datasets': 'http://global.census.okfn.org/api/datasets/score/{year}.cascade.json',
    'questions': 'http://global.census.okfn.org/api/api/questions.json',
}
```

* If you have a large dataset, rebuilding and watching on your local development server can be a pain. You can limit the build in this case wth the following settings, and run with `odi populate --limited`:

```
from config_default import *
ODI['limited']: {
    'places': ['au'], # any place identifier
    'datasets': ['timetables'] # any dataset identifier
}
```

### Preparation

Data is prepared by the python script. This pulls data
from the Open Data Index Survey (Census), processes it in various ways and then
writes it to the `data` directory.

To run the script do:

odi prepare

### Populate

Once the data has been prepared via the `process.py` script, there is an additional step to create the source files that will be used to generate the static site. All source files for rendering pages live under `content/pages`, and in this location, everything under `datasets`, `historical` and `places` is generated by running the following command:

odi populate

If your your census data is sizable, or if you have many languages, Pelican's build times can be considerable, and this is a pain in development. To counter this, there is a `--limited` flag for populate that will generate a subset of `places` and `datasets` (configured on the ODI object in the settings file):

odi populate --limited

## Deployment

Steps to create a snapshot for deployment::

odi populate
    odi deploy

## Data

Data is found in the `data` directory. This both powers the site build and is
usuable in its own right (as a Tabular Data Package).

## Visualisations

The Open Data Index has a small set of tools and patterns for implementing visualisations. The first visualisation is the **Choropleth map** (described below), and this is the reference implementation for other visualisations to follow.

Visualisations have the following features:

* A permalink for the visualisation (e.g.: /vis/map/)
* An embeddable version of the visualisation (e.g.: /vis/map/embed/)
* Tools for sharing the visualisation to social networks
* Tools to filter data in the visualisation
* An interface to pass state to a visualisation via URL params

### Choropleth map

Displays Open Data Index data via a map interface.

This visualisation is embedded in the project home page, and also available via its permalink:

* /vis/map/

Or its embeddable version:

* /vis/map/embed/

#### Map state

The state of the map is configurable: internally, via the uiState object, and via query params passed on the URL to the map.

The use case for this is to enable embedding of a particular state. The uiState object has the following defaults:

uiStateDefaults = {
        embed: {
            width: '100%', // min-width for the map is 400px
            height: '100%',// min-height for the map is 360px
            title: 'Open Data Index'
        },
        filter: {
            year: currentYear,
            dataset: 'all'
        },
        panel: {
            name: true,
            tools: true,
            help: true,
            legend: true,
        },
        map: {
            place: undefined
        },
        asQueryString: undefined
    }

These defaults can be customized on initialisation of the map via URL params (a subset of the full uiState object):

* `filter_year`
* `filter_dataset`
* `panel_logo`
* `panel_name`
* `panel_tools`
* `panel_help`
* `panel_legend`
* `map_place`
* `embed_width`
* `embed_height`
* `embed_title`

##### Examples

1. 2014 / All datsets

`http://staging.index.okfn.org/vis/map/embed/?embed_height=360`

2. 2014 / Government spending

`http://staging.index.okfn.org/vis/map/embed?filter_dataset=spending&embed_height=360`

### Data tables

Displays the Index data in a tabular format. **This visualisation is not currently embeddedable.**

There are two types of tables:

* Overview
* Slice

## API

Open Data Index exposes a (simple) API for programmatic access to data. Currently, the API is available in both JSON and CSV formats.

### API endpoints

* {format} refers to either `json` or `csv`

#### /api/entries.{format}

Returns all entries in the database.

#### /api/entries/{year}.{format}

Returns entries sliced by year.

Available years:

* 2014
* 2013

#### /api/datasets.{format}

Returns all datasets in the database.

#### /api/datasets/{category}.{format}

Returns all datasets sliced by category, where `category` is a slugified string of the dataset category.

Available categories:

* civic-information
* environment
* finance
* geodata
* transport

#### /api/places.{format}

Returns all places in the database.

#### /api/questions.{format}

Returns all questions in the database.

## Translations

There are two types of Open Data Index content to translate: strings in the codebase, and strings in the source data.

Translations are managed with the Transifex client:

http://docs.transifex.com/developer/client/

* We use Babel for translations, and the commands are nicelt wrapped up in our CLI:

* `odi trans init`
* `odi trans extract`
* `odi trans update`
* `odi trans compile`

## URL Aliases

If migrating from a previous Index site to the new static site, you'll need to implement aliases from some old URLs to the new ones. For all common URLs related to the Open Data Index, this is taken care of via the `alias` meta data per page. For the content sources that are **not** auto-generated (such as about, methodology, etc.), you can simply add a list of comma-separated URL paths to `alias`.

## Forms

### Download feedback form

The site features integration with Google Forms for feedback on the download page.

opendataindex

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README