pdist

KO | EN

gitlite — search

#javascript #python #react #llm #machine-learning #rust #deep-learning #linux #windows #awesome #cli #kubernetes #openai #open-source

(★ 11)

Calculate mean of pairwise weighted distances between points using great circle metric.

pdist Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# pdist

[![Build Status](https://travis-ci.org/oliviaguest/pdist.svg?branch=master)](https://travis-ci.org/oliviaguest/pdist)
[![codecov](https://codecov.io/gh/oliviaguest/pdist/branch/master/graph/badge.svg)](https://codecov.io/gh/oliviaguest/pdist)
[![license](https://img.shields.io/badge/license-CC0-lightgrey.svg)](https://github.com/oliviaguest/pdist/blob/master/LICENSE)
<a href="https://twitter.com/intent/follow?screen_name=o_guest">
        <img src="https://img.shields.io/twitter/follow/o_guest.svg?style=social"
            alt="follow on Twitter"></a>

Calculate the mean of the pairwise weighted distances between points using the great circle metric :globe_with_meridians: for a very big dataset without running out of RAM :bomb: and/or waiting till the end of the universe. :joy:

## Usage
### Mean of Pairwise Weighted Distances Using Great Circle
This code takes a set of 2D data points ```X``` and calculates the mean of the pairwise weighted distances between points using the great circle metric.
It offers extensive speedup over Python-only implementations, so it is useful when dealing with _very big data_.

To call use:
``` python
mean_distances = c_mean_dist(X, weights)
```
where ```X``` are your data points, and ```weights``` are the weights or counts (depending on how you want to conceptualise them).
Weights affect the mean of the pairwise distances in the same as including more of the point which the weight corresponds to.
So if a data point with value (0, 1) has a weight of 2, the average pairwise distances will be affected in the same way as if you had added another data point with value (0, 1) to ```X``` and had set both their weights to 1.

### Great Circle Distance
It also implements great circle, also known as orthodromic or geodesic, distance metric faster than GeoPy in ```cdist```.

## Example
For an example of both functions see ```compare.py```.

## Information
The C and Python code were written by Olivia Guest — using [this tutorial]( https://equatorialmaths.wordpress.com/2010/10/16/python-extensions-with-c-libraries-made-easy-by-cython/) by Dmitrii V Pasechnik to call C functions from Python using Cython, and using the [haversine function](https://rosettacode.org/wiki/Haversine_formula#C) from Rosetta Code.

## Installation
Make sure you have Cython and its dependencies installed (refer to ```requirements.txt```).
Run ```make```.
Subsequently, run ```python compare.py``` to confirm compilation, and to see the comparison between using the C version and using a Python-only way.
See ```requirements.txt``` in case you need to install GeoPy, etc.

If you want to use this function from outside this directory, e.g., ```import```, I have not yet found a way of doing so without adding the path to the library to ```LD_LIBRARY_PATH```, e.g., ```export LD_LIBRARY_PATH=/local/path/to/this/repo```.
For adding it permanently (so you do not have to do this every time) add it to your ```~/.bashrc``` or whatever your set-up dictates.

## Notes
There were [many attenpts](https://github.com/oliviaguest/pairwise_distance) :dizzy_face: to make this work Python-only. :snake:
Alas — none of them worked out, but feel free to play around with the various Python versions.
The main stumbling block was the GIL. :cry:
For very huge data sometimes Python-only is not the best idea. :grimacing:

## To Do

* Fill in the ```setup.py``` (template: https://github.com/uwescience/shablona/blob/master/setup.py)
* Submit to pypi (https://github.com/oliviaguest/pdist/pull/2#issuecomment-339713987)

pdist

pdist Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

pdist

pdist Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README