KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #ai-agents #dsh-plugin #codex #developer-tools #claude-code #windows #desktop-app #typescript #rust #ai-agent #cli

SparkClean

★ 29 Open GitHub ↗

A Scalable Data Cleaning Library for PySpark.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# SparkClean: A Scalable DataClean Library for PySpark

![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg) 
[![996.icu](https://img.shields.io/badge/link-996.icu-red.svg)](https://996.icu)

---
## Requirements

- Python >= 3.4
- Apache Spark >= 2.2.0

---

## Usage

~~~shell
>>> git clone https://github.com/NYUBigDataProject/SparkClean.git
>>> cd SparkClean
>>> pip3 install -r requirements.txt
>>> pyspark
~~~

Then in the Pyspark Shell
~~~python
import sparkclean
~~~

---

## Architecture

1. **Transformer**: Transform DataFrame and Break linage
2. **Deduplicator**: Cluster and Remove Duplications
3. **OutlierDetector**: Find the outliers
4. **IntegrityConstraint Corrector**: Check Integrity Constraints

---

## Transformer

Transform and Break linage

1. Originally from [Optimus](https://github.com/ironmussa/Optimus). Fixed it for fitting Python3.4
2. Use checkpoints for breaking lineage
3. Provide lots of useful functions

---

## Transformer: Utilities and Break Linage

---

## Deduplicator

Clustering and Remove Duplications

1. Fingerprint clustering 
2. Locality-Sensitive Hashing clustering
3. Support all kinds of similarity  (Edit,Token,Sequence,Phonetic, by intergating "TextDistance") 
4. Column-wise Deduplication
5. Optimiezed transformatons

---

## Deduplicator: Clustering By Fingerprint

---

## Deduplicator: Clustering By Similarity

~~~
>>> # Nothing showed up, because the threshold is large 
>>> col,cluster = de.localitySensitiveHashing("country", 
            \blockSize=2, method = "levenshtein", threshold = 0.9)
>>> de.preview(col,cluster)

>>> # Find a cluster, because the threshold is smaller
>>> col,cluster = de.localitySensitiveHashing("country", 
            \blockSize=2, method = "levenshtein", threshold = 0.8)
>>> de.preview(col,cluster)
------------ Cluster 0 -------------------
colName: country| Item: America, Count:2
colName: country| Item: Amarica, Count:1
colName: country|  Will be changed to "America", takes 2/3

---

## Deduplicator: Record Matching

~~~
>>> col,cluster=de.recordMatching(matchColNames=["city","country"], 
                  \ fixColNames=["population"])
>>> de.previewReords(col,cluster)
------------ Cluster 0 -------------------
Record id: [8589934592, 17179869185, 25769803776]
colName: population| Item: 19795791, Count:2
colName: population| Item: 9795791, Count:1
colName: population| Change: Will be changed to "19795791", takes 2/3

---

## Deduplicator: Deduplication

---

## IntegrityConstraints Corrector

Check and Correct Integrity Constraints

1. Reasonably fix violations of functional dependencies using a greedy algorithm
2. Do an exploration on the interaction between two or more FDs (holistic data repairing)

---