KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #typescript #dsh #dsh-plugin #ai-agent #codex #ai-agents #deepseek-harness #claude-code #macos #cli #react

cleanframes

★ 79 Open GitHub ↗

type-class based data cleansing library for Apache Spark SQL

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Clean Frames

[![Build Status](https://travis-ci.org/funkyminds/cleanframes.svg?branch=master)](https://travis-ci.org/funkyminds/cleanframes)

**Clean Frames** is a library for Apache Spark SQL module. It provides a type class for data cleansing.

## Getting Clean Frames

The current stable version is 0.3.0, which is cross built against Scala (2.11-2.12) and Apache Spark (2.1.0-2.4.3).

If you're using SBT, add the following line to your build file:

```scala
libraryDependencies += "io.funkyminds" %% "cleanframes" % "2.4.3_0.3.0"
```

or Maven dependency:

```xml
<dependency>
  <groupId>io.funkyminds</groupId>
  <artifactId>cleanframes_2.12</artifactId>
  <version>2.4.3_0.3.0</version>
</dependency>
```

## Quick Start

Assuming DataFrame is loaded from a csv file with following content:
```csv
1,true,1.0
lmfao,true,2.0
3,false,3.0
4,true,yolo data
5,true,5.0
```

and a domain model is defined as:

```scala
case class Example(col1: Option[Int], col2: Option[Boolean], col3: Option[Float])
```

library clean data to:

```scala
Example(Some(1),  Some(true),   Some(1.0f)),
Example(None,     Some(true),   Some(2.0f)),
Example(Some(3),  Some(false),  Some(3.0f)),
Example(Some(4),  Some(true),   None),
Example(Some(5),  Some(true),   Some(5.0f))
```

with a minimal code:

```scala
import cleanframes.instances.all._
import cleanframes.syntax._
  
frame
  .clean[Example]
```

### What is so different?

We would like to live in a world where data quality is superb but only unicorns are perfect.

Apache Spark by default discards entire row if it contains any invalid values.

Having called Spark for same data:

```scala
frame
  .as[Example]
```

would give a dataset with content:

```scala
Example(Some(1),  Some(true),   Some(1.0f)),
Example(None,     None,         None),
Example(Some(3),  Some(false),  Some(3.0f)),
Example(None,     None,         None),
Example(Some(5),  Some(true),   Some(5.0f))
```

As noticed, data in second and forth rows are lost due to particular malformed cells. Such behaviour might not be accepted in some domains.

### Pure Spark-SQL API

To save valid cells and discard only invalid ones, such Spark SQL API might be called:

```scala
val cleaned = frame.withColumn(
  "col1",
  when(
    not(
      frame.col("col1").isNaN
    ),
    frame.col("col1")
  ) cast IntegerType
).withColumn(
  "col2",
  when(
    trim(lower(frame.col("col2"))) === "true",
    lit(true)
  ).otherwise(false)
).withColumn(
  "col3",
  when(
    not(
      frame.col("col3").isNaN
    ),
    frame.col("col3")
  ) cast FloatType
)
```

### cleanframes

__cleanframes__ is a small library that does such boilerplate as above for you by calling:

```scala
frame
  .clean[CaseClass]
```

It resolves type-related transformations in a compile time using implicit resolutions in a type-safe way.

The library is shipped with common basic transformations and can be extended via custom ones.

There is no performance penalty, all code is generated by the compiler (currently by [shapeless](https://github.com/milessabin/shapeless)).

## Instructions

For further instructions, refer to:
* project's [Wiki](https://github.com/funkyminds/cleanframes/wiki) tab
* [cleanframes-examples](https://github.com/funkyminds/cleanframes-examples) project
* a bundle of unit tests with instructions how to use the library

## FAQ

* __Why minimal Spark version is 2.1.0 when Datasets where introduced in 1.6.0?__
  
   There is a problem with value classes support in versions 2.0.x, Spark throws runtime exception during its code generation. Spark in 1.6.x has a problem with testing library.

## Contributors

+ Dawid Rutowicz <dawid.rutowicz@gmail.com> [@dawrutowicz](https://twitter.com/dawrutowicz)

cleanframes

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README