KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #deepseek-harness #typescript #ai-agent #codex #developer-tools #claude-code #windows #mcp #python #desktop-app

sauerkraut

★ 74 Open GitHub ↗

A reimagined scala-pickling in the Scala 3 world

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Sauerkraut

The library for those cabbage lovers out there who want
to send data over the wire.

A revitalization of [Pickling](http://github.com/scala/pickling) in the
[Scala 3](https://dotty.epfl.ch/docs/index.html) world.

## Usage

When defining over-the-wire messages, do this:

```scala
import sauerkraut.core.{Buildable,Writer,given}
case class MyMessage(field: String, data: Int)
  derives Buildable, Writer
```

Then, when you need to serialize, pick a format and go:

```scala
import format.json.{Json,given}
import sauerkraut.{pickle,read,write}

val out = StringWriter()
pickle(Json).to(out).write(MyMessage("test", 1))
println(out.toString())

val msg = pickle(Json).from(out.toString()).read[MyMessage]
```

# Current Formats

Here's a feature matrix for each format:

| Format | Reader | Writer | All Types | Evolution Friendly | Notes                                    |
| ------ | ------ | ------ | --------- | ------------------ | ---------------------------------------- |
| Json   | Yes    | Yes    | Yes       | Yes                | Uses Jawn for parsing                    |
| Protos | Yes    | Yes    | Yes       | Yes                | Binary format evolution friendly format  |
| NBT    | Yes    | Yes    | Yes       |                    | For the kids.                            |
| XML    | Yes    | Yes    | Yes       |                    | Inefficient prototype.                   |
| Pretty | No     | Yes    | No        |                    | For pretty-printing strings              |

See [Compliance](compliance/README.md) for more details on what this means.

## Json
Everyone's favorite non-YAML web data transfer format!   This uses Jawn under the covers for parsing, but
can write Json without any dependencies.

Example:
```scala
import sauerkraut.{pickle,read,write}
import sauerkraut.core.{Buildable,Writer, given}
import sauerkraut.format.json.Json

case class MyWebData(value: Int, someStuff: Array[String])
    derives Buildable, Writer

def read(in: java.io.InputStream): MyWebData =
  pickle(Json).from(in).read[MyWebData]
def write(out: java.io.OutputStream): Unit = 
  pickle(Json).to(out).write(MyWebData(1214, Array("this", "is", "a", "test")))
```

sbt build:
```scala
libraryDependencies += "com.jsuereth.sauerkraut" %% "json" % "<version>"
```

See [json project](json/README.md) for more information.

## Protos
A new encoding for protocol buffers within Scala!  This supports a subset of all possible protocol buffer messages
but allows full definition of the message format within your Scala code.

Example:
```scala
import sauerkraut.{pickle,write,read, Field}
import sauerkraut.core.{Writer, Buildable, given}
import sauerkraut.format.pb.{Proto,,given}

case class MyMessageData(value: Int @Field(3), someStuff: Array[String] @Field(2))
    derives Writer, Buildable

def write(out: java.io.OutputStream): Unit = 
  pickle(Proto).to(out).write(MyMessageData(1214, Array("this", "is", "a", "test")))
```

This example serializes to the equivalent of the following protocol buffer message:

```proto
message MyMessageData {
  int32 value = 3;
  repeated string someStuff = 2;
}
```

sbt build:
```scala
libraryDependencies += "com.jsuereth.sauerkraut" %% "pb" % "<version>"
```

See [pb project](pb/README.md) for more information.

# NBT
Named-Binary-Tags, a format popularized by Minecraft.

Example:
```scala
import sauerkraut.{pickle,read,write}
import sauerkraut.core.{Buildable,Writer, given}
import sauerkraut.format.nbt.Nbt

case class MyGameData(value: Int, someStuff: Array[String])
    derives Buildable, Writer

def read(in: java.io.InputStream): MyGameData =
  pickle(Nbt).from(in).read[MyGameData]
def write(out: java.io.OutputStream): Unit = 
  pickle(Nbt).to(out).write(MyGameData(1214, Array("this", "is", "a", "test")))
```

sbt build:
```scala
libraryDependencies += "com.jsuereth.sauerkraut" %% "nbt" % "<version>"
```

See [nbt project](nbt/README.md) for more information.

# XML
Everyone's favorite markup language for data transfer!

Example:
```scala
import sauerkraut.{pickle,read,write}
import sauerkraut.core.{Buildable,Writer, given}
import sauerkraut.format.xml.{Xml, given}

case class MySlowWebData(value: Int, someStuff: Array[String])
    derives Buildable, Writer

def read(in: java.io.InputStream): MySlowWebData =
  pickle(Xml).from(in).read[MySlowWebData]
def write(out: java.io.Writer): Unit = 
  pickle(Xml).to(out).write(MySlowWebData(1214, Array("this", "is", "a", "test")))
```

sbt build:
```scala
libraryDependencies += "com.jsuereth.sauerkraut" %% "xml" % "<version>"
```

See [xml project](xml/README.md) for more information.

# Pretty
A format that is solely used to pretty-print object contents to strings.  This does not have
a [PickleReader] only a [PickleWriter].

Example:
```scala
import sauerkraut._, sauerkraut.core.{Writer,given}
case class MyAwesomeData(theBest: Int, theCoolest: String) derives Writer

scala> MyAwesomeData(1, "The Greatest").prettyPrint
val res0: String = Struct(rs$line$2.MyAwesomeData) {
  theBest: 1
  theCoolest: The Greatest
}
```

# Design

We split Serialization into three layers:

1. The `source` layer.  It is expected these are some kind of stream.
2. The `Format` layer.  This is responsible for reading a raw source and converting into
   the component types used in the `Shape` layer.  See `PickleReader` and `PickleWriter`.
3. The `Shape` layer.  This is responsible for turning Primitives, Structs, Choices and Collections
   into component types.

It's the circle of data:
```
   Source   =>     format    =>  shape => memory =>  shape  =>   format    =>   Destination

[PickleData] => PickleReader => Builder[T] => T => Writer[T] => PickleWriter => [PickleData]
```

This, hopefully, means we can reuse a lot of logic betwen various formats with light loss to efficiency.

*Note:  This library is not measuring performance yet.*

### Shape layer
The Shape layer is responsible for extracting Scala types into known shapes that can be used for
serialization.  These shapes, current, are `Collection`, `Structure` and `Primitive`.   Custom
shapes can be created in terms of these three shapes.

The Shape layer defines these three classes:
- `sauerkraut.core.Writer[T]`:
  Can translate a value into write* calls of Primitive, Structure or Collection.
- `sauerkraut.core.Builder[T]`:  
  Can accept an incomiing stream of collections/structures/primitives and build a value of T from them.
- `sauerkraut.core.Buildable[T]`:
  Can provide a `Builder[T]` when asked.

### Format layer
The format layer is responsible for mapping sauerkraut shapes (`Collection`, `Structure`, `Primitive`, `Choice`) into
the underlying format.  Not all shapes in sauerkraut will map exactly to underlying formats, and so each
format may need to adjust/tweak incoming data as appropriate.

The format layer has these primary classes:

- `sauerkraut.format.PickleReader`:  Can load data and push it into a Builder of type T
- `sauerkraut.format.PickleWriter`:  Accepts pushed structures/collections/primitives and places it into a Pickle

### Source Layer
The `source` layer is allowed to be any type that a format wishes to support.   Inputs and outputs are
provided to the API via these two classes:

- `sauerkraut.format.PickleReaderSupport[Input, Format]`:
  A given of this instance will allow the `PickleReader` to be constructed from a type of input.
- `sauerkraut.format.PickleWriterSupport[Output,Format]`:
  A given of this instance will allow `PickleWriter` to be constructed from a type of output.

This layer is designed to support any type of input and output, not just an in-memory store (like a Json Ast) or
a streaming input.  Formats can define what types of input/output (or execution environment) they allow.

## Writing a new format.

New formats are expected to provide the "format" + "source" layer implementations they require.

TODO - a bit more here.

# Differences from Scala Pickling

There are a few major differences from the old [scala pickling project](http://github.com/scala/pickling).

- The core library is built for 100% static code generation.   While we think that dynamic (i.e. runtime-reflection-based)
  pickling could be built using this library, it is a non-goal.
  - Users are expected to rely on typeclass derivation to generate Reader/Writers, rather than using macros
  - The supported types that can be pickled are limited to the same supported by typeclass derivation or that
    can have hand-written `Writer[_]`/`Builder[_]` instances.
- Readers are no longer driven by the Scala type.  Instead we use a new `Buildable[A]`/`Builder[A}` design
  to allow each `PickleReader` to push value into a `Builder[A]` that will then construct the scala class.
- There have been no runtime performance optimisations around codegen.   Those will come as we test the
  limits of Scala 3 / Dotty.
- Format implementations are separate libraries.
- The `PickleWriter` contract has been split into several types to avoid misuse.  This places a heavier amount
  of lambdas in play, but may be offsite with optimisations in modern versions of Scala/JVM.
- The name is more German.

# Benchmarking

Benchmarking is still being built-out, and is pending the final design on Choice/Sum-Types within the Format/Shape layer.

You can see benchmark results via: ` benchmarks/jmh:run -rf csv`.

Latest status/analysis can be found in the [benchmarks directory](benchmarks/latest-results.md).

## Benchmarking TODOs

- [X] Basic comparison of all formats
- [X] Size-of-Pickle measurement
- [ ] Well-thought out dataset for reading/writing
- [X] Isolated read vs. write testing
- [ ] Comparison against other frameworks.
  - [X] Protos vs. protocol buffer java implementation
  - [ ] Json Reading vs. raw JAWN to AST (measure overhead)
  - [ ] Jackson
  - [X] Kryo
  - [ ] Thrift
  - [ ] Circe
  - [X] uPickle
- [ ] Automatic well-formatted graph dump in Markdown of results.

# Thanks

Thanks to everyone who contributed to the original pickling library for inspiration, with a few callouts.

- Heather Miller + Philipp Haller for the original idea, innovation and motivation for Scala.
- Havoc Pennington + Eugene Yokota for helping define what's important when pickling a protocol and evolving that protocol.

sauerkraut

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README