cramjam - GitLite

KO | EN

gitlite — search

#javascript #python #llm #go #android #linux #docker #windows #kubernetes #vue #security #ios #openai #open-source

cramjam

(★ 125)

Your go-to for easy access to a plethora of compression algorithms, all neatly bundled in one simple installation.

File Explorer

Download Latest Version (.zip)

- config.toml
- - CI.yml
- FUNDING.yml
- - alice29.txt.bz2
  - asyoulik.txt.bz2
  - COPYING
  - dickens.bz2
  - fireworks.jpeg.bz2
  - geo.protodata.bz2
  - html.bz2
  - html_x_4.bz2
  - kppkn.gtb.bz2
  - lcet10.txt.bz2
  - Mark.Twain-Tom.Sawyer.txt.bz2
  - mozilla.bz2
  - mr.bz2
  - nci.bz2
  - ooffice.bz2
  - osdb.bz2
  - paper-100k.pdf.bz2
  - plrabn12.txt.bz2
  - reymont.bz2
  - samba.bz2
  - sao.bz2
  - urls.10K.bz2
  - webster.bz2
  - x-ray.bz2
  - xml.bz2
- README.md
- test_bench.py
- - lib.rs
- - index.test.ts
  - setup.ts
- .gitignore
- Cargo.lock
- Cargo.toml
- package.json
- README.md
- tsconfig.json
- vitest.config.ts
- index.md
- - __init__.py
  - __init__.pyi
  - blosc2.pyi
  - brotli.pyi
  - bzip2.pyi
  - deflate.pyi
  - experimental.pyi
  - gzip.pyi
  - ideflate.pyi
  - igzip.pyi
  - izlib.pyi
  - lz4.pyi
  - py.typed
  - snappy.pyi
  - xz.pyi
  - zlib.pyi
  - zstd.pyi
- blosc2.rs
- brotli.rs
- bzip2.rs
- deflate.rs
- exceptions.rs
- experimental.rs
- gzip.rs
- ideflate.rs
- igzip.rs
- io.rs
- izlib.rs
- lib.rs
- lz4.rs
- snappy.rs
- xz.rs
- zlib.rs
- zstd.rs
- __init__.py
- conftest.py
- test_blosc2.py
- test_buffer_view.py
- test_cramjam_pyodide.js
- test_integration.py
- test_rust_io.py
- test_typing_package.py
- test_variants.py
- test_public_api.py
.gitignore
benchmark-requirements.txt
build.rs
Cargo.lock
Cargo.toml
Cross.toml
LICENSE
Makefile
mkdocs.yml
package.json
pyproject.toml
README.md
rustfmt.toml
uv.lock

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# cramjam

---

## Project Status

Over the course of the last year or two I've lost a lot of interest in open-source work and thus, this project. I thought it was simply
new time limitations, but as those lifted, it became clear something has changed. In me, the environment...probably both.

This world of open-source, software engineering, deep thought of puzzles, creativity within constraints; it all use to make me happy.
It was an escape. Seeing how it's evolving, the changes in people, the difference in tragectory, disregard for consequences...It makes me sad now.

Anyway, I may touch this again or I may not.

---

[![CI](https://github.com/milesgranger/cramjam/actions/workflows/CI.yml/badge.svg)](https://github.com/milesgranger/cramjam/actions/workflows/CI.yml)
[![Downloads](https://pepy.tech/badge/cramjam/month)](https://pepy.tech/project/cramjam)
[![PyPI](https://img.shields.io/pypi/v/cramjam.svg)](https://pypi.org/project/cramjam)
[![Conda Version](https://img.shields.io/conda/vn/conda-forge/cramjam.svg)](https://anaconda.org/conda-forge/cramjam)
[![NPM Version](https://img.shields.io/npm/v/cramjam)](https://www.npmjs.com/package/cramjam)

[API Documentation](https://milesgranger.github.io/cramjam/cramjam.html)

### Install (Python)
```commandline
pip install --upgrade cramjam  # Requires no Python or system dependencies!
```

## Development

Install the locked development dependencies, build the extension, and run the
test suite:

```bash
make dev-setup
make test
```

Rust checks can be run separately with `cargo test`.

### Install (JavaScript / TypeScript)
```commandline
npm install cramjam
```

### CLI

A CLI interface is available as [`cramjam-cli`](https://github.com/cramjam/cramjam-cli)

### libcramjam

A Rust crate and C friendly library available at [libcramjam](https://github.com/cramjam/libcramjam)

---

Extremely thin and easy-to-install Python bindings to de/compression algorithms in Rust.
Allows for using algorithms such as Snappy, without any system or other python dependencies.

---

##### Benchmarks

Some basic benchmarks are available [in the benchmarks directory](./benchmarks/README.md)

---

Available algorithms:

- [X] Snappy&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`cramjam.snappy`
- [X] Brotli&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`cramjam.brotli`
- [X] Bzip2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`cramjam.bzip2`
- [X] Lz4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`cramjam.lz4`
- [X] Gzip&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`cramjam.gzip`
- [X] Zlib&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`cramjam.zlib`
- [X] Deflate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`cramjam.deflate`
- [X] ZSTD&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`cramjam.zstd`
- [X] XZ / LZMA&nbsp;&nbsp;`cramjam.xz`

Experimental (Requires build from source enabling each feature):

- [X] Blosc2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`cramjam.experimental.blosc2`
- [X] ISA-L backend  _(only on 64-bit targets)_
  - [X] igzip&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`cramjam.experimental.igzip`
  - [X] ideflate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`cramjam.experimental.ideflate`
  - [X] izlib&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`cramjam.experimental.izlib`

All available for use as:

```python
>>> import cramjam
>>> import numpy as np
>>> compressed = cramjam.snappy.compress(b"bytes here")
>>> decompressed = cramjam.snappy.decompress(compressed)
>>> decompressed
cramjam.Buffer(len=10)  # an object which implements the buffer protocol
>>> bytes(decompressed)
b"bytes here"
>>> np.frombuffer(decompressed, dtype=np.uint8)
array([ 98, 121, 116, 101, 115,  32, 104, 101, 114, 101], dtype=uint8)
```

Where the API is `cramjam.<compression-variant>.compress/decompress` and accepts 
`bytes`/`bytearray`/`numpy.array`/`cramjam.File`/`cramjam.Buffer` / `memoryview` objects.

**de/compress_into**
Additionally, all variants support `decompress_into` and `compress_into`. 
Ex.
```python
>>> import numpy as np
>>> from cramjam import snappy, Buffer
>>>
>>> data = np.frombuffer(b'some bytes here', dtype=np.uint8)
>>> data
array([115, 111, 109, 101,  32,  98, 121, 116, 101, 115,  32, 104, 101,
       114, 101], dtype=uint8)
>>>
>>> compressed = Buffer()
>>> snappy.compress_into(data, compressed)
33  # 33 bytes written to compressed buffer
>>>
>>> compressed.tell()  # Where is the buffer position?
33  # goodie!
>>>
>>> compressed.seek(0)  # Go back to the start of the buffer so we can prepare to decompress
>>> decompressed = b'0' * len(data)  # let's write to `bytes` as output
>>> decompressed
b'000000000000000'
>>>
>>> snappy.decompress_into(compressed, decompressed)
15  # 15 bytes written to decompressed
>>> decompressed
b'some bytes here'
```

[TypeScript](./cramjam-js/README.md):

```typescript

import {Compress, Decompress} from 'cramjam';

const decoder = new TextDecoder();
const encoder = new TextEncoder();

const str = 'hello, world';
const encoded = encoder.encode(str);

const compressed = Compress.brotli(encoded);
const decompressed = Decompress.brotli(compressed);

const decoded = decoder.decode(decompressed);

```