KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agent #developer-tools #codex #typescript #cli #deepseek-harness #claude-code #ai-agents #mcp #rust

byte-streams

★ 430 Open GitHub ↗

A Rosetta stone for JVM byte representations

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![Clojars Project](https://img.shields.io/clojars/v/org.clj-commons/byte-streams.svg)](https://clojars.org/org.clj-commons/byte-streams)
[![cljdoc badge](https://cljdoc.org/badge/org.clj-commons/byte-streams)](https://cljdoc.org/d/org.clj-commons/byte-streams)
[![CircleCI](https://circleci.com/gh/clj-commons/byte-streams.svg?style=svg)](https://circleci.com/gh/clj-commons/byte-streams)

![](https://raw.githubusercontent.com/clj-commons/byte-streams/master/docs/header.jpg)

Java has a lot of different ways to represent a stream of bytes.  Depending on the author and age of a library, it might use `byte[]`, `InputStream`, `ByteBuffer`, or `ReadableByteChannel`.  If the bytes represent strings, there's also `String`, `Reader`, and `CharSequence` to worry about.  Remembering how to convert between all of them is a thankless task, made that much worse by libraries which define their own custom representations, or composing them with Clojure's lazy sequences and stream representations.

This library is a Rosetta stone for all the byte representations Java has to offer, and gives you the freedom to forget all the APIs you never wanted to know in the first place.  Complete documentation can be found [here](https://cljdoc.org/d/org.clj-commons/byte-streams).

### Usage

Leiningen:
```clojure
[org.clj-commons/byte-streams "0.3.4"]
```

deps.edn:
```clojure
org.clj-commons/byte-streams {:mvn/version "0.3.4"}
```

### Converting types

To convert one byte representation to another, use `convert`:

```clojure
clj-commons.byte-streams> (convert "abcd" java.nio.ByteBuffer)
#<HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=4 cap=4]>
clj-commons.byte-streams> (convert *1 String)
"abcd"
```

`(convert data to-type options?)` converts, if possible, the data from its current type to the destination type.  This destination type can either be a Java class or a Clojure protocol.  However, since there's no direct route from a string to a byte-buffer, under the covers `convert` is doing whatever it takes to get the desired type:

```clojure
clj-commons.byte-streams> (conversion-path String java.nio.ByteBuffer)
([java.lang.String [B]
 [[B java.nio.ByteBuffer])
```

While we can't turn a string into a `ByteBuffer`, we can turn a string into a `byte[]`, and `byte[]` into a `ByteBuffer`.  When invoked, `convert` will choose the minimal path along the graph of available conversions.  Common conversions are exposed via `to-byte-buffer`, `to-byte-buffers`, `to-byte-array`, `to-input-stream`, `to-readable-channel`, `to-char-sequence`, `to-string`, and `to-line-seq`.

Every type can exist either by itself, or as a sequence.  For instance, we can create an `InputStream` representing an infinite number of repeated strings:

```clojure
clj-commons.byte-streams> (to-input-stream (repeat "hello"))
#<InputStream byte_streams.InputStream@3962a02c>
```

And then we can turn that into a lazy sequence of `ByteBuffers`:

```clojure
clj-commons.byte-streams> 
  (take 2 (convert *1
                   (seq-of java.nio.ByteBuffer)
                   {:chunk-size 128}))
(#<HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=128 cap=128]>
 #<HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=128 cap=128]>)
```

Notice that we describe a sequence of a type as `(seq-of type)`, and that we've passed a map to `convert` describing the size of the `ByteBuffers` we want to create.  Available options include:

* `:chunk-size` - the size in bytes of each chunk when converting a stream into a lazy seq of discrete chunks, defaults to `4096`
* `:direct?` - whether any `ByteBuffers` which are created should be [direct](http://stackoverflow.com/a/5671880/228387), defaults to `false`
* `:encoding` - the character set for any strings we're encoding or decoding, defaults to `"UTF-8"`

To create a [Manifold stream](https://github.com/clj-commons/manifold/), use `(stream-of type)`.  To convert a core.async channel, convert it using `manifold.stream/->source`.

### Custom conversions

While there are conversions defined for all common byte types, this can be extended to other libraries via `def-conversion`:

```clojure
;; a conversion from byte-buffers to my-byte-representation
(def-conversion [ByteBuffer MyByteRepresentation]
  [buf options]
  (buffer->my-representation buf options))

;; everything that can be converted to a ByteBuffer is transitively fair game now
(convert "abc" MyByteRepresentation)
```

This mechanism can even be used for types unrelated to byte streams, if you're feeling adventurous.

### Transfers

Simple conversions are useful, but sometimes we'll need to do more than just keep the bytes in memory.  When you need to write bytes to a file, network socket, or other endpoints, you can use `transfer`.

```clojure
clj-commons.byte-streams> (def f (File. "/tmp/salutations"))
#'clj-commons.byte-streams/f
clj-commons.byte-streams> (transfer "hello" f {:append? false})
nil
clj-commons.byte-streams> (to-string f)
"hello"
```

`(transfer source sink options?)` allows you pipe anything that can produce bytes into anything that can receive bytes, using the most efficient mechanism available.  Custom transfer mechanisms can also be defined:

```clojure
(def-transfer [InputStream MyByteSink]
  [stream sink options]
  (send-stream-to-my-sink stream sink))
```

### Some utilities

`print-bytes` will print both hexadecimal and ascii representations of a collection of bytes:

```clojure
clj-commons.byte-streams> (print-bytes (-> #'print-bytes meta :doc))
50 72 69 6E 74 73 20 6F  75 74 20 74 68 65 20 62      Prints out the b
79 74 65 73 20 69 6E 20  62 6F 74 68 20 68 65 78      ytes in both hex
20 61 6E 64 20 41 53 43  49 49 20 72 65 70 72 65       and ASCII repre
73 65 6E 74 61 74 69 6F  6E 73 2C 20 31 36 20 62      sentations, 16 b
79 74 65 73 20 70 65 72  20 6C 69 6E 65 2E            ytes per line.
```

`(compare-bytes a b)` will return a value which is positive if `a` is lexicographically first, zero if they're equal, and negative otherwise:

```clojure
clj-commons.byte-streams> (compare-bytes "abc" "abd")
-1
```

`bytes=` will return true if two byte streams are equal, and false otherwise.

`conversion-path` returns all the intermediate steps in transforming one type to another, if one exists:

```clojure
;; each element is a conversion pair of to/from
clj-commons.byte-streams> (conversion-path java.io.File String)
([java.io.File java.nio.channels.ReadableByteChannel]
 [#'byte-streams/ByteSource java.lang.CharSequence]
 [java.lang.CharSequence java.lang.String])

;; but if a conversion is impossible...
clj-commons.byte-streams> (conversion-path java.io.OutputStream java.io.InputStream)
nil
```

`possible-conversions` returns a list of possible conversion targets for a type.

```clojure
clj-commons.byte-streams> (possible-conversions String)
(java.lang.String java.io.InputStream java.nio.DirectByteBuffer java.nio.ByteBuffer (seq-of java.nio.ByteBuffer) java.io.Reader java.nio.channels.ReadableByteChannel [B java.lang.CharSequence)
```

`byte-streams/optimized-transfer?` returns true if there is an optimized transfer method for two types:

```clojure
clj-commons.byte-streams> (optimized-transfer? String java.io.File)
true
```

### Deprecation notice

There exists an older, top-level namespace called just `byte-streams` that has been deprecated. Top-level namespaces can cause problems, particularly with Graal. Please switch to `clj-commons.byte-streams` in all your requires.

### License

Distributed under the [MIT License](http://opensource.org/licenses/MIT)

byte-streams

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README