canoto

KO | EN

gitlite — search

#hacktoberfest #react #llm #machine-learning #deep-learning #docker #java #windows #cli #awesome-list #pytorch #security #mcp #open-source

(★ 19)

Canonical Proto formatting in golang

canoto Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Canoto

Canoto is a serialization format designed to be:
1. Fast
2. Compact
3. Canonical
4. Backwards compatible
5. Read compatible with [Protocol Buffers](https://protobuf.dev/).

## Install

```sh
go install github.com/StephenButtolph/canoto/canoto@latest
```

## Define Messages

Canoto messages are defined as normal golang structs:
```golang
type ExampleStruct0 struct {
	Int32              int32           `canoto:"int,1"`
	Int64              int64           `canoto:"int,2"`
	Uint32             uint32          `canoto:"uint,3"`
	Uint64             uint64          `canoto:"uint,4"`
	Sfixed32           int32           `canoto:"fint32,5"`
	Fixed32            uint32          `canoto:"fint32,6"`
	Sfixed64           int64           `canoto:"fint64,7"`
	Fixed64            uint64          `canoto:"fint64,8"`
	Bool               bool            `canoto:"bool,9"`
	String             string          `canoto:"string,10"`
	Bytes              []byte          `canoto:"bytes,11"`
	OtherStruct        ExampleStruct1  `canoto:"value,12"`
	OtherStructPointer *ExampleStruct1 `canoto:"pointer,13"`

canotoData canotoData_ExampleStruct0
}

type ExampleStruct1 struct {
	Int32 int32 `canoto:"int,536870911"`

canotoData canotoData_ExampleStruct1
}
```

All structs must include a field called `canotoData` that will cache the results of calculating the size of the struct.

The type `canotoData_${structName}` is automatically generated by Canoto.

For a given `Struct`, Canoto automatically implements the `Message` interface:

```golang
// Message defines a type that can be a stand-alone Canoto message.
type Message interface {
	Field
	// MarshalCanoto returns the Canoto representation of this message.
	//
	// It is assumed that this message is ValidCanoto.
	MarshalCanoto() []byte
	// UnmarshalCanoto unmarshals a Canoto-encoded byte slice into the message.
	UnmarshalCanoto(bytes []byte) error
}

// Field defines a type that can be included inside of a Canoto message.
type Field interface {
	// CanotoSpec returns the specification of this canoto message.
	//
	// If there is not a valid specification of this type, it returns nil.
	CanotoSpec(types ...reflect.Type) *Spec
	// MarshalCanotoInto writes the field into a [Writer] and returns the
	// resulting [Writer].
	//
	// It is assumed that CalculateCanotoCache has been called since the last
	// modification to this field.
	//
	// It is assumed that this field is ValidCanoto.
	MarshalCanotoInto(w Writer) Writer
	// CalculateCanotoCache populates internal caches based on the current
	// values in the struct.
	CalculateCanotoCache()
	// CachedCanotoSize returns the previously calculated size of the Canoto
	// representation from CalculateCanotoCache.
	//
	// If CalculateCanotoCache has not yet been called, or the field has been
	// modified since the last call to CalculateCanotoCache, the returned size
	// may be incorrect.
	CachedCanotoSize() uint64
	// UnmarshalCanotoFrom populates the field from a [Reader].
	UnmarshalCanotoFrom(r Reader) error
	// ValidCanoto validates that the field can be correctly marshaled into the
	// Canoto format.
	ValidCanoto() bool
}
```

## Generate

In order to generate canoto information for all of the structs in a file, simply run the `canoto` command with one or more files.

```sh
canoto example0.go example1.go
```

The above example will generate `example0.canoto.go` and `example1.canoto.go`.

The corresponding `proto` file for a `canoto` file can also be generated by adding the `--proto` flag.

```sh
canoto --proto example.go
```

The above example will generate `example.canoto.go` and `example.proto`.

### go:generate

To automatically generate the `.canoto.go` version of a file, it is recommended to use `go:generate`. By using `go run` inside the `go:generate`, the version of the command being run will be taken from the local `go.mod`.

Place

```golang
//go:generate go run github.com/StephenButtolph/canoto/canoto $GOFILE
```

at the top of a file to update the `.canoto.go` version of the file every time `go generate ./...` is run.

### Best Practices

`canoto` only inspects a single golang file at a time, so it is recommended to define nested messages in the same file to be able to generate the most useful `proto` file.

Additionally, while fully supported in the `canoto` output, type aliases and generic types will result in `proto` files with default types. It is still guaranteed for the generated `proto` file to be able to parse `canoto` data, but the types may not be as specific as they could be.

If type aliases are needed, it may make sense to modify the generated proto file to specify the most specific proto type possible.

### Generics

Canoto supports generic structs through the `canoto.FieldPointer[T]` constraint. To guarantee safe usage of a struct, type constraints can be used to implement a struct with a generic field `T`. Canoto inspects the generic types, so the struct must include a type parameter of `canoto.FieldPointer[T]`. For example:

```golang
type GenericField[T any, _ canoto.FieldPointer[T]] struct {
	Value   T  `canoto:"value,1"`
	Pointer *T `canoto:"pointer,2"`

canotoData canotoData_GenericField
}
```

If `canoto.FieldPointer` is aliased to a different type or is otherwise re-implemented, Canoto will not be able to correctly tie the type constraints together.

### Pass-By-Value Messages

The auto-generated `canotoData` struct utilizes atomic reads and writes to ensure that `MarshalCanoto` can be called at the same time on multiple threads. Concurrent calls to `MarshalCanoto` are expected for what appears to be a read-only method.

However, concurrently marshalling and passing messages by value may cause race detection failures. To explicitly disallow passing messages by value, the `canoto:"nocopy"` tag can be added to the `canotoData` field. This results in the [NoCopy](https://github.com/golang/go/issues/8005) pattern being used to warn against incorrect usage.

For example:

```golang
type NotPassableByValue struct {
	Int int64 `canoto:"int,1"`

canotoData canotoData_NotPassableByValue `canoto:"nocopy"`
}
```

### Standalone Implementations

In some instances, it may be desirable for the generated code to avoid introducing the dependency on this repo into the `go.mod` file. As an example, if the user must support having multiple versions of canoto utilized in the same application.

There are two CLI flags that enable using canoto without impacting the `go.mod`.

1. `--library` when specified generates the canoto library in the provided folder. For example `--library="./internal"` generates the `canoto` library in the `./internal/canoto` package.
2. `--import` specifies the canoto library to depend on in any generated code.

For example:

```sh
canoto --library="./internal" --import="github.com/StephenButtolph/canoto/internal/canoto" ./canoto.go
```

Will generate the canoto library in `./internal/canoto` and will import `"github.com/StephenButtolph/canoto/internal/canoto"` rather than the default `"github.com/StephenButtolph/canoto"` when generating `./canoto.canoto.go`.

### Custom Identifiers

Six CLI flags control the naming of generated Go identifiers:

| Flag                   | Default                                | Scope   | Description                              |
|------------------------|----------------------------------------|---------|------------------------------------------|
| `--format-cache`       | `canotoData_{struct}`                  | Message | Name of the generated cache struct type  |
| `--format-number`      | `canotoNumber_{cStruct}__{cField}`     | Field   | Name of generated field number constants |
| `--format-tag`         | `canotoTag_{cStruct}__{cField}`        | Field   | Name of generated field tag constants    |
| `--format-oneof-type`  | `canotoOneOfType_{cStruct}__{cOneOf}`  | OneOf   | Name of generated oneOf type aliases     |
| `--format-oneof-unset` | `canotoOneOfUnset_{cStruct}__{cOneOf}` | OneOf   | Name of generated unset oneOf constants  |
| `--format-oneof-field` | `canotoOneOf_{cStruct}__{cField}`      | Field   | Name of generated oneOf field constants  |

Each flag accepts a template string. The available variables depend on the flag's scope, and each scope extends the one above it:

| Scope   | Variables                                                             |
|---------|-----------------------------------------------------------------------|
| Message | `{struct}`, `{cStruct}`                                               |
| OneOf   | `{struct}`, `{cStruct}`, `{oneOf}`, `{cOneOf}`                        |
| Field   | `{struct}`, `{cStruct}`, `{oneOf}`, `{cOneOf}`, `{field}`, `{cField}` |

| Variable    | Description                                                           |
|-------------|-----------------------------------------------------------------------|
| `{struct}`  | Original struct name (e.g. `My_Struct`)                               |
| `{cStruct}` | Canonicalized struct name: `_` replaced with `_1` (e.g. `My_1Struct`) |
| `{oneOf}`   | Original oneOf name                                                   |
| `{cOneOf}`  | Canonicalized oneOf name: `_` replaced with `_1`                      |
| `{field}`   | Original field name                                                   |
| `{cField}`  | Canonicalized field name: `_` replaced with `_1`                      |

The `_` → `_1` canonicalization prevents ambiguity with `__`, which can be used as a separator between variables.

For example:

```sh
canoto \
  --format-cache="cache_{struct}" \
  --format-number="fieldNum_{cStruct}__{cField}" \
  --format-tag="fieldTag_{cStruct}__{cField}" \
  --format-oneof-type="{struct}{oneOf}" \
  --format-oneof-unset="{struct}{oneOf}__Unset" \
  --format-oneof-field="{struct}{oneOf}__{field}" \
  example.go
```

For a struct `Foo` with a OneOf `Type` and a field `Bar`, this would generate:

```go
type cache_Foo struct { ... }

const (
    fieldNum_Foo__Bar = 1
    fieldTag_Foo__Bar = "\x0a"
)

type FooType uint32

const (
    FooType__Unset FooType = 0
    FooType__Bar   FooType = fieldNum_Foo__Bar
)
```

## Supported Types

| go type         | canoto type                  | proto type                 | wire type |
|-----------------|------------------------------|----------------------------|-----------|
| `int8`          | `int`                        | `sint32`                   | `varint`  |
| `int16`         | `int`                        | `sint32`                   | `varint`  |
| `int32`         | `int`                        | `sint32`                   | `varint`  |
| `int64`         | `int`                        | `sint64`                   | `varint`  |
| `uint8`         | `uint`                       | `uint32`                   | `varint`  |
| `uint16`        | `uint`                       | `uint32`                   | `varint`  |
| `uint32`        | `uint`                       | `uint32`                   | `varint`  |
| `uint64`        | `uint`                       | `uint64`                   | `varint`  |
| `int32`         | `fint32`                     | `sfixed32`                 | `i32`     |
| `uint32`        | `fint32`                     | `fixed32`                  | `i32`     |
| `int64`         | `fint64`                     | `sfixed64`                 | `i64`     |
| `uint64`        | `fint64`                     | `fixed64`                  | `i64`     |
| `bool`          | `bool`                       | `bool`                     | `varint`  |
| `string`        | `string`                     | `string`                   | `len`     |
| `[]byte`        | `bytes`                      | `bytes`                    | `len`     |
| `[x]byte`       | `fixed bytes`                | `bytes`                    | `len`     |
| `T Message`     | `value`                      | `message`                  | `len`     |
| `*T Message`    | `pointer`                    | `message`                  | `len`     |
| `[]int8`        | `repeated int`               | `repeated sint32`          | `len`     |
| `[]int16`       | `repeated int`               | `repeated sint32`          | `len`     |
| `[]int32`       | `repeated int`               | `repeated sint32`          | `len`     |
| `[]int64`       | `repeated int`               | `repeated sint64`          | `len`     |
| `[]uint8`       | `repeated uint`              | `repeated uint32`          | `len`     |
| `[]uint16`      | `repeated uint`              | `repeated uint32`          | `len`     |
| `[]uint32`      | `repeated uint`              | `repeated uint32`          | `len`     |
| `[]uint64`      | `repeated uint`              | `repeated uint64`          | `len`     |
| `[]int32`       | `repeated fint32`            | `repeated sfixed32`        | `len`     |
| `[]uint32`      | `repeated fint32`            | `repeated fixed32`         | `len`     |
| `[]int64`       | `repeated fint64`            | `repeated sfixed64`        | `len`     |
| `[]uint64`      | `repeated fint64`            | `repeated fixed64`         | `len`     |
| `[]bool`        | `repeated bool`              | `repeated bool`            | `len`     |
| `[]string`      | `repeated string`            | `repeated string`          | `len`     |
| `[][]byte`      | `repeated bytes`             | `repeated bytes`           | `len`     |
| `[][x]byte`     | `repeated fixed bytes`       | `repeated bytes`           | `len`     |
| `[]T Message`   | `repeated value`             | `repeated message`         | `len`     |
| `[]*T Message`  | `repeated pointer`           | `repeated message_pointer` | `len`     |
| `[x]int8`       | `fixed repeated int`         | `repeated sint32`          | `len`     |
| `[x]int16`      | `fixed repeated int`         | `repeated sint32`          | `len`     |
| `[x]int32`      | `fixed repeated int`         | `repeated sint32`          | `len`     |
| `[x]int64`      | `fixed repeated int`         | `repeated sint64`          | `len`     |
| `[x]uint8`      | `fixed repeated uint`        | `repeated uint32`          | `len`     |
| `[x]uint16`     | `fixed repeated uint`        | `repeated uint32`          | `len`     |
| `[x]uint32`     | `fixed repeated uint`        | `repeated uint32`          | `len`     |
| `[x]uint64`     | `fixed repeated uint`        | `repeated uint64`          | `len`     |
| `[x]int32`      | `fixed repeated fint32`      | `repeated sfixed32`        | `len`     |
| `[x]uint32`     | `fixed repeated fint32`      | `repeated fixed32`         | `len`     |
| `[x]int64`      | `fixed repeated fint64`      | `repeated sfixed64`        | `len`     |
| `[x]uint64`     | `fixed repeated fint64`      | `repeated fixed64`         | `len`     |
| `[x]bool`       | `fixed repeated bool`        | `repeated bool`            | `len`     |
| `[x]string`     | `fixed repeated string`      | `repeated string`          | `len`     |
| `[x][]byte`     | `fixed repeated bytes`       | `repeated bytes`           | `len`     |
| `[x][y]byte`    | `fixed repeated fixed bytes` | `repeated bytes`           | `len`     |
| `[x]T Message`  | `fixed repeated value`       | `repeated message`         | `len`     |
| `[x]*T Message` | `fixed repeated pointer`     | `repeated message_pointer` | `len`     |

### OneOf Fields

OneOfs allow message definitions to declare that a set of fields are mutually exclusive. When serialized, Canoto ensures that at most one field in each OneOf group appears on the wire.

Each OneOf has a group name, specified as an additional option after the field number in the struct tag:
```golang
type OneOf struct {
	Int  int64 `canoto:"int,1,Type"`
	Bool bool  `canoto:"bool,2,Type"`

canotoData canotoData_OneOf
}
```

For every OneOf group, the generated code includes a helper method that lets you quickly determine which field was populated.

In the example above, the method `CachedWhichOneOfType` will be generated, and it returns a generated typed enum for the `Type` OneOf.

All OneOf accessor methods follow the naming pattern:
```
CachedWhichOneOf<GroupName>
```

By default, that enum uses collision-resistant `canoto...` identifiers. If you want the enum type and its constants exported, customize `--format-oneof-type`, `--format-oneof-unset`, and `--format-oneof-field`.

The enum's underlying values still match the generated field number constants, so exported number constants can be used in comparisons.

After the cache has been initialized by calling one of `UnmarshalCanoto`, `UnmarshalCanotoFrom`, or `CalculateCanotoCache`, the method returns the populated field's enum value.
If no field in the OneOf group was set, the method returns `0`.

### Non-standard encoding

It is valid to define a `Field` that implements a non-standard format. However, this format should still be canonical and the corresponding Proto file should report opaque bytes.

## Why not Proto?

Proto is a fast, compact encoding format with extensive language support. However, [Proto is not canonical](https://protobuf.dev/programming-guides/serialization-not-canonical/).

Proto is designed to be forwards-compatible. Almost by definition, a forwards-compatible serialization format cannot be canonical. The format of a field cannot be validated to be canonical if the expected type of the field is not known during decoding.

## Why is being canonical important?

In some cases, non-canonical serialization formats are subtle to work with.

For example, if the hash of the serialized data is important or if the serialized data is cryptographically signed.

In order to ensure that the hash of the serialized data does not change, it is important to carefully avoid re-serializing a message that was previously serialized.

For canonical serialization formats, the hash of the serialized data is guaranteed never to change. Every correct implementation of the format will produce the same hash.

## Why be read compatible with Proto?

By being read compatible with Proto, users of the Canoto format inherit some of Proto's cross-language support.

If an application only needs to read Canoto messages, but not write them, it can simply treat the Canoto message as a Proto message.

## Is Canoto Fast?

Canoto is typically more performant for both serialization and deserialization than Proto. However, Proto does not typically validate that fields are canonical. If a field is expensive to inspect, it's possible Canoto can be slightly slower.

Canoto is optimized to perform no unnecessary memory allocations, so careful management to ensure messages are stack allocated can significantly improve performance over Proto.

## Is Canoto Forwards Compatible?

No. Canoto chooses to be a canonical serialization format rather than being forwards compatible.

canoto

canoto Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

canoto

canoto Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README