KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #claude-code #codex #typescript #ai-agents #deepseek-harness #mcp #cli #rust

protocompile

★ 342 Open GitHub ↗

A parsing/linking engine for protobuf; the guts for a pure Go replacement of protoc.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

![The Buf logo](https://raw.githubusercontent.com/bufbuild/protocompile/main/.github/buf-logo.svg)

# Protocompile

[![Build](https://github.com/bufbuild/protocompile/actions/workflows/ci.yaml/badge.svg?branch=main)](https://github.com/bufbuild/protocompile/actions/workflows/ci.yaml)
[![Report Card](https://goreportcard.com/badge/github.com/bufbuild/protocompile)](https://goreportcard.com/report/github.com/bufbuild/protocompile)
[![GoDoc](https://pkg.go.dev/badge/github.com/bufbuild/protocompile.svg)](https://pkg.go.dev/github.com/bufbuild/protocompile)

This repo contains a parsing/linking engine for Protocol Buffers, written in pure Go. It is suitable as an alternative
to `protoc` (Google's official reference compiler for Protocol Buffers). This is the compiler that powers [Buf](https://buf.build)
and its bevy of tools.

This repo is also the spiritual successor to the [`github.com/jhump/protoreflect/desc/protoparse`](https://godoc.org/github.com/jhump/protoreflect/desc/protoparse)
package. If you are looking for a newer version of `protoparse` that natively works with the newer Protobuf runtime
API for Go (`google.golang.org/protobuf`), you have found it!

## Protocol Buffers

If you've come across this repo but don't know what Protocol Buffers are, you might acquaint yourself with the [official
documentation](https://developers.google.com/protocol-buffers). Protocol Buffers, or Protobuf for short, is an IDL for
describing APIs and data structures and also a binary encoding format for efficiently transmitting and storing that
data.

If you want to know more about the language itself, which is what this repo implements, take a look at Buf's
[Protobuf Guide](https://protobuf.com), which includes a very detailed language specification.

### Descriptors

Descriptors are the "lingua franca" for describing Protobuf data schemas. They are the basis of runtime features like
reflection and dynamic messages. They are also the output of a Protobuf compiler: a compiler can produce them and write
them to a file (whose contents are the binary-encoded form of a [`FileDescriptorSet`](https://github.com/protocolbuffers/protobuf/blob/v21.7/src/google/protobuf/descriptor.proto#L55-L59))
or send them to a [plugin](https://docs.buf.build/reference/images#plugins) to generate code for a particular
programming language.

Descriptors are similar to nodes in a syntax tree: the contents of a file descriptor correspond closely to the elements
in the source file from which it was generated. Also, the descriptor model's data structures are themselves defined in
[Protobuf](https://github.com/protocolbuffers/protobuf/blob/v21.7/src/google/protobuf/descriptor.proto).

## Using This Repo

The primary API of this repo is in this root package: `github.com/bufbuild/protocompile`. This is the suggested entry
point and provides a type named `Compiler`, for compiling Protobuf source files into descriptors. There are also
numerous sub-packages, most of which implement various stages of the compiler. Here's an overview (_not_ in alphabetical
order):

* [`protocompile`](https://pkg.go.dev/github.com/bufbuild/protocompile):
    This is the entry point, used to configure and initiate a compilation operation.
  * [`parser`](https://pkg.go.dev/github.com/bufbuild/protocompile/parser):
    This is the first stage of the compiler. It parses Protobuf source code and produces an AST. This package can also
    generate a file descriptor proto from an AST.
  * [`ast`](https://pkg.go.dev/github.com/bufbuild/protocompile/ast):
    This package models an Abstract Syntax Tree (AST) for the Protobuf language.
  * [`linker`](https://pkg.go.dev/github.com/bufbuild/protocompile/linker):
    This is the second stage of the compiler. The descriptor proto (generated from an AST) is linked, producing a more
    useful data structure than simple descriptor protos. This step also performs numerous validations on the source,
    like making sure that all type references are correct and that sources don't try to define two elements with the same
    name.
  * [`options`](https://pkg.go.dev/github.com/bufbuild/protocompile/options):
    This is the next stage of the compiler: interpreting options. The linked data structures that come from the previous
    stage are used to validate and interpret all options.
  * [`sourceinfo`](https://pkg.go.dev/github.com/bufbuild/protocompile/sourceinfo):
    This is the last stage of the compiler: generating source code info. Source code info contains metadata that maps
    elements in the descriptor to the location in the original source file from which it came. This includes access to
    comments. In order to provide correct source info for options, it must happen last, after options have been
    interpreted.
  * [`reporter`](https://pkg.go.dev/github.com/bufbuild/protocompile/reporter): This package provides error types
    generated by the compiler and interfaces used by the compiler to report errors and warnings to the calling code.
  * [`walk`](https://pkg.go.dev/github.com/bufbuild/protocompile/walk):
    This package provides functions for walking through all of the elements in a descriptor (or descriptor proto)
    hierarchy.
  * [`protoutil`](https://pkg.go.dev/github.com/bufbuild/protocompile/protoutil):
    This package contains some other useful functions for interacting with Protobuf descriptors.

### Migrating from `protoparse`

There are a few differences between this repo and its predecessor, `github.com/jhump/protoreflect/desc/protoparse`.

* If you want to include "standard imports", for the well-known files that are included with `protoc`, you have to do
  so explicitly. To do this, wrap your resolver using `protocompile.WithStandardImports`.
* If you used `protoparse.FileContentsFromMap`, in this new repo you'll use a `protocompile.SourceResolver` and then use
  `protocompile.SourceAccessorFromMap` as its accessor function.
* If you used `Parser.ParseToAST`, you won't use the `protocompile` package but instead directly use `parser.Parse` in
  this repo's `parser` sub-package. This returns an AST for the given file contents.
* If you used `Parser.ParseFilesButDoNotLink`, that is still possible in this repo, but not provided directly via a
  single function. Instead, you need to take a few steps:
  1. Parse the source using `parser.Parse`. Then use `parser.ResultFromAST` to construct a result that contains a file
     descriptor proto.
  2. Interpret whatever options can be interpreted without linking using `options.InterpretUnlinkedOptions`. This may
     leave some options in the descriptor proto uninterpreted (including all custom options).
  3. If you want source code info for the file, finally call `sourceinfo.GenerateSourceInfo` using the index returned
     from the previous step and store that in the file descriptor proto.

protocompile

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README