KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agents #developer-tools #typescript #codex #claude-code #mcp #rust #deepseek-harness #ai-agent #cli

pyroenc

★ 46 Open GitHub ↗

A simple library to encode video in real-time with Vulkan video

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# PyroEnc

PyroEnc is a simple and standalone library for encoding video with Vulkan video,
suitable for direct integration in a Vulkan application.
It should also be useful as a sample of how to use Vulkan video encoding.

## Partnership

Development of this Vulkan video encoding library has been made in partnership with
[3dverse technologies inc](https://3dverse.com/).

## Device support

As of February 2025, pyroenc has been tested to work on NVIDIA and RADV.
RADV requires mesa-git as some important driver bugs have been fixed that affects pyroenc.
Eventually, these fixes should end up in a stable release.

The goal is to support every correct Vulkan video implementation.
PyroEnc is actively tested against Vulkan validation layers.

## Features

Currently, H.264 encode with a simple I and P GOP structure is supported.
H.265 is also supported for 8 and 10-bit (non-HDR) encode.
The end goal is to support:

- H.264
- H.265 (8-bit / 10-bit / HDR)
- AV1

The focus of this library is real-time, low-latency streaming, suitable for game streaming.
This focus may extend to high-quality, high-latency encoding as well should
the need arise.

### Currently supported

- Take RGB as input (YCbCr conversion happens on GPU)
  - The default MPEG style narrow range, left chroma, BT.709 is used in this path.
- Get direct access to underlying YCbCr encode image for custom conversion.
  - Supports passing down custom VUI information which is needed for HDR, full range, center chroma, etc ...
  - Could be more efficient when you're already scaling anyway.
- Adjust rate control dynamically
- Insert IDR frames on-demand
- Intra-refresh (not well-tested yet)

## API

NOTE: The API is not frozen yet and will change as the implementation matures.

### Creating Encoder

To create an encoder, various information must be provided, such as:

- VkInstance
  - Must enable Vulkan 1.3 or above in apiVersion
- VkPhysicalDevice
- VkDevice
  - Must support Vulkan 1.3 and have `VK_KHR_push_descriptor` extension enabled, or Vulkan 1.4 with pushDescriptor feature enabled
  - Must have video extensions and queues enabled 
    - Potentially, PyroEnc can provide an interface to help doing this, but currently does not
- vkGetInstanceProcAddr callback
- VkQueue + queue family index for conversion queue and encode queue
  - Conversion queue can be either graphics or compute. PyroEnc only uses compute shaders
- Video encoding parameters
  - Width
  - Height
  - Encoding profile
  - Frame rate
  - Tuning parameters
  - Quality level. 0.0f maps to qualityLevel = 0 and 1.0f maps to maxQualityLevels - 1.

```
#include "pyroenc.hpp"
using namespace PyroEnc;

Encoder encoder;
EncoderCreateInfo info = {};
encoder.init_encoder(info);
```

### Updating rate control

This can be called at any time and takes effect from next `send_frame`.
VBR, CBR and ConstantQP mode are supported which directly maps to Vulkan video rate control.
It should be called once before the first frame is sent to encoder.

```
RateControlInfo info = {};
encoder.set_rate_control_info(info);
```

### Sending frame to be encoded

Currently, only RGB input is supported.

```
FrameInfo info = {};
info.view = imageView; // Must be UNORM format with usage VK_IMAGE_USAGE_SAMPLED_BIT.
info.width = width; // Must match the encoder. Scaling is currently not supported.
info.height = height; // Must match the encoder. Scaling is currently not supported.
info.pts = pts++; // Not semantically important to PyroEnc, but it is passed back in EncodedFrame.
info.force_idr = ...; // If true, forces IDR frame to be generated, restarting the GOP.
encoder.send_frame(info);
```

`Encoder::send_frame()` will call `vkQueueSubmit2`, and the application must ensure
that `Encoder::send_frame()` is not called concurrently with any other uses
of either the conversion VkQueue or encoding VkQueue. It is invalid to
concurrently submit to the same VkQueue in Vulkan unless `VK_KHR_internally_synchronized_queues` is used.

For synchronization, the image view's memory must be visible and owned by the queue.
Generally, this will be `VK_PIPELINE_STAGE_2_COMPUTE_SHADER_BIT` and `VK_ACCESS_2_SHADER_SAMPLED_READ_BIT`,
with image layout `VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL`. `Encoder::get_conversion_dst_stage()` and friends
can be called to determine this. For example, if you were rendering to the image,
and then want to send it to encode.

```
VkImageMemoryBarrier2 barrier = { VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2 };
barrier.oldLayout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL;
barrier.srcStageMask = VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT;
barrier.srcAccessMask = VK_ACCESS_COLOR_ATTACHMENT_WRITE_BIT;
barrier.newLayout = encoder.get_conversion_image_layout();
barrier.dstStageMask = encoder.get_conversion_dst_stage();
barrier.dstAccessMask = encoder.get_conversion_dst_access();
// Fill in the rest
```

To avoid a write-after-read hazard, wait for `encoder.get_conversion_stage_stage()` to complete
execution. The only operation done on the image is to convert color space to YCbCr, so
it should complete very quickly and not stall the graphics queue for long.

### Receiving frame

After sending a frame for encode, you can receive 0 or more frames.
In the current implementation without B-frames, there is a 1:1 correspondence between
sending a frame and receiving it. This API style is similar to FFmpeg.

```
EncodedFrame frame;
while (encoder.receive_encoded_frame(frame) == Result::Success)
{
    // The API is async. Wait for frame to complete encode before we can grab bitstream.
    frame.wait();
    if (frame.get_status() == VK_QUERY_RESULT_STATUS_COMPLETE_KHR)
    {
        // For IDR frames, make sure we write out SPS/PPS as well.
        // To avoid additional memory allocations to pack the payloads together,
        // let application handle it.
        if (frame.is_idr())
            write_packet(encoder.get_encoded_parameters(), encoder.get_encoded_parameters_size());
        write_packet(frame.get_pts(), frame.get_dts(), frame.get_payload(), frame.get_size());
    }
}
```

`EncodedFrame` holds ownership of an entry in the encoder frame pool.
If there are too many `EncodedFrame` objects in flight when calling `send_frame()`,
it may return `Result::NotReady`.
Generally, there is little need to go beyond double-buffering the encoded frame.
All `EncodedFrame` objects must be destroyed before destroying the parent `Encoder`.

## Building

The project is just a simple single C++ file and single header.
The intended use is a static library.

```
add_subdirectory(pyroenc)
target_link_libraries(my-project PRIVATE pyroenc)
```

Alternatively, you can just add `pyroenc.cpp` to your build system.
The only include path required is Vulkan headers.

## Sample

As a sample demonstrating how to use the API, see the `test/` folder.
The samples are written using Granite to avoid a ton of boilerplate.
Run `checkout_granite.sh` from top folder before invoking CMake. In this case, tests are enabled.
This checkout is not done automatically to avoid bloating the simple nature of this library.

```
# Decode 10 seconds of 1080p test video as raw RGBA
ffmpeg -i mytest.mkv -vf scale=1920:1080 -an -t 10 -pix_fmt rgba test.rgb

# Encode raw H.264 at 1920x1080, 60 fps, 15 mbits. The test assumes low-latency CBR encoding.
./pyroenc-test test.h264 test.rgb 1920 1080 60 15000

# Playback
ffplay test.h264
```

If the raw video path is `*.nv12`, then it will be treated as raw NV12 and encoded appropriately.

pyroenc

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README