pgstream
PostgreSQL replication with DDL changes
File Explorer
Download Latest Version (.zip)- bug_report.md
- feature_request.md
- policy.yml
- review-guidance.md
- backport.yml
- build.yml
- coverage-report.yml
- dependabot-auto-merge.yml
- nightly.yml
- pr-approval-agent.yml
- dependabot.yml
- pull_request_template.md
- pgstream-banner.svg
- pgstream-banner@2x.png
- pgstream-black-symbol.svg
- pgstream-white-symbol.svg
- pgstream-black-wordmark.svg
- pgstream-white-wordmark.svg
- README.md
- Dockerfile
- postgresql.conf
- Dockerfile
- postgresql.conf
- Dockerfile
- postgresql.conf
- Dockerfile
- postgresql.conf
- Dockerfile
- postgresql.conf
- histogramQuantile
- default-password.xml
- cluster.xml
- config.xml
- custom-function.xml
- storage.xml
- users.xml
- pgstream.json
- otel-collector-config.yaml
- otel-collector-opamp-config.yaml
- prometheus.yml
- docker-compose-signoz.yml
- docker-compose.yml
- test_config.env
- test_config.yaml
- test_transformer_rules.yaml
- bytesize.go
- bytesize_test.go
- config.go
- config_env.go
- config_env_test.go
- config_test.go
- config_yaml.go
- config_yaml_fuzz_test.go
- config_yaml_test.go
- env_expansion.go
- env_expansion_test.go
- helper_test.go
- check_cmd.go
- health_cmd.go
- init_cmd.go
- root_cmd.go
- root_cmd_test.go
- run_cmd.go
- snapshot_cmd.go
- status_cmd.go
- validate_cmd.go
- kafka2os.env
- kafka2os.yaml
- kafka2pg.yaml
- pg2kafka.env
- pg2kafka.yaml
- pg2os.env
- pg2os.yaml
- pg2pg.env
- pg2pg.yaml
- pg2webhook.env
- pg2webhook.yaml
- snapshot2pg.env
- snapshot2pg.yaml
- transformer_rules.yaml
- pgstream_arch_v1.png
- pgstream_diagram_v2.svg
- pgstream_diagram_v2_kafka.svg
- pgstream_snapshot_benchmarks.png
- pgstream_snapshot_diagram.svg
- pgstream_snapshot_sequence.svg
- pgstream_transformer_diagram.svg
- pgstream_tutorial_kafka.svg
- pgstream_tutorial_pg2os.svg
- pgstream_tutorial_pg2pg.svg
- pgstream_tutorial_pg2webhooks.svg
- pgstream_tutorial_snapshot2pg.svg
- pgstream_tutorial_transformer.svg
- pgstream_v1_schema_replication.png
- RELEASE_NOTES_v1.md
- kafka2os_tutorial.env
- kafka2os_tutorial.yaml
- kafka2pg_tutorial.env
- kafka2pg_tutorial.yaml
- pg2kafka_tutorial.env
- pg2kafka_tutorial.yaml
- pg2os_tutorial.env
- pg2os_tutorial.yaml
- pg2pg_transformer_tutorial.env
- pg2pg_transformer_tutorial.yaml
- pg2pg_tutorial.env
- pg2pg_tutorial.yaml
- pg2webhook_tutorial.env
- pg2webhook_tutorial.yaml
- postgres_kafka.md
- postgres_snapshot.md
- postgres_to_opensearch.md
- postgres_to_postgres.md
- postgres_to_webhooks.md
- postgres_transformer.md
- snapshot2pg_tutorial.env
- snapshot2pg_tutorial.yaml
- tutorial_transformer_rules.yaml
- architecture.md
- aws.md
- cli.md
- configuration.md
- container_image.md
- gcp_cloudsql.md
- glossary.md
- neon.md
- Observability.md
- privileges.md
- replicas.md
- snapshots.md
- transformers.md
- xata.md
- health.go
- health_test.go
- mock_http_client.go
- http.go
- json.go
- json_test.go
- zerolog.go
- zerolog_test.go
- math.go
- math_test.go
- migrator.go
- object_type_filter.go
- object_type_filter_test.go
- phase.go
- phase_test.go
- instrumented_pg_dump_restore.go
- instrumented_querier.go
- instrumented_querier_builder.go
- instrumented_tx.go
- mock_pg_mapper.go
- mock_pg_querier.go
- mock_pg_replication_conn.go
- mock_rows.go
- mock_tx.go
- pg_querier_retrier.go
- pg_querier_retrier_test.go
- errors.go
- errors_test.go
- helper_test.go
- pg_conn.go
- pg_conn_pool.go
- pg_conn_pool_test.go
- pg_dump.go
- pg_dump_pg_restore_integration_test.go
- pg_dumpall.go
- pg_lazy_conn.go
- pg_mapper.go
- pg_mapper_test.go
- pg_primary_key.go
- pg_primary_key_test.go
- pg_querier.go
- pg_querier_builder.go
- pg_replication_conn.go
- pg_restore.go
- pg_restore_test.go
- pg_snapshot_probe.go
- pg_snapshot_probe_test.go
- pg_test.go
- pg_tx.go
- pg_tx_bench_test.go
- pg_tx_repro_test.go
- pg_utils.go
- pg_utils_integration_test.go
- pg_utils_test.go
- schema_table_map.go
- schema_table_map_test.go
- profiling.go
- mock_progress_bar.go
- progress_bar.go
- elasticsearch_client.go
- elasticsearch_client_test.go
- elasticsearch_mapper.go
- mock_client.go
- mock_mapper.go
- opensearch_client.go
- opensearch_client_test.go
- opensearch_mapper.go
- search_api.go
- search_client.go
- search_errors.go
- search_mapper.go
- mock_weighted_semaphore.go
- map.go
- semaphore.go
- Dockerfile
- test_elasticsearch_container.go
- test_kafka_container.go
- test_opensearch_container.go
- test_postgres_container.go
- test_postgres_replica_container.go
- 1_create_is_system_schema_function.down.sql
- 1_create_is_system_schema_function.up.sql
- 2_create_emit_ddl_function_and_triggers.down.sql
- 2_create_emit_ddl_function_and_triggers.up.sql
- 3_fix_duplicate_ddl_emission.down.sql
- 3_fix_duplicate_ddl_emission.up.sql
- migrations.go
- 1_create_pgstream_xid.down.sql
- 1_create_pgstream_xid.up.sql
- 2_create_pgstream_tableids_table.down.sql
- 2_create_pgstream_tableids_table.up.sql
- 3_update_emit_ddl_function.down.sql
- 3_update_emit_ddl_function.up.sql
- 4_fix_duplicate_ddl_emission.down.sql
- 4_fix_duplicate_ddl_emission.up.sql
- 5_handle_duplicate_table_mapping.down.sql
- 5_handle_duplicate_table_mapping.up.sql
- migrations.go
- migrations.go
- mock_backoff.go
- backoff.go
- backoff_test.go
- instrumented_kafka_reader.go
- instrumented_kafka_writer.go
- mock_kafka_parser.go
- mock_kafka_reader.go
- mock_kafka_writer.go
- config.go
- conn.go
- kafka_offset_parser.go
- kafka_offset_parser_test.go
- kafka_reader.go
- kafka_writer.go
- log.go
- logger.go
- logger.go
- config.go
- otel_instrumentation.go
- otel_provider.go
- span.go
- version.go
- instrumented_snapshot_generator.go
- mock_snapshot_generator.go
- config.go
- helper_test.go
- instrumented_table_snapshot_generator.go
- pg_snapshot_generator.go
- pg_snapshot_generator_integration_test.go
- pg_snapshot_generator_test.go
- pg_snapshot_loadbalanced_test.go
- pg_snapshot_row_adapter.go
- test_dump.sql
- test_dump_constraints.sql
- test_dump_event_triggers.sql
- test_dump_filtered.sql
- test_dump_views.sql
- helper_test.go
- object_type_filter.go
- object_type_filter_test.go
- pg_options_generator.go
- pg_options_generator_test.go
- pg_role.go
- pg_role_parser.go
- pg_role_parser_test.go
- pg_snapshot_tracker.go
- pg_snapshot_tracker_test.go
- pg_snapshot_wal_restore.go
- pg_snapshot_wal_restore_test.go
- snapshot_pg_dump_restore_generator.go
- snapshot_pg_dump_restore_generator_test.go
- instrumented_table_discovery.go
- pg_snapshot_table_finder.go
- pg_snapshot_table_finder_test.go
- helper_test.go
- snapshot_generator.go
- snapshot_generator_recorder.go
- snapshot_generator_recorder_test.go
- instrumented_snapshot_store.go
- mock_snapshot_store.go
- pg_snapshot_store_integration_test.go
- pg_snapshot_store.go
- pg_snapshot_store_test.go
- snapshot_store.go
- errors.go
- errors_test.go
- snapshot.go
- snapshot_test.go
- postgresql.conf
- object_types_fixture.sql
- helper_test.go
- pg_kafka_integration_test.go
- pg_pg_batch_coalesce_integration_test.go
- pg_pg_dropped_batch_metrics_integration_test.go
- pg_pg_filter_integration_test.go
- pg_pg_integration_test.go
- pg_pg_integration_transformer_test.go
- pg_pg_object_type_filter_integration_test.go
- pg_replica_integration_test.go
- pg_search_integration_test.go
- pg_search_tls_integration_test.go
- pg_webhook_integration_test.go
- phase_status_integration_test.go
- setup_test.go
- snapshot_migration_window_integration_test.go
- snapshot_pg_integration_test.go
- wal_level_logical.conf
- access.go
- access_integration_test.go
- access_test.go
- builder.go
- CLAUDE.md
- connectivity.go
- preflight.go
- preflight_test.go
- printer.go
- replica_identity.go
- replica_identity_test.go
- replication.go
- replication_integration_test.go
- replication_test.go
- resources.go
- resources_test.go
- schema.go
- schema_test.go
- snapshot_source_instance.go
- snapshot_source_instance_test.go
- version.go
- version_test.go
- config.go
- config_test.go
- helper_test.go
- phase_metrics.go
- phase_metrics_test.go
- processor_chain_test.go
- snapshot_config_test.go
- stream.go
- stream_init.go
- stream_init_test.go
- stream_run.go
- stream_snapshot.go
- stream_status.go
- stream_status_checker.go
- stream_status_checker_test.go
- stream_status_test.go
- test.csr
- test.key
- test.pem
- tls.go
- tls_test.go
- transformer_builder.go
- transformer_builder_test.go
- transformer_concurrency_test.go
- transformer_uniqueness_test.go
- deterministic_bytes_generator.go
- generator.go
- pooled_generator.go
- random_bytes_generator.go
- greenmask_boolean_transformer.go
- greenmask_boolean_transformer_test.go
- greenmask_choice_transformer.go
- greenmask_choice_transformer_test.go
- greenmask_date_transformer.go
- greenmask_date_transformer_test.go
- greenmask_firstname_transformer.go
- greenmask_firstname_transformer_test.go
- greenmask_float_transformer.go
- greenmask_float_transformer_test.go
- greenmask_integer_transformer.go
- greenmask_integer_transformer_test.go
- greenmask_string_transformer.go
- greenmask_string_transformer_test.go
- greenmask_timestamp_transformer.go
- greenmask_timestamp_transformer_test.go
- greenmask_transformer.go
- greenmask_unix_timestamp_transformer.go
- greenmask_unix_timestamp_transformer_test.go
- greenmask_uuid_transformer.go
- greenmask_uuid_transformer_test.go
- instrumented_transformer.go
- pool.go
- template.go
- template_test.go
- mock_builder.go
- mock_transformer.go
- errors.go
- neosync_email_transformer.go
- neosync_email_transformer_test.go
- neosync_firstname_transformer.go
- neosync_firstname_transformer_test.go
- neosync_fullname_transformer.go
- neosync_fullname_transformer_test.go
- neosync_lastname_transformer.go
- neosync_lastname_transformer_test.go
- neosync_string_transformer.go
- neosync_string_transformer_test.go
- neosync_transformer.go
- email_transformer.go
- email_transformer_test.go
- encrypted_aes_siv_transformer.go
- encrypted_aes_siv_transformer_test.go
- hstore_transformer.go
- hstore_transformer_operation.go
- hstore_transformer_test.go
- json_transformer.go
- json_transformer_operation.go
- json_transformer_test.go
- literal_string_transformer.go
- literal_string_transformer_test.go
- masking_transformer.go
- masking_transformer_test.go
- pg_anonymizer_transformer.go
- pg_anonymizer_transformer_test.go
- phone_number_transformer.go
- phone_number_transformer_test.go
- string_transformer.go
- string_transformer_test.go
- template_transformer.go
- template_transformer_env_test.go
- template_transformer_test.go
- transformer.go
- transformer_test.go
- uniqueness.go
- uniqueness_test.go
- wal_kafka_checkpointer.go
- wal_kafka_checkpointer_test.go
- helper_test.go
- wal_pg_checkpointer.go
- wal_pg_checkpointer_test.go
- wal_checkpointer.go
- wal_kafka_reader.go
- wal_kafka_reader_test.go
- bytea.go
- bytea_test.go
- helper_test.go
- wal_pg_listener.go
- wal_pg_listener_test.go
- config.go
- config_test.go
- wal_snapshot_generator_adapter.go
- wal_snapshot_generator_adapter_test.go
- config.go
- wal_listener_snapshot_generator_builder.go
- wal_snapshot_listener.go
- wal_listener.go
- mock_batch_sender.go
- helper_test.go
- wal_batch.go
- wal_batch_bytes_setting.go
- wal_batch_bytes_setting_test.go
- wal_batch_bytes_tuner.go
- wal_batch_bytes_tuner_property_test.go
- wal_batch_bytes_tuner_test.go
- wal_batch_dropped_counter.go
- wal_batch_dropped_counter_test.go
- wal_batch_sender.go
- wal_batch_sender_config.go
- wal_batch_sender_test.go
- wal_message.go
- wal_filter.go
- wal_filter_test.go
- helper_test.go
- wal_injector.go
- wal_injector_test.go
- instrumented_wal_processor.go
- config.go
- wal_kafka_batch_writer.go
- wal_kafka_batch_writer_close_test.go
- wal_kafka_batch_writer_test.go
- wal_kafka_message_key_property_test.go
- mock_processor.go
- config.go
- config_test.go
- ddl_object_type_filter.go
- ddl_object_type_filter_test.go
- helper_test.go
- instrumented_wal_adapter.go
- jsonb_serialization_test.go
- postgres_batch_writer.go
- postgres_batch_writer_test.go
- postgres_bulk_ingest_writer.go
- postgres_bulk_ingest_writer_test.go
- postgres_query_msg.go
- postgres_query_msg_test.go
- postgres_schema_observer.go
- postgres_schema_observer_test.go
- postgres_wal_adapter.go
- postgres_wal_adapter_test.go
- postgres_wal_ddl_adapter.go
- postgres_wal_ddl_adapter_test.go
- postgres_wal_dml_adapter.go
- postgres_wal_dml_adapter_bulk.go
- postgres_wal_dml_adapter_bulk_test.go
- postgres_wal_dml_adapter_test.go
- postgres_wal_message.go
- postgres_wal_message_test.go
- postgres_writer.go
- wal_sanitizer.go
- wal_sanitizer_test.go
- instrumented_search_store.go
- mock_search_mapper.go
- search_adapter.go
- search_index_name.go
- search_pg_mapper.go
- search_pg_mapper_test.go
- search_store.go
- search_store_test.go
- config.go
- errors.go
- helper_test.go
- id_hasher.go
- search_adapter.go
- search_adapter_test.go
- search_batch_indexer.go
- search_batch_indexer_close_test.go
- search_batch_indexer_test.go
- search_msg_batch.go
- search_store_retrier.go
- search_store_retrier_test.go
- store.go
- wal_stdout_writer.go
- wal_stdout_writer_test.go
- config.go
- wal_postgres_anon_rule_parser.go
- wal_postgres_anon_rule_parser_test.go
- wal_postgres_transformer_parser.go
- wal_postgres_transformer_parser_test.go
- wal_transformer.go
- wal_transformer_map.go
- wal_transformer_parser.go
- wal_transformer_parser_test.go
- wal_transformer_rules.go
- wal_transformer_test.go
- wal_transformer_uniqueness.go
- wal_transformer_uniqueness_integration_test.go
- wal_transformer_uniqueness_test.go
- config.go
- config_test.go
- helper_test.go
- webhook_notifier.go
- webhook_notifier_test.go
- webhook_notify_msg.go
- config.go
- subscription_server.go
- subscription_server_test.go
- config.go
- helper_test.go
- subscription_store_cache.go
- subscription_store_cache_test.go
- mock_subscription_store.go
- pg_subscription_store.go
- pg_subscription_store_test.go
- subscription_store.go
- subscription.go
- subscription_test.go
- webhook.go
- errors.go
- wal_processor.go
- instrumented_replication_handler.go
- replication_metrics_cache.go
- replication_metrics_cache_test.go
- mock_replication_handler.go
- mock_replication_lsn_parser.go
- helper_test.go
- pg_lsn_parser.go
- pg_replication_handler.go
- pg_replication_handler_test.go
- replication_handler_retrier.go
- replication_handler_retrier_test.go
- replication_handler.go
- wal_data.go
- wal_ddl.go
- wal_ddl_test.go
- wal_schema_diff.go
- wal_schema_diff_test.go
- analyze_results.py
- BENCHMARK_SUMMARY.md
- network_baseline.sh
- README.md
- run_benchmark.sh
- run_multi_config_benchmark.sh
- setup_network_conditions.sh
- policy.yml
- .gitignore
- config.go
- config_test.go
- gates.go
- gates_test.go
- github.go
- go.mod
- go.sum
- main.go
- README.md
- render.go
- reviewer.go
- reviewer_test.go
- build-transformers-definition.go
- webhook_server.go
- build-cli-definition.go
- .gitignore
- .golangci.yml
- .goreleaser.nightly.yaml
- .goreleaser.yaml
- .pre-commit-config.yaml
- CLAUDE.md
- cli-definition.json
- CODE_OF_CONDUCT.md
- config_template.yaml
- CONTRIBUTING.md
- Dockerfile
- go.mod
- go.sum
- LICENSE
- license-header.txt
- main.go
- Makefile
- README.md
- transformers-definition.json
# Installation Guide
1. Get the code
git clone https://github.com/xataio/pgstream
Downloads the entire project code from GitHub to your computer.
cd pgstream
Moves into the project folder you just downloaded.
2. Official Install Script
Easy Recommendedgo install github.com/xataio/pgstream@latest
Downloads and installs the specified version directly β no need to clone the repo yourself.
brew tap xataio/pgstream
Registers this project's repository (tap) with Homebrew.
brew install pgstream
Installs the pre-built package via Homebrew β no source build required.
After installing, open a new terminal and run the program's version command (e.g. --version) to confirm it worked.
Pulled directly from this repo's README.
3. Docker
EasyPrerequisites
- Git Needed to download the project code from GitHub.
- Docker Desktop Needed to build and run containers. Install it and keep it running in the background.
docker-compose -f build/docker/docker-compose.yml up
Runs the command against the services defined in the compose file.
docker-compose -f build/docker/docker-compose.yml --profile pg2pg up
Runs the command against the services defined in the compose file.
docker-compose -f build/docker/docker-compose.yml --profile pg2pg --profile kafka up
Runs the command against the services defined in the compose file.
Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.
Pulled directly from this repo's README.
4. Go
MediumPrerequisites
go install github.com/xataio/pgstream@latest
Downloads and installs the specified version directly β no need to clone the repo yourself.
go build -o pgstream ./cmd
Compiles the Go program into an executable.
If the build finishes without errors, it worked. If you used go run ., check the terminal output.
Pulled directly from this repo's README.
5. Make
MediumPrerequisites
- Git Needed to download the project code from GitHub.
- Make Usually pre-installed on Linux/macOS. On Windows, install separately (e.g. via MSYS2 or WSL).
make
Compiles the code based on the generated build configuration to produce an executable.
If it finishes without errors, it worked. Try running the generated executable directly.
// repository documentation
Was this content helpful?
(0 ratings)