bulker
Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)
File Explorer
Download Latest Version (.zip)- bulker-summary.excalidraw.png
- stream-batch.excalidraw.png
- data_types.md
- http.md
- db-feature-matrix.md
- http-api.md
- server-config.md
- tag-release.yml
- test-build-image.yml
- app.go
- config.go
- failover_reprocessor.go
- go.mod
- go.sum
- k8s_client.go
- kafka.go
- main.go
- reprocessing_db.go
- reprocessing_db_schema.go
- REPROCESSING_K8S.md
- router.go
- router_reprocessing_handler.go
- types.go
- badbatch.ndjson
- bulker.env
- config.yaml
- goodbatch.ndjson
- single.ndjson
- docker-compose.yml
- kafka_container.go
- abstract_batch_consumer.go
- abstract_consumer.go
- app.go
- app_config.go
- batch_consumer.go
- configuration_source.go
- cron.go
- fast_store.go
- http_configuration_source.go
- integration_test.go
- load_test.go
- metrics_server.go
- multi_configuration_source.go
- postgres_configuration_source.go
- producer.go
- redis_configuration_source.go
- repository.go
- retry_consumer.go
- router.go
- stream_consumer.go
- topic_manager.go
- topic_manager_test.go
- metrics.go
- default.pgo
- go.mod
- go.sum
- main.go
- mixpanel.go
- transactional_stream.go
- webhook.go
- empty.ndjson
- no_repeated_ids.ndjson
- repeated_ids.ndjson
- repeated_ids_discr.ndjson
- minio_container.go
- abstract.go
- bulker_test.go
- gcs_bulker.go
- replacepartition_stream.go
- replacetable_stream.go
- s3_bulker.go
- transactional_stream.go
- client.go
- connection.go
- connection_test.go
- connector.go
- driver.go
- driver_test.go
- dsn.go
- dsn_test.go
- errors.go
- result.go
- rows.go
- statement.go
- tx.go
- utils.go
- columns_added.ndjson
- columns_added2.ndjson
- columns_added_partial.ndjson
- columns_added_partial2.ndjson
- data_types.ndjson
- date_mix.ndjson
- date_type.ndjson
- emoji.ndjson
- empty.ndjson
- existing_table1.ndjson
- existing_table2.ndjson
- existing_table_num.ndjson
- existing_table_text.ndjson
- identifiers.ndjson
- merge_window1.ndjson
- merge_window2.ndjson
- merge_window3.ndjson
- merge_window4.ndjson
- multiline.ndjson
- nested.ndjson
- non_utc_timestamp.ndjson
- partition1.ndjson
- partition1_1.ndjson
- partition2.ndjson
- repeated_ids.ndjson
- repeated_ids2.ndjson
- repeated_ids_discr.ndjson
- repeated_ids_multi.ndjson
- repeated_ids_multi2.ndjson
- replace_table.ndjson
- schema_option.ndjson
- simple.ndjson
- simple2.ndjson
- type_hints.ndjson
- type_hints_bq.ndjson
- types.ndjson
- types2.ndjson
- types_coalesce.ndjson
- types_collision.ndjson
- types_collision2.ndjson
- types_json.ndjson
- types_json_noarr_part1.ndjson
- types_json_noarr_part2.ndjson
- types_json_part1.ndjson
- types_json_part2.ndjson
- types_json_part3.ndjson
- config.xml
- users.xml
- config.xml
- users.xml
- config.xml
- users.xml
- config.xml
- users.xml
- ch_cluster_container.go
- docker-compose.yml
- config.xml
- users.xml
- config.xml
- users.xml
- ch_cluster_container.go
- docker-compose.yml
- ch_container.go
- mysql_container.go
- pg_container.go
- abstract.go
- abstract_transactional.go
- autocommit_stream.go
- autocommit_stream_test.go
- batch_header.go
- batch_header_test.go
- bigdata_test.go
- bigquery.go
- bulker_test.go
- clickhouse.go
- datasource_config.go
- dedup_with_discr_test.go
- delete_condition.go
- duckdb.go
- existing_table_test.go
- mergewindow_test.go
- mysql.go
- namespace_test.go
- naming_test.go
- options.go
- postgres.go
- processor.go
- reconnect_test.go
- redshift.go
- redshift_iam.go
- replacepartition_stream.go
- replacepartition_stream_test.go
- replacetable_stream.go
- replacetable_stream_test.go
- schema_freeze_test.go
- snowflake.go
- sql_adapter.go
- sql_adapter_base.go
- table.go
- table_helper.go
- transactional_stream.go
- transactional_stream_test.go
- tx_wrapper.go
- type_resolver.go
- types_test.go
- utils.go
- file.go
- flattener.go
- google_cloud_storage.go
- s3.go
- avro.go
- converter.go
- datatype.go
- datatype_test.go
- error.go
- marshaller.go
- object.go
- schema.go
- sqltype.go
- bulker.go
- bulker_state_test.go
- go.mod
- go.sum
- loader.go
- options.go
- app.go
- config.go
- go.mod
- main.go
- router.go
- schema.go
- cmdparser.go
- doc.go
- go.mod
- inferschema.go
- LICENCE
- protocol.go
- README.md
- safewriter.go
- source.go
- sourceRunner.go
- trackers.go
- firebase.go
- firebase_test.go
- go.mod
- go.sum
- main.go
- firebase.Dockerfile
- firebasebuild.sh
- redis.go
- ch_events_log.go
- events_log.go
- events_log_config.go
- events_log_test.go
- go.mod
- go.sum
- metrics.go
- redis_events_log.go
- app.go
- backup_logger.go
- config.go
- consumer_monitor.go
- default.pgo
- destination_types.go
- filters.go
- go.mod
- go.sum
- main.go
- metrics.go
- metrics_server.go
- repository.go
- router.go
- router_batch_handler.go
- router_batch_handler_test.go
- router_classic_handler.go
- router_ingest_handler.go
- router_pixel_handler.go
- router_script_handler.go
- router_segment_settings_handler.go
- script_repository.go
- types.go
- app.go
- config.go
- go.mod
- go.sum
- k8s.go
- main.go
- manager.go
- router.go
- abstract_repository.go
- app_base.go
- http_repository.go
- router_base.go
- router_base_test.go
- service_base.go
- interface.go
- types.go
- adapter.go
- any.go
- any_array.go
- any_bool.go
- any_float.go
- any_int32.go
- any_int64.go
- any_invalid.go
- any_nil.go
- any_number.go
- any_object.go
- any_str.go
- any_uint32.go
- any_uint64.go
- config.go
- example_test.go
- iter.go
- iter_array.go
- iter_float.go
- iter_int.go
- iter_object.go
- iter_skip.go
- iter_skip_sloppy.go
- iter_skip_sloppy_test.go
- iter_skip_strict.go
- iter_str.go
- jsoniter.go
- LICENSE
- pool.go
- README.md
- reflect.go
- reflect_array.go
- reflect_dynamic.go
- reflect_extension.go
- reflect_json_number.go
- reflect_json_raw_message.go
- reflect_map.go
- reflect_marshaler.go
- reflect_native.go
- reflect_optional.go
- reflect_slice.go
- reflect_struct_decoder.go
- reflect_struct_encoder.go
- stream.go
- stream_float.go
- stream_int.go
- stream_str.go
- stream_test.go
- adapter.go
- any.go
- any_array.go
- any_bool.go
- any_float.go
- any_int32.go
- any_int64.go
- any_invalid.go
- any_nil.go
- any_number.go
- any_object.go
- any_str.go
- any_uint32.go
- any_uint64.go
- config.go
- example_test.go
- iter.go
- iter_array.go
- iter_float.go
- iter_int.go
- iter_object.go
- iter_skip.go
- iter_skip_sloppy.go
- iter_skip_sloppy_test.go
- iter_skip_strict.go
- iter_str.go
- json_test.go
- jsoniter.go
- LICENSE
- list.go
- orderedmap.go
- orderedset.go
- pool.go
- README.md
- reflect.go
- reflect_array.go
- reflect_dynamic.go
- reflect_extension.go
- reflect_json_number.go
- reflect_json_raw_message.go
- reflect_map.go
- reflect_marshaler.go
- reflect_native.go
- reflect_optional.go
- reflect_orderedmap.go
- reflect_slice.go
- reflect_struct_decoder.go
- reflect_struct_encoder.go
- stream.go
- stream_float.go
- stream_int.go
- stream_str.go
- stream_test.go
- types.go
- dual.go
- global_logger.go
- level.go
- object_logger.go
- proxy.go
- query_logger.go
- rolling_writer.go
- string_writer.go
- task_logger.go
- utils.go
- writer_mock.go
- pgpool.go
- safego.go
- safego_test.go
- format.go
- timestamp.go
- timestamp_test.go
- json.go
- json_test.go
- LICENSE
- README.md
- set.go
- arrays.go
- bool.go
- bytes.go
- bytes_test.go
- cache.go
- crypto.go
- crypto_test.go
- errors.go
- hash.go
- maps.go
- net.go
- numbers.go
- objects.go
- sets.go
- strings.go
- time.go
- uuid.go
- go.mod
- go.sum
- main.go
- failover_config.go
- failover_logger.go
- failover_logger_test.go
- go.mod
- go.sum
- kafka_config.go
- metrics.go
- producer.go
- utils.go
- go.mod
- go.sum
- main.go
- README.md
- repository.go
- catalog.json
- config.json
- github_compose.yaml
- README.md
- state.json
- github_pod.yaml
- README.md
- app.go
- config.go
- db_schema.go
- go.mod
- go.sum
- job_runner.go
- k8s.go
- main.go
- README.md
- router.go
- task.go
- task_manager.go
- db.go
- go.mod
- go.sum
- main.go
- read.go
- README.md
- spec_catalog.go
- types.go
- .dockerignore
- .gitignore
- all.Dockerfile
- allbuild.sh
- bulker.Dockerfile
- bulkerbuild.sh
- CONTRIBUTING.md
- go.work
- go.work.sum
- GOWORK.md
- ingest.Dockerfile
- ingestbuild.sh
- LICENSE
- README.md
- release.sh
- reprocessing-worker-build.sh
- reprocessing-worker.Dockerfile
- sidecar.Dockerfile
- sidecarbuild.sh
- syncctl.Dockerfile
- syncctlbuild.sh
- TODO.md
# Installation Guide
1. Get the code
git clone https://github.com/jitsucom/bulker
Downloads the entire project code from GitHub to your computer.
cd bulker
Moves into the project folder you just downloaded.
2. Docker
Easy RecommendedPrerequisites
- Git Needed to download the project code from GitHub.
- Docker Desktop Needed to build and run containers. Install it and keep it running in the background.
β οΈ This is a large repository, so this method may point to an internal sub-package rather than the actual core product. Check the full README as well.
docker compose -f bulkerapp/app/testcontainers/kafka/docker-compose.yml up -d --build
Runs the command against the services defined in the compose file.
Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.
3. Go
MediumPrerequisites
cd admin
This project's files live in a subfolder, so move into it first.
go build ./...
Compiles the Go program into an executable.
go run .
Runs the Go program directly without a separate build step.
If the build finishes without errors, it worked. If you used go run ., check the terminal output.
// repository documentation
Was this content helpful?
(0 ratings)