optimus
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
File Explorer
Download Latest Version (.zip)- bug_report.md
- custom.md
- feature_request.md
- story.md
- publish-latest.yml
- release.yml
- verify.yml
- backup.go
- create.go
- list.go
- status.go
- activate.go
- clean.go
- describe.go
- extension.go
- install.go
- rename.go
- run.go
- uninstall.go
- upgrade.go
- initialize.go
- auth.go
- keyring.go
- connection.go
- insecure.go
- secure.go
- token_auth.go
- logger.go
- printer.go
- progress_bar.go
- backup_create.go
- extension.go
- initialize.go
- job.go
- job_addhook.go
- job_create.go
- namespace.go
- resource_create.go
- secret_set.go
- survey.go
- load_client_config.go
- manage_plugins.go
- mark_flag_required.go
- addhook.go
- change_namespace.go
- create.go
- export.go
- inspect.go
- job.go
- refresh.go
- replace_all.go
- run_input.go
- run_list.go
- validate.go
- describe.go
- list.go
- namespace.go
- register.go
- constant.go
- model.go
- window.go
- playground.go
- install.go
- plugin.go
- sync.go
- validate.go
- describe.go
- project.go
- register.go
- create.go
- list.go
- replay.go
- status.go
- apply.go
- change_namespace.go
- create.go
- export.go
- resource.go
- upload_all.go
- scheduler.go
- upload_all.go
- delete.go
- list.go
- secret.go
- set.go
- version.go
- commands.go
- factory.go
- factory_test.go
- activate.go
- activate_test.go
- error.go
- install.go
- install_test.go
- internal.go
- rename.go
- rename_test.go
- run.go
- run_test.go
- uninstall.go
- uninstall_test.go
- upgrade.go
- upgrade_test.go
- AssetOperator.go
- client.go
- manifester.go
- asset_operator.go
- error.go
- manifest.go
- metadata.go
- model.go
- provider.go
- client.go
- client_test.go
- model.go
- parse.go
- parse_test.go
- registry.go
- default_asset_operator.go
- default_asset_operator_test.go
- default_manifester.go
- default_manifester_test.go
- extension.go
- extension_test.go
- manager.go
- manager_test.go
- internal.go
- job_spec.go
- job_spec_test.go
- resource_spec.go
- resource_spec_test.go
- export_test.go
- job_spec_io.go
- job_spec_io_opt.go
- job_spec_io_test.go
- resource_spec_io.go
- resource_spec_io_test.go
- specio.go
- build.go
- config.go
- config_client.go
- config_client_test.go
- config_common.go
- config_server.go
- loader.go
- loader_test.go
- validation.go
- validation_test.go
- handler.go
- handler_test.go
- worker.go
- event.go
- job.go
- resource.go
- scheduler.go
- job.go
- job_adapter.go
- job_test.go
- external_upstream_resolver.go
- external_upstream_resolver_test.go
- internal_upstream_resolver.go
- internal_upstream_resolver_test.go
- upstream_resolver.go
- upstream_resolver_test.go
- util.go
- filter.go
- filter_opt.go
- filter_test.go
- job_service.go
- job_service_test.go
- plugin_service.go
- plugin_service_test.go
- job.go
- job_test.go
- spec.go
- spec_test.go
- backup.go
- backup_test.go
- resource.go
- resource_test.go
- backup_service.go
- backup_service_test.go
- resource_manager.go
- resource_manager_test.go
- resource_service.go
- resource_service_test.go
- backup.go
- backup_test.go
- resource.go
- resource_status.go
- resource_test.go
- status.go
- status_test.go
- store.go
- store_test.go
- job_run.go
- job_run_test.go
- replay.go
- replay_test.go
- simple_resolver.go
- simple_resolver_test.go
- deployment_service.go
- deployment_service_test.go
- executor_input_compiler.go
- executor_input_compiler_test.go
- job_run_asset_compiler.go
- job_run_asset_compiler_test.go
- job_run_service.go
- job_run_service_test.go
- notification_service.go
- notification_service_test.go
- replay_manager.go
- replay_manager_test.go
- replay_service.go
- replay_service_test.go
- replay_validator.go
- replay_validator_test.go
- replay_worker.go
- replay_worker_test.go
- event.go
- event_test.go
- executor.go
- executor_test.go
- job.go
- job_run.go
- job_test.go
- replay.go
- replay_test.go
- status.go
- status_test.go
- secret.go
- namespace.go
- namespace_test.go
- project.go
- project_test.go
- secret.go
- secret_test.go
- namespace_service.go
- namespace_service_test.go
- project_service.go
- project_service_test.go
- secret_service.go
- secret_service_test.go
- tenant_service.go
- tenant_service_test.go
- namespace.go
- namespace_test.go
- project.go
- project_test.go
- secret.go
- secret_test.go
- tenant.go
- tenant_test.go
- optimus-plugin-echo.yaml
- .gitignore
- airflow.values.yaml
- get_artifacts.sh
- load_secrets.sh
- Makefile
- optimus.values.yaml
- postgres.yaml
- pvc.yaml
- README.md
- setup.yaml
- 2021-10-02-optimus-launch.md
- introduction.md
- tutorial.md
- applying-job-specifications.md
- backup-bigquery-resource.md
- configuration.md
- create-job-specifications.md
- installing-plugin.md
- manage-bigquery-resource.md
- managing-project-namespace.md
- managing-secrets.md
- organizing-specifications.md
- replay-a-job.md
- setting-up-alert.md
- uploading-jobs-to-scheduler.md
- verifying-jobs.md
- work-with-extension.md
- architecture.md
- dependency.md
- intervals-and-windows.md
- job-run.md
- job.md
- macros.md
- namespace.md
- plugin.md
- project.md
- replay-and-backup.md
- resource.md
- secret.md
- contribution-process.md
- developer-env-setup.md
- installation.md
- quick-start.md
- API.md
- FAQ.md
- metrics.md
- simplify_plugins.png
- simplify_plugins_executor.png
- 20211002_secret_management.md
- 20220123_support_for_depending_on_external_sensors.md
- 20220124_revisit_automated_dependency_resolution.md
- 20220216_improving_time_and_flow_of_deployment.md
- 20220507_simplify_plugin_maintenance.md
- 20220517_optimus_dashboarding.md
- 20220525_replay_rate_limiting.md
- README.md
- template.md
- configuration.md
- db-migrations.md
- installing-plugins.md
- starting-optimus-server.md
- introduction.md
- Container.js
- GridBlock.js
- custom.css
- theme.css
- help.js
- index.js
- Concept_JobRun.png
- CreateJobSpecFlow.png
- OptimusArchitecture.png
- OptimusIntro.png
- ReplayParallel.png
- ReplaySequential.png
- banner-0.6.png
- banner.svg
- favicon.ico
- logo.svg
- optimus.svg
- overview.svg
- pattern.svg
- gojek.png
- goto.png
- jago.png
- mapan.png
- midtrans.png
- moka.png
- paylater.png
- .nojekyll
- .gitignore
- babel.config.js
- docusaurus.config.js
- package-lock.json
- package.json
- README.md
- sidebars.js
- yarn.lock
- pagerdutynotifier.go
- pagerdutynotifier_test.go
- pagerdutyservice.go
- slack.go
- slack_test.go
- notify.go
- model.go
- optimus.go
- optimus_test.go
- factory.go
- gcs.go
- compiler.go
- compiler_test.go
- dag.py.tmpl
- expected_dag.py
- models.go
- template_func.go
- template_func_test.go
- upstream.go
- __lib.py
- airflow.go
- client.go
- README.md
- backup.go
- backup_test.go
- batch.go
- batch_test.go
- bigquery.go
- bigquery_test.go
- client.go
- client_test.go
- copier.go
- copier_test.go
- dataset.go
- dataset_spec.go
- dataset_spec_test.go
- dataset_test.go
- external_table.go
- external_table_spec.go
- external_table_spec_test.go
- external_table_test.go
- job.go
- job_test.go
- schema.go
- schema_test.go
- table.go
- table_spec.go
- table_spec_test.go
- table_test.go
- view.go
- view_spec.go
- view_spec_test.go
- view_test.go
- writer.go
- doc.go
- engine.go
- engine_test.go
- template_context.go
- template_context_test.go
- template_func.go
- template_func_test.go
- errors.go
- errors_test.go
- multi.go
- multi_test.go
- cron.go
- cron_test.go
- multi_root_tree.go
- multi_root_tree_test.go
- tree_node.go
- tree_node_test.go
- job.go
- plugin.go
- plugin_test.go
- window.go
- window_test.go
- window_v1.go
- window_v1_test.go
- window_v2.go
- window_v2_test.go
- adapter.go
- job_repository.go
- job_repository_test.go
- 000001_create_project_table.down.sql
- 000001_create_project_table.up.sql
- 000002_create_job_table.down.sql
- 000002_create_job_table.up.sql
- 000003_create_instance_table.down.sql
- 000003_create_instance_table.up.sql
- 000004_update_job_table_add_destination_column.down.sql
- 000004_update_job_table_add_destination_column.up.sql
- 000005_update_job_table_add_description_labels_column.down.sql
- 000005_update_job_table_add_description_labels_column.up.sql
- 000006_create_secret_table.down.sql
- 000006_create_secret_table.up.sql
- 000007_create_resource_table.down.sql
- 000007_create_resource_table.up.sql
- 000008_add_namespace.down.sql
- 000008_add_namespace.up.sql
- 000009_update_job_table_add_behavior_column.down.sql
- 000009_update_job_table_add_behavior_column.up.sql
- 000010_create_replay_table.down.sql
- 000010_create_replay_table.up.sql
- 000011_update_replay_table_add_execution_tree_column.down.sql
- 000011_update_replay_table_add_execution_tree_column.up.sql
- 000012_update_instance_as_job_run_table.down.sql
- 000012_update_instance_as_job_run_table.up.sql
- 000013_update_resource_table_add_urn_column.down.sql
- 000013_update_resource_table_add_urn_column.up.sql
- 000014_create_backup_table.down.sql
- 000014_create_backup_table.up.sql
- 000015_create_instance_table_index.down.sql
- 000015_create_instance_table_index.up.sql
- 000016_update_job_table_add_metadata.down.sql
- 000016_update_job_table_add_metadata.up.sql
- 000017_update_replay_table_add_config_column.down.sql
- 000017_update_replay_table_add_config_column.up.sql
- 000018_update_secret_table_add_namespace_and_type_column.down.sql
- 000018_update_secret_table_add_namespace_and_type_column.up.sql
- 000019_update_job_table_add_external_dependencies_column.down.sql
- 000019_update_job_table_add_external_dependencies_column.up.sql
- 000020_add_job_dependencies_table.down.sql
- 000020_add_job_dependencies_table.up.sql
- 000021_update_index_on_namespace_table.down.sql
- 000021_update_index_on_namespace_table.up.sql
- 000022_add_job_deployment_table.down.sql
- 000022_add_job_deployment_table.up.sql
- 000023_rename_job_run.down.sql
- 000023_rename_job_run.up.sql
- 000024_create_job_run.down.sql
- 000024_create_job_run.up.sql
- 000025_create_task_sensor_hook_run.down.sql
- 000025_create_task_sensor_hook_run.up.sql
- 000026_add_job_source_table.down.sql
- 000026_add_job_source_table.up.sql
- 000027_deprecate_job_dependency.down.sql
- 000027_deprecate_job_dependency.up.sql
- 000028_job_run_delete_cascade.down.sql
- 000028_job_run_delete_cascade.up.sql
- 000029_drop_fkey_task_run_table.down.sql
- 000029_drop_fkey_task_run_table.up.sql
- 000030_update_job_for_window_version.down.sql
- 000030_update_job_for_window_version.up.sql
- 000031_deprecate_instance.down.sql
- 000031_deprecate_instance.up.sql
- 000032_deprecate_job_run_old.down.sql
- 000032_deprecate_job_run_old.up.sql
- 000033_replace_resource.down.sql
- 000033_replace_resource.up.sql
- 000034_replace_backup.down.sql
- 000034_replace_backup.up.sql
- 000035_alter_operator_run_scheduler_bc.down.sql
- 000035_alter_operator_run_scheduler_bc.up.sql
- 000036_alter_job_run_scheduler_bc.down.sql
- 000036_alter_job_run_scheduler_bc.up.sql
- 000037_rename_job.down.sql
- 000037_rename_job.up.sql
- 000038_create_job.down.sql
- 000038_create_job.up.sql
- 000039_create_job_upstream.down.sql
- 000039_create_job_upstream.up.sql
- 000040_drop_job_source.down.sql
- 000040_drop_job_source.up.sql
- 000041_add_sla_alert_log_job_run.down.sql
- 000041_add_sla_alert_log_job_run.up.sql
- 000042_update_project_table.down.sql
- 000042_update_project_table.up.sql
- 000043_update_namespace_table.down.sql
- 000043_update_namespace_table.up.sql
- 000044_update_secret_table.down.sql
- 000044_update_secret_table.up.sql
- 000045_update_foreign_keys.down.sql
- 000045_update_foreign_keys.up.sql
- 000046_update_job_table.down.sql
- 000046_update_job_table.up.sql
- 000047_update_secret_table.down.sql
- 000047_update_secret_table.up.sql
- 000048_run_state_migration.down.sql
- 000048_run_state_migration.up.sql
- 000049_add_monitoring_column_job_run.down.sql
- 000049_add_monitoring_column_job_run.up.sql
- 000050_add_index_operator_run.down.sql
- 000050_add_index_operator_run.up.sql
- 000051_replace_replay.down.sql
- 000051_replace_replay.up.sql
- 000052_create_replay_run.down.sql
- 000052_create_replay_run.up.sql
- 000053_cleanup_old_data.down.sql
- 000053_cleanup_old_data.up.sql
- 000054_job_run_constraint.down.sql
- 000054_job_run_constraint.up.sql
- 000055_job_enable_status.down.sql
- 000055_job_enable_status.up.sql
- backup_repository.go
- backup_repository_test.go
- repository.go
- repository_test.go
- resource.go
- job_operator_repository.go
- job_operator_repository_test.go
- job_repository.go
- job_repository_test.go
- job_run_repository.go
- job_run_repository_test.go
- replay_repository.go
- replay_repository_test.go
- namespace_repository.go
- namespace_repository_test.go
- project_repository.go
- project_repository_test.go
- secret_repository.go
- secret_repository_test.go
- migration.go
- pgx.go
- tracer.go
- add_monitoring_role.down.sql
- add_monitoring_role.up.sql
- Readme.md
- prometheus.go
- telemetry.go
- contains.go
- contains_test.go
- convert.go
- convert_test.go
- file.go
- file_test.go
- map.go
- map_test.go
- proto.go
- proto_test.go
- validator.go
- validator_test.go
- check_job_specification_response_writer.go
- deploy_resource_specification_response_writer.go
- logwriter.go
- refresh_job_response_writer.go
- replace_all_job_specifications_response_writer.go
- writer.go
- plugin.go
- adapter.go
- base.go
- client.go
- connector.go
- server.go
- sample_plugin.yaml
- sample_plugin_schema_invalid.yaml
- sample_plugin_without_shell.yaml
- sample_plugin_without_version.yaml
- plugin.go
- plugin_test.go
- plugin.go
- plugin_manager.go
- backup.pb.go
- backup.pb.gw.go
- backup.swagger.json
- backup_grpc.pb.go
- job_run.pb.go
- job_run.pb.gw.go
- job_run.swagger.json
- job_run_grpc.pb.go
- job_spec.pb.go
- job_spec.pb.gw.go
- job_spec.swagger.json
- job_spec_grpc.pb.go
- namespace.pb.go
- namespace.pb.gw.go
- namespace.swagger.json
- namespace_grpc.pb.go
- project.pb.go
- project.pb.gw.go
- project.swagger.json
- project_grpc.pb.go
- replay.pb.go
- replay.pb.gw.go
- replay.swagger.json
- replay_grpc.pb.go
- resource.pb.go
- resource.pb.gw.go
- resource.swagger.json
- resource_grpc.pb.go
- runtime.pb.go
- runtime.pb.gw.go
- runtime.swagger.json
- runtime_grpc.pb.go
- secret.pb.go
- secret.pb.gw.go
- secret.swagger.json
- secret_grpc.pb.go
- status.pb.go
- status.swagger.json
- event.pb.go
- event.swagger.json
- dependency_resolver.pb.go
- dependency_resolver.swagger.json
- dependency_resolver_grpc.pb.go
- mods.go
- plugin.go
- cli.go
- plugin.go
- plugin_test.go
- questions.go
- resolver.go
- go.mod
- go.sum
- Readme.md
- migrate_to.go
- migration.go
- rollback.go
- serve.go
- version.go
- version_test.go
- logger.go
- optimus.go
- scheduler.go
- server.go
- job_repo_test.go
- backup_repo_test.go
- resource_repo_test.go
- job_repo_test.go
- job_run_repo_test.go
- operator_run_repo_test.go
- namespace_repo_test.go
- project_repo_test.go
- secret_repo_test.go
- database.go
- job.go
- plugin.go
- .gitignore
- .golangci.yml
- .goreleaser.latest.yml
- .goreleaser.yml
- buf.gen.yaml
- config.sample.yaml
- CONTRIBUTING.md
- Dockerfile
- entrypoint_init_container.sh
- go.mod
- go.sum
- LICENSE
- main.go
- Makefile
- optimus.sample.yaml
- README.md
๐ Installation Guide
1. Get the code
git clone https://github.com/raystack/optimus
Downloads the entire project code from GitHub to your computer.
cd optimus
Moves into the project folder you just downloaded.
2. Docker
Easy RecommendedPrerequisites
- Git Needed to download the project code from GitHub.
- Docker Desktop Needed to build and run containers. Install it and keep it running in the background.
docker build -t optimus .
Builds a runnable image based on the Dockerfile.
docker run -p 8080:80 optimus
Runs the built image as an actual container.
Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.
3. Node.js
EasyPrerequisites
cd docs
This project's files live in a subfolder, so move into it first.
npm install
Downloads and installs the libraries listed in package.json.
npm start
Starts the development/run server.
After running the command, open the address shown in the terminal (usually something like http://localhost:3000) in your browser.
4. Go
MediumPrerequisites
go build ./...
Compiles the Go program into an executable.
go run .
Runs the Go program directly without a separate build step.
If the build finishes without errors, it worked. If you used go run ., check the terminal output.
5. Make
MediumPrerequisites
- Git Needed to download the project code from GitHub.
- Make Usually pre-installed on Linux/macOS. On Windows, install separately (e.g. via MSYS2 or WSL).
make
Compiles the code based on the generated build configuration to produce an executable.
If it finishes without errors, it worked. Try running the generated executable directly.
// repository documentation
Was this content helpful?
(0 ratings)