certmate

KO | EN

gitlite — search

#python #hacktoberfest #llm #golang #go #machine-learning #deep-learning #java #awesome-list #security #ios #openai #mcp #open-source

(★ 1,392)

Self-hosted certificate lifecycle management: issue, renew, discover, inventory and deploy TLS certificates across your infrastructure

File Explorer

Download Latest Version (.zip)

- - bug_report.md
  - config.yml
  - feature_request.md
- - announce-release.yml
  - ci.yml
  - codeql.yml
  - docker-multiplatform.yml
  - e2e-staging.yml
  - lint-emoji.yml
  - publish-chart.yml
  - publish-clients.yml
  - scorecard.yml
  - ui-tests.yml
- dependabot.yml
- FUNDING.yml
- PULL_REQUEST_TEMPLATE.md
- - Chart.yaml
  - README.md
  - values.yaml
- - pyproject.toml
  - README.md
- - pyproject.toml
  - README.md
- README.md
- certmate-cli.gif
- certmate-cli.mp4
- certmate-cli.tape
- README.md
- - api.md
  - architecture.md
  - ca-providers.md
  - compliance.md
  - deploy-hooks.md
  - dns-providers.md
  - docker.md
  - guide.md
  - index.md
  - installation.md
  - kubernetes.md
  - mcp.md
  - probes.md
  - README.md
  - testing.md
  - THEME_MIGRATION.md
- - api.md
  - architecture.md
  - ca-providers.md
  - compliance.md
  - deploy-hooks.md
  - dns-providers.md
  - docker.md
  - guide.md
  - index.md
  - installation.md
  - kubernetes.md
  - mcp.md
  - probes.md
  - README.md
  - testing.md
  - THEME_MIGRATION.md
- - api.md
  - architecture.md
  - ca-providers.md
  - compliance.md
  - deploy-hooks.md
  - dns-providers.md
  - docker.md
  - guide.md
  - index.md
  - installation.md
  - kubernetes.md
  - mcp.md
  - probes.md
  - README.md
  - testing.md
  - THEME_MIGRATION.md
- - api.md
  - architecture.md
  - ca-providers.md
  - compliance.md
  - deploy-hooks.md
  - dns-providers.md
  - docker.md
  - guide.md
  - index.md
  - installation.md
  - kubernetes.md
  - mcp.md
  - probes.md
  - README.md
  - testing.md
  - THEME_MIGRATION.md
- api.md
- architecture.md
- ca-providers.md
- compliance.md
- deploy-hooks.md
- discovery-inventory.md
- dns-providers.md
- docker.md
- guide.md
- index.md
- installation.md
- kubernetes.md
- mcp.md
- probes.en.md
- README.md
- testing.md
- THEME_MIGRATION.md
- walkthrough-01-dashboard.png
- walkthrough-02-create-form.png
- walkthrough-03-create-form-filled.png
- walkthrough-04-settings-dns.png
- walkthrough-05-settings-backup.png
- walkthrough-06-help.png
- walkthrough-07-help-report-issue.png
- walkthrough.webm
- webhooks.md
- index.js
- package-lock.json
- package.json
- test-smoke.js
- test-tools.js
- - __init__.py
  - client_certificates.py
  - models.py
  - resources.py
- - __init__.py
  - audit.py
  - audit_chain.py
  - audit_context.py
  - audit_prune.py
  - audit_signing.py
  - audit_sink.py
  - audit_verify.py
  - auth.py
  - ca_manager.py
  - cache.py
  - cert_adopt.py
  - cert_discovery.py
  - cert_inventory.py
  - cert_jobs.py
  - cert_probe.py
  - cert_service.py
  - certificates.py
  - client_certificates.py
  - constants.py
  - crypto_report.py
  - csr_handler.py
  - ct_monitor.py
  - deploy_targets.py
  - deployer.py
  - digest.py
  - dns_alias_hook.py
  - dns_providers.py
  - dns_strategies.py
  - dns_zone_discovery.py
  - events.py
  - factory.py
  - file_operations.py
  - inventory_view.py
  - metrics.py
  - notifier.py
  - ocsp_crl.py
  - oidc.py
  - private_ca.py
  - rate_limit.py
  - settings.py
  - shell.py
  - storage_backends.py
  - structured_logging.py
  - utils.py
  - zombie.py
- - __init__.py
  - auth_routes.py
  - backup_cache_routes.py
  - cert_routes.py
  - misc_routes.py
  - oidc_routes.py
  - routes.py
  - settings_routes.py
  - ui_routes.py
- __init__.py
- grafana-dashboard.json
- prometheus-alerts.yml
- prometheus-scrape.example.yml
- README.md
- check_providers.py
- check_wiki.py
- record_walkthrough.py
- release.sh
- reset_admin_password.py
- solidserver_hook.py
- theme_baseline.py
- theme_codemod.py
- - fontawesome.min.css
  - input.css
  - tailwind.min.css
- - apple-touch-icon.png
  - certmate_logo.png
  - certmate_logo_256.png
  - favicon.ico
- - alpine.min.js
  - certmate.js
  - client-certs.js
  - cmd-palette.js
  - dashboard.js
  - inventory.js
  - provider-icons.js
  - redoc.standalone.js
  - report-issue.js
  - settings-apikeys.js
  - settings-deploy.js
  - settings-notifications.js
  - settings-oidc.js
  - settings-probe.js
  - settings.js
  - setup-wizard.js
  - shortcuts.js
- - fa-brands-400.ttf
  - fa-brands-400.woff2
  - fa-regular-400.ttf
  - fa-regular-400.woff2
  - fa-solid-900.ttf
  - fa-solid-900.woff2
  - InterVariable.woff2
  - JetBrainsMono-Regular.woff2
- manifest.json
- - _client_certs.html
  - _client_create.html
  - _modal.html
  - settings_apikeys.html
  - settings_ca.html
  - settings_deploy.html
  - settings_dns.html
  - settings_general.html
  - settings_notifications.html
  - settings_oidc.html
  - settings_probe.html
  - settings_storage.html
  - settings_users.html
- 403.html
- activity.html
- base.html
- crypto_report.html
- help.html
- index.html
- inventory.html
- login.html
- notifications.html
- redoc.html
- settings.html
- setup.html
- __init__.py
- conftest.py
- e2e_support.py
- run_tests.sh
- test_587_deliberate_no_auth.py
- test_acme_challenge_serving.py
- test_acme_dns_checker_is_wired.py
- test_acme_dns_live.py
- test_acme_dns_native_hook.py
- test_advertised_endpoints_exist.py
- test_agentic_audit_trail.py
- test_api_docs_paths_resolve.py
- test_api_key_expiry_and_masking.py
- test_apikey_scope_escalation.py
- test_apikeys.py
- test_async_issuance.py
- test_async_issuance_e2e.py
- test_audit.py
- test_audit_backup_hardening.py
- test_audit_certbot_stderr_sanitizer.py
- test_audit_chain.py
- test_audit_chain_concurrent_writers.py
- test_audit_chain_streaming.py
- test_audit_checkpoint_crosscheck.py
- test_audit_client_cert_rbac.py
- test_audit_domain_validation_gaps.py
- test_audit_export_slice_anchor.py
- test_audit_hardening.py
- test_audit_log_rotation.py
- test_audit_notifications_and_acmedns_masking.py
- test_audit_phase3.py
- test_audit_prune.py
- test_audit_scope_filters.py
- test_audit_settings_mutating_routes.py
- test_audit_sink.py
- test_auth.py
- test_auth_manager_coverage.py
- test_auth_me_bypass.py
- test_auth_setup_mode.py
- test_azure_dns_credentials_format.py
- test_azure_keyvault_certificate_storage.py
- test_backup.py
- test_backup_encryption.py
- test_backup_excludes_certbot_scratch.py
- test_backup_includes_pki_and_audit.py
- test_backup_offsite_s3.py
- test_backup_restore_key_permissions.py
- test_backup_share_safe_excludes_key_material.py
- test_bcrypt_long_password_compat.py
- test_ca_endpoints_are_live.py
- test_ca_manager_eab.py
- test_ca_provider_test_endpoint.py
- test_ca_provider_wiring_consistency.py
- test_cert_adopt.py
- test_cert_creation_finally_block.py
- test_cert_discovery.py
- test_cert_discovery_manager.py
- test_cert_domain_dir_filter.py
- test_cert_info_cache_and_storage_summary.py
- test_cert_info_inproc_parse.py
- test_cert_inventory.py
- test_cert_lifecycle.py
- test_cert_probe.py
- test_cert_service.py
- test_certificates_concurrency_and_alias.py
- test_certificates_route_dedup.py
- test_check_renewals_summary.py
- test_ci_marker_coverage.py
- test_cli_audit_verify_output.py
- test_client_cert_path_traversal.py
- test_client_cert_renewal_preserves_identity.py
- test_client_cert_renewal_runaway.py
- test_client_certificate_lifecycle.py
- test_client_pfx_export.py
- test_clients_cli_unit.py
- test_clients_e2e.py
- test_clients_sdk_unit.py
- test_create_cert_io.py
- test_crypto_report.py
- test_crypto_report_api.py
- test_csp_img_src_airgap.py
- test_csr_handler_coverage.py
- test_ct_monitor.py
- test_custom_script_dns.py
- test_delete_certificate_clears_storage_backend.py
- test_dependabot_holds_the_stack.py
- test_dependency_collisions.py
- test_deploy_hook_param_expansion_bypass.py
- test_deploy_hook_timeout_coercion.py
- test_deploy_status_cache_invalidation.py
- test_deploy_targets.py
- test_deployer.py
- test_deployment_status_route.py
- test_desec_scaleway_provider.py
- test_diagnostics_snapshot.py
- test_digest.py
- test_disaster_recovery_round_trip.py
- test_dns_config_for_strategy_azure.py
- test_dns_manager_coverage.py
- test_dns_zone_discovery.py
- test_docs_navigation.py
- test_documented_numbers_are_real.py
- test_documented_pins_match_requirements.py
- test_documented_python_version.py
- test_domain_alias.py
- test_domain_lock_wait.py
- test_download_file_param.py
- test_duckdns_provider.py
- test_edgedns_credentials_format.py
- test_env_example_is_honest.py
- test_factory_directories.py
- test_factory_logging.py
- test_factory_scheduler.py
- test_flat_copy_is_published_as_one_unit.py
- test_frontend_provider_coverage.py
- test_frontend_state_regressions.py
- test_health_ready_e2e.py
- test_helm_chart.py
- test_help_browser_redirect.py
- test_insufficient_role_page.py
- test_inventory_adopt_api.py
- test_inventory_api.py
- test_inventory_view.py
- test_issue101_test_hook_error.py
- test_issue108_bearer_token_env.py
- test_issue113_azure_dns_ambiguous.py
- test_issue130_setup_wizard_loop.py
- test_issue152_deploy_history.py
- test_issue158_downgrade_protection.py
- test_issue162_bearer_token_preservation.py
- test_issue164_api_json_error_handlers.py
- test_issue165_deploy_history_silent_fail.py
- test_issue171_no_random_sleep_renew.py
- test_issue212_download_path_style.py
- test_issue398_json_key_format.py
- test_issue564_oidc_logout_follows_idp.py
- test_issue76_masked_token.py
- test_issue82_rfc2136_config.py
- test_key_options.py
- test_listing_settings_reuse.py
- test_log_file_rotation.py
- test_log_sanitizer.py
- test_log_stream_does_not_spin.py
- test_login_rate_limit_dict_bounded.py
- test_metadata_concurrent_writes.py
- test_metadata_corruption_handling.py
- test_metrics_endpoint_auth.py
- test_metrics_endpoint_context.py
- test_minor_queue_lot1_telemetry_and_status_codes.py
- test_nginx_conf_is_an_example.py
- test_notification_channels.py
- test_notifications_config_route.py
- test_notifications_routes.py
- test_notifier_coverage.py
- test_ocsp_crl_coverage.py
- test_oidc_disabled_and_role_sync.py
- test_oidc_manager.py
- test_oidc_routes.py
- test_oidc_settings_masking.py
- test_p2_quick_wins.py
- test_pages.py
- test_password_rehash_on_login.py
- test_patch_probe_config.py
- test_per_cert_renewal.py
- test_perf_quickwins.py
- test_pfx_download_api.py
- test_pfx_export.py
- test_pip_is_pinned_where_it_is_used.py
- test_private_ca_coverage.py
- test_privkey_pkcs1_export.py
- test_provider_availability_is_truthful.py
- test_provider_checker_is_wired.py
- test_provider_credential_key_contract.py
- test_provider_wiring_consistency.py
- test_rate_limit_bucket_bypass.py
- test_rate_limit_per_token.py
- test_rate_limits_config.py
- test_reissue_endpoint_e2e.py
- test_reissue_preserves_metadata.py
- test_reissue_san_edit.py
- test_release_gate.py
- test_renewal_job_settings_reuse.py
- test_renewal_private_ca_and_metadata.py
- test_renewal_process_lock.py
- test_renewal_reliability_hardening.py
- test_renewal_threshold_validation.py
- test_requirements_are_consistent.py
- test_requirements_sets.py
- test_reset_admin_password_script.py
- test_restore_repairs_certbot_lineage.py
- test_rfc2136_alias_hook.py
- test_s3_compatible_backend.py
- test_safe_file_write_unbound_temp_file.py
- test_san_domains.py
- test_scheduled_renewal_failure_notifies.py
- test_scheduler_status_health.py
- test_secret_key_env.py
- test_security_policy_matches_the_code.py
- test_session_revocation_on_user_change.py
- test_settings.py
- test_settings_atomic_update.py
- test_settings_list_secret_generic_restore.py
- test_settings_lost_update.py
- test_settings_masking_allowlist.py
- test_settings_request_scoped_cache.py
- test_settings_secret_preservation.py
- test_setup_mode_is_a_one_way_door.py
- test_sprint1_5_audit_followup.py
- test_sprint1_6_audit_polish.py
- test_sprint1_security.py
- test_staging_as_ca.py
- test_staging_settings_migration.py
- test_static_csp.py
- test_storage_backends_cloud_crud.py
- test_storage_backends_coverage.py
- test_storage_backends_live.py
- test_storage_fallback_health.py
- test_storage_migrate_endpoint.py
- test_storage_retry_contract.py
- test_storage_warning_lifecycle.py
- test_storage_warning_surfaced.py
- test_storage_wiring_consistency.py
- test_structured_logger_call_shape.py
- test_tls_probe_proxy.py
- test_tls_probe_timeout.py
- test_ui.py
- test_ui_hash_navigation.py
- test_ui_inventory.py
- test_ui_sso_failure_state.py
- test_unified_backup_one_pass_iteration.py
- test_version_consistency.py
- test_webhook_payload_template.py
- test_webhook_secret_preservation.py
- test_wiki_checker_is_wired.py
- test_wizard_dismissed.py
- test_zombie_scanner.py
.airgap.yml
.dockerignore
.env.example
.flake8
.gitignore
app.py
build-docker.sh
build-multiplatform.sh
certmate.service
certmate_logo.png
CODE_OF_CONDUCT.md
codecov.yml
conftest.py
CONTRIBUTING.md
debug-docker.sh
docker-compose.yml
Dockerfile
LICENSE
licensing.pdf
Makefile
nginx.conf.example
package-lock.json
package.json
pytest.ini
quick_test.sh
README.dockerhub.md
README.md
RELEASE_NOTES.md
requirements-aws-storage.txt
requirements-aws.txt
requirements-azure-storage.txt
requirements-azure.txt
requirements-extended.txt
requirements-gcp.txt
requirements-infisical-storage.txt
requirements-minimal.txt
requirements-storage-all.txt
requirements-test.txt
requirements-vault-storage.txt
requirements.txt
run-docker.sh
run-tests.sh
screenshot_1.png
SECURITY.md
setup.sh
start-certmate.sh
start.sh
tailwind.config.js
test-multiplatform.sh

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# CertMate - Certificate Lifecycle Management

**CertMate** is a self-hosted certificate lifecycle management platform: it issues and renews TLS certificates, **discovers the ones you did not issue**, keeps a single inventory of what exists across your estate — what is served where, who issued it, when it expires, which cryptography it uses — and deploys renewed certificates to where they are needed. It supports 29 DNS providers, runs its own private CA for internal names, keeps a tamper-evident audit trail of every operation, and exposes all of it through a REST API.

[![Live Demo](https://img.shields.io/badge/Live%20Demo-try%20it%20now-2563eb?logo=probot&logoColor=white)](https://demo.certmate.org)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/)
[![Docker](https://img.shields.io/badge/docker-ready-blue)](https://hub.docker.com/)
[![API Documentation](https://img.shields.io/badge/API-Swagger-green)](https://raw.githubusercontent.com/fabriziosalmi/certmate/main/)
[![PyPI - certmate-cli](https://img.shields.io/pypi/v/certmate-cli?label=certmate-cli&color=3775A9)](https://pypi.org/project/certmate-cli/)
[![CI](https://github.com/fabriziosalmi/certmate/actions/workflows/ci.yml/badge.svg)](https://github.com/fabriziosalmi/certmate/actions/workflows/ci.yml)
[![Build Multi-Platform Docker Images](https://github.com/fabriziosalmi/certmate/actions/workflows/docker-multiplatform.yml/badge.svg)](https://github.com/fabriziosalmi/certmate/actions/workflows/docker-multiplatform.yml)
[![CodeQL](https://github.com/fabriziosalmi/certmate/actions/workflows/codeql.yml/badge.svg)](https://github.com/fabriziosalmi/certmate/actions/workflows/codeql.yml)
[![OSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/fabriziosalmi/certmate/badge)](https://securityscorecards.dev/viewer/?uri=github.com/fabriziosalmi/certmate)
[![codecov](https://codecov.io/gh/fabriziosalmi/certmate/branch/main/graph/badge.svg)](https://codecov.io/gh/fabriziosalmi/certmate)
[![European Open Source](https://img.shields.io/badge/European%20Open%20Source-Catalogue-0046ad)](https://europeanopensource.eu/)
[![Listed on Hacker News](https://img.shields.io/badge/Listed%20on-Hacker%20News-F0652F)](https://news.ycombinator.com/item?id=44427452)
[![Listed on Reddit](https://img.shields.io/badge/Listed%20on-Reddit-FF4500)](https://www.reddit.com/r/selfhosted/comments/1lkvbcj/ssl_certificates_automation/)
[![Shared on Mastodon](https://img.shields.io/badge/Shared%20on-Mastodon-6364FF)](https://mastodon.social/@nixCraft/114799880340326467)
[![As seen on LinkedIn](https://img.shields.io/badge/As%20seen%20on-LinkedIn-0A66C2)](https://www.linkedin.com/posts/labyrinthlabs_github-fabriziosalmicertmate-ssl-certificate-activity-7434531689097248768-Ac63)
 
![screenshot1](https://raw.githubusercontent.com/fabriziosalmi/certmate/main/screenshot_1.png)

[Quick Start](#quick-start-with-docker) • [CLI](#command-line-interface) • [Documentation](#documentation) • [Installation](#installation-methods) • [DNS Providers](#supported-dns-providers) • [CA Providers](docs/ca-providers.md) • [Storage Backends](#certificate-storage-configuration) • [Backup and Recovery](#backup-and-recovery) • [API Reference](#api-usage)

</div>

---

## Ecosystem

CertMate is the open-source core of a small, focused toolset:

- **[certmate-tools](https://github.com/fabriziosalmi/certmate-tools)** — free, privacy-first, client-side TLS / certificate / ACME diagnostics (runs entirely in your browser).
- **[certmate-agent](https://github.com/fabriziosalmi/certmate-agent)** — conversational assistant: a local LLM mapped 1:1 to CertMate's REST API, with RAG over the docs.
- **[nis2-public](https://github.com/fabriziosalmi/nis2-public)** — NIS2 continuous posture management & remediation.

**Enterprise / high-scale** — multi-tenant, mTLS, white-label and NIS2-aligned deployments are available through *CertMate-ng* (source-available, BSL 1.1, EU-built). For access or a deployment discussion, email **fabrizio.salmi@gmail.com**.

---

## Command-line interface

The whole certificate lifecycle from your terminal — `pip install certmate-cli`:

![CertMate CLI — the full SSL certificate lifecycle from the terminal](https://raw.githubusercontent.com/fabriziosalmi/certmate/main/demo/certmate-cli.gif)

```bash
pip install certmate-cli

export CERTMATE_URL=https://certmate.example.com
export CERTMATE_TOKEN=...                 # omit on a fresh (setup-mode) instance

certmate health
certmate cert create app.example.com --dns cloudflare --wait   # issue, block until live
certmate cert ls
certmate cert info app.example.com
certmate audit verify
```

`certmate-cli` is a thin layer over **`certmate-sdk`** (`pip install certmate-sdk`) — a small,
`httpx`-based Python client for the same REST API the web UI and MCP server drive. Both are
first-party, live in [`clients/`](clients/), and are published to PyPI. The clip above is a real
issuance over DNS-01 (Let's Encrypt staging); see [`demo/`](demo/).

---

## AI agents (MCP server)

CertMate ships a first-party **Model Context Protocol** server, so an assistant like Claude can
drive the same REST API the web UI and the CLI use — with the same auth and the same audit trail.
It lives in [`mcp/`](mcp/), is Node.js (>= 20), and exposes **16 tools**: inventory and status
(`certmate_list_certificates`, `certmate_get_certificate`, `certmate_diagnostics`,
`certmate_get_activity`, …), lifecycle operations (`certmate_create_certificate`,
`certmate_renew_certificate`, `certmate_get_job`, `certmate_set_auto_renew`, …), and delivery
(`certmate_download_certificate`, `certmate_deploy_certificate`).

```bash
cd mcp && npm install
```

```jsonc
// claude_desktop_config.json — or any MCP-capable client
{
  "mcpServers": {
    "certmate": {
      "command": "node",
      "args": ["/path/to/certmate/mcp/index.js"],
      "env": {
        "CERTMATE_URL": "https://certmate.example.com",
        "CERTMATE_TOKEN": "<a scoped API key with is_agent: true>"
      }
    }
  }
}
```

**Give it a scoped key, not your admin token.** A key created with `is_agent: true` (a checkbox
under Settings → API Keys, or `is_agent` in `POST /api/keys`) makes every action the agent takes
land in the audit chain as `actor.kind="agent"` rather than being indistinguishable from a human
operator — which is the difference between an audit trail and a rumour. Scope the key to the
domains the agent is allowed to touch.

Full tool reference, attribution model and safety notes: **[docs/mcp.md](docs/mcp.md)**.

---

## Why CertMate?

CertMate solves the complexity of SSL certificate management in modern distributed architectures. Whether you're running a single application or managing certificates across multiple datacenters, CertMate provides:

- **Zero-Downtime Automation** - Certificates renew automatically 30 days before expiry, with deploy hooks to reload services
- **Multi-Cloud Support** - Works with two dozen+ DNS providers (Cloudflare, AWS, Azure, GCP, Akamai Edge DNS, Hetzner, Porkbun, GoDaddy, and more — see [docs/dns-providers.md](docs/dns-providers.md) for the full list)
- **Enterprise-Ready** - RBAC, scoped API keys, Docker, Kubernetes, REST API, and monitoring built-in
- **Simple Integration** - One-URL certificate downloads for easy automation
- **Security-First** - Role-based access control, scoped API keys, audit logging, HMAC-signed webhooks
- **Unified Backup System** - Atomic backups of settings and certificates ensuring data consistency
- **Real-Time Dashboard** - SSE-powered live updates, command palette, keyboard shortcuts, dark mode

## Key Features

### **Certificate Management**
- **Multiple CA Providers** - Support for Let's Encrypt, ZeroSSL, Google Trust Services, Actalis, DigiCert ACME, SSL.com, and Private CAs
- **Let's Encrypt Integration** - Free, automated SSL certificates, with the staging environment available as a dedicated CA entry for testing
- **DigiCert ACME Support** - Enterprise-grade certificates with External Account Binding (EAB)
- **Actalis Support** - Free 90-day DV certificates from a European CA via ACME with EAB
- **Private CA Support** - Internal/corporate CAs with custom trust bundles and ACME compatibility
- **Wildcard Support** - Single certificate for `*.example.com` and `example.com`
- **Multi-Domain Certificates** - SAN certificates for multiple domains
- **DNS Alias via CNAME Delegation** - Delegate ACME DNS validation to an alternative domain using standard CNAME records
- **Automatic Renewal** - Smart renewal 30 days before expiry
- **Certificate Validation** - Real-time SSL certificate status checking
- **Per-Certificate CA Selection** - Choose different CAs for different certificates
- **Zombie Certificate Scanner** - filesystem scanner to identify and clean up orphan ("zombie") certificates no longer tracked in the active configuration

### **Multi-DNS Provider Support**
- **Multi-Account Support** - Manage multiple accounts per provider for enterprise environments
- **Cloudflare** - Global CDN with edge locations worldwide (Multi-Account)
- **AWS Route53** - Amazon's scalable DNS service (Multi-Account)
- **Azure DNS** - Microsoft's cloud DNS solution (Multi-Account)
- **Google Cloud DNS** - Google's high-performance DNS (Multi-Account)
- **DigitalOcean** - Cloud infrastructure DNS (Multi-Account)
- **PowerDNS** - Open-source DNS server with REST API (Multi-Account)

### **Enterprise Features**
- **Role-Based Access Control** - Three-tier RBAC with viewer, operator, and admin roles
- **Scoped API Keys** - Create, revoke, and manage API keys with per-key role and optional expiration
- **Multi-Account Management** - Support multiple accounts per DNS provider for enterprise workflows
- **REST API** - Complete programmatic control with Swagger/OpenAPI docs
- **Web Dashboard** - Modern, responsive UI built with Tailwind CSS and Alpine.js
- **Setup Wizard** - Guided first-run configuration for DNS, CA, and authentication
- **Real-Time Updates** - Server-Sent Events (SSE) push live status to the dashboard
- **Docker Ready** - Full containerization with Docker Compose
- **Kubernetes Compatible** - Deploy in any Kubernetes cluster
- **Monitoring Integration** - Health checks, Prometheus metrics, and structured JSON logging

### **Backup and Recovery**
- **Unified Backups** - Atomic snapshots of both settings and certificates ensuring data consistency
- **Automatic Backups** - Settings and certificates backed up automatically on changes
- **Manual Backup Creation** - On-demand backup creation via web UI or API
- **Comprehensive Coverage** - Backs up DNS configurations, certificates, and application settings
- **Retention Management** - 50 most recent archives per type, and nothing older than 30 days — both constants; keep disaster-recovery archives off the host
- **Easy Restore** - Simple restore process from any backup point with atomic consistency
- **Download Support** - Export backups for external storage and disaster recovery

### **Certificate Storage Backends**
- **Local Filesystem** - Default secure local storage with proper file permissions (600/700)
- **Azure Key Vault** - Enterprise-grade secret management with Azure integration and HSM protection
- **AWS Secrets Manager** - Scalable secret storage with AWS ecosystem integration and cross-region replication
- **HashiCorp Vault** - Industry-standard secret management with versioning, audit logging, and fine-grained policies
- **Infisical** - Modern open-source secret management with team collaboration and end-to-end encryption
- **S3-Compatible Object Storage** - One backend for any S3 endpoint via a configurable endpoint URL (Hetzner, Contabo, OVHcloud, Scaleway, Exoscale, Wasabi, MinIO, AWS) — ideal for EU-sovereign object storage; no extra dependency
- **Pluggable Architecture** - Easy to extend with additional storage backends
- **Migration Support** - Seamless migration between storage backends without downtime
- **Backward Compatibility** - Existing installations continue working without changes

### **Notifications & Automation**
- **Multi-Channel Notifications** - Email (SMTP), Slack, Discord, Telegram, ntfy, Gotify, and generic webhooks
- **Webhook HMAC Signatures** - SHA-256 signed payloads for secure webhook verification
- **Deploy Hooks** - Post-issuance shell commands to reload Nginx/Apache or run custom scripts
- **Weekly Digest** - Scheduled email summary of certificate status and upcoming renewals
- **SSE Real-Time Events** - Live push updates for certificate operations and deploy hook results

### **Security & Compliance**
- **Role-Based Access Control** - Viewer, operator, and admin roles with hierarchical permissions
- **Scoped API Keys** - Create keys with specific roles and optional expiration dates
- **Bearer Token Authentication** - Secure API access control
- **File Permissions** - Proper certificate file security (600/700)
- **Audit Logging** - Complete certificate lifecycle tracking with timeline view
- **Environment Variables** - Secure credential management
- **Rate Limit Handling** - Let's Encrypt rate limit awareness
- **Log Sanitizer** - Automatically redacts sensitive parameters, private keys, and API tokens from application logs

### **User Interface**
- **Command Palette** - Cmd+K / Ctrl+K quick search and navigation
- **Keyboard Shortcuts** - Power-user shortcuts for navigation and common actions
- **Dark Mode** - System-aware dark/light theme toggle
- **Mobile-Friendly** - Responsive layout with bottom tab bar on small screens
- **Activity Timeline** - Chronological view of all certificate and system events

### **Developer Experience**
- **One-URL Downloads** - Simple certificate retrieval for automation (`/api/certificates/{domain}/download`)
- **Individual Component Downloads** - Fetch cert, key, chain, or fullchain separately
- **Multiple Output Formats** - PEM, ZIP, individual files
- **SDK Examples** - Python, Bash, Ansible, Terraform examples
- **Webhook Support** - Certificate lifecycle notifications with HMAC verification
- **Deploy Hook API** - Configure and test post-issuance hooks via REST API
- **Backup API** - Programmatic backup creation and restoration
- **Swagger & ReDoc** - Interactive API documentation at `/docs/` and `/redoc/`
- **Model Context Protocol (MCP) Server** - Built-in Node.js MCP server providing tools for agentic AI assistants to manage certificates and run diagnostics

## Supported DNS Providers

CertMate supports a wide range of DNS providers through Let's Encrypt DNS-01 challenge via individual certbot plugins that provide reliable, well-tested DNS challenge support. The complete list is in the table below. **Multi-account support** is available for major providers, enabling enterprise-grade deployments with separate accounts for production, staging, and disaster recovery.

| Provider               | Credentials Required          | Multi-Account | Use Case                        | Availability |
| ---------------------- | ----------------------------- | ------------- | ------------------------------- | ------------ |
| **Cloudflare**         | API Token                     | **Yes**       | Global CDN, Free tier available | **Stable** |
| **AWS Route53**        | Access Key, Secret Key        | **Yes**       | AWS infrastructure, Enterprise  | **Stable** |
| **Azure DNS**          | Service Principal credentials | **Yes**       | Microsoft ecosystem             | **Stable** |
| **Google Cloud DNS**   | Service Account JSON          | **Yes**       | Google Cloud Platform           | **Stable** |
| **DigitalOcean**       | API Token                     | **Yes**       | Cloud infrastructure            | **Stable** |
| **PowerDNS**           | API URL, API Key              | **Yes**       | Self-hosted, On-premises        | **Separate install** |
| **EfficientIP SOLIDserver** | Host, API Credentials     | **Yes**       | Enterprise DDI / Smart Architecture | **Stable** |
| **RFC2136**            | Nameserver, TSIG Key/Secret   | **Yes**       | Standard DNS update protocol    | **Stable** |
| **Linode** (Akamai Connected Cloud) | API Key             | Single        | Cloud hosting                   | **Stable** |
| **Akamai Edge DNS**    | EdgeGrid (.edgerc) credentials| Single        | Enterprise managed DNS          | **Stable** |
| **Gandi**              | API Token                     | Single        | Domain registrar                | **Stable** |
| **OVH**                | API Credentials               | Single        | European hosting                | **Stable** |
| **Namecheap**          | Username, API Key             | Single        | Domain registrar                | **Unavailable** |
| **Vultr**              | API Key                       | Single        | Global cloud infrastructure     | **Stable** |
| **DNS Made Easy**      | API Key, Secret Key           | Single        | Enterprise DNS management       | **Stable** |
| **NS1**                | API Key                       | Single        | Intelligent DNS platform        | **Stable** |
| **Hetzner** (legacy DNS) | API Token                   | Single        | European cloud hosting          | **Stable** |
| **Hetzner Cloud**      | API Token                     | Single        | Hetzner Cloud DNS (hcloud)      | **Stable** |
| **Porkbun**            | API Key, Secret Key           | Single        | Domain registrar with DNS       | **Stable** |
| **GoDaddy**            | API Key, Secret               | Single        | Popular domain registrar        | **Stable** |
| **Hurricane Electric** | Username, Password            | Single        | Free DNS hosting                | **Extended image** |
| **Dynu**               | API Token                     | Single        | Dynamic DNS service             | **Extended image** |
| **ArvanCloud**         | API Key                       | Single        | Iranian cloud provider          | **Stable** |
| **Infomaniak**         | API Token                     | Single        | Swiss ISP & cloud provider      | **Stable** |
| **ACME-DNS**           | JSON Config                   | Single        | Generic ACME-DNS server         | **Stable** |
| **Scaleway**           | API Token (secret key)        | Single        | European cloud (EU-sovereign)   | **Separate install** |
| **deSEC**              | API Token                     | Single        | Free, non-profit DNSSEC DNS     | **Stable** |
| **DuckDNS**            | Token                         | Single        | Free dynamic DNS                | **Stable** |
| **Custom Script**      | User-provided hook scripts    | Single        | Any provider via custom hooks   | **Stable** |

**What the availability column means.** Until now it said `Stable` on all
twenty-nine rows, which is not a status — a column with one value cannot be
wrong about any particular provider, and four of them were wrong:

| Value | Meaning |
| ----- | ------- |
| **Stable** | The plugin is pinned in `requirements.txt`, so it is in the default image. |
| **Extended image** | Pinned in `requirements-extended.txt`. Build with `--build-arg EXTRA_REQUIREMENTS=requirements-extended.txt`. |
| **Separate install** | Not shipped in any image. `pip install` it yourself, in its own environment where noted. |
| **Unavailable** | No usable plugin exists for the supported stack. CertMate can be configured for it, but issuance will fail. |

`Namecheap` is the **Unavailable** one: the only release on PyPI is
`certbot-dns-namecheap` 1.0.0, an alpha targeting Python 2.7-3.8, incompatible
with certbot 2.x on Python 3.12 — as `NamecheapStrategy`'s own docstring has
said all along. Use ACME-DNS or the custom-script hook for Namecheap domains.
`Scaleway` is alpha (0.0.7, declares Python 2.7 support) and `PowerDNS` pulls
`dns-lexicon<=3.5.6`, which conflicts with the rest of the extended set — hence
its own environment.

### Provider Categories

- **Enterprise Multi-Account**: Cloudflare, AWS Route53, Azure DNS, Google Cloud DNS, DigitalOcean, PowerDNS, RFC2136
- **Cloud Providers**: AWS Route53, Azure DNS, Google Cloud DNS, DigitalOcean, Linode, Akamai Edge DNS, Vultr, Hetzner
- **Enterprise DNS**: Cloudflare, DNS Made Easy, NS1, PowerDNS, EfficientIP SOLIDserver
- **Domain Registrars**: Gandi, OVH, Namecheap, Porkbun, GoDaddy 
- **European Providers**: OVH, Gandi, Hetzner
- **Free Services**: Hurricane Electric, Dynu
- **Standard Protocols**: RFC2136 (for BIND and compatible servers)

### Multi-Account Benefits

For supported providers, you can configure multiple accounts to enable:

- **Environment Separation**: Different accounts for production, staging, and development
- **Multi-Region Management**: Separate accounts for different geographical regions
- **Team Isolation**: Department-specific accounts with tailored permissions
- **Disaster Recovery**: Backup accounts for high-availability scenarios
- **Permission Scoping**: Accounts with minimal required permissions for security

> **Detailed Setup Instructions**: See [DNS Providers Guide](docs/dns-providers.md) for provider-specific configuration. 
> **Step-by-Step Installation**: See [Installation Guide](docs/installation.md) for complete setup guide. 
> **Multi-Account Examples**: See [DNS Providers Guide](docs/dns-providers.md#multi-account-support) for enterprise configuration examples.

## Quick Start with Docker

Get CertMate running in under 5 minutes with Docker Compose:

### Prerequisites
- Docker 20.10+
- Docker Compose 2.0+
- Domain with DNS managed by supported provider

### 1. Clone and Setup

```bash
# Clone the repository
git clone https://github.com/fabriziosalmi/certmate.git
cd certmate

# Copy environment template
cp .env.example .env
```

### 2. Configure Environment

Edit `.env` file with your credentials:

```bash
# Recommended for any network-exposed deployment: API Security.
# Auto-generated if unset, but a not-yet-onboarded instance serves the
# first-run setup bypass to anyone who can reach it — set this (or bind to
# localhost) before exposing CertMate. When set, the first-run screen asks
# you to paste this same token once to create the initial admin.
API_BEARER_TOKEN=your_super_secure_api_token_here_change_this

# DNS Provider Configuration (choose one or multiple)

# Option 1: Cloudflare (Recommended for beginners)
CLOUDFLARE_TOKEN=your_cloudflare_api_token_here

# Option 2: AWS Route53
# AWS_ACCESS_KEY_ID=your_aws_access_key
# AWS_SECRET_ACCESS_KEY=your_aws_secret_key
# AWS_DEFAULT_REGION=us-east-1

# Option 3: Azure DNS
# AZURE_SUBSCRIPTION_ID=your_azure_subscription_id
# AZURE_RESOURCE_GROUP=your_resource_group
# AZURE_TENANT_ID=your_tenant_id
# AZURE_CLIENT_ID=your_client_id
# AZURE_CLIENT_SECRET=your_client_secret

# Option 4: Google Cloud DNS
# GOOGLE_PROJECT_ID=your_gcp_project_id
# GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json

# Option 5: PowerDNS
# POWERDNS_API_URL=https://your-powerdns-server:8081
# POWERDNS_API_KEY=your_powerdns_api_key

# Optional: Application Settings
SECRET_KEY=your_flask_secret_key_here
FLASK_ENV=production
PORT=8000
# Note: there is no HOST variable. The container binds 0.0.0.0 in its own
# namespace — publish it as 127.0.0.1:8000:8000 to reach it on loopback only,
# and front it with a reverse proxy for external access.
```

> **Storage Backends**: By default, certificates are stored locally. For enterprise deployments, you can configure Azure Key Vault, AWS Secrets Manager, HashiCorp Vault, Infisical, or any S3-compatible object storage via the web interface after startup. See [Storage Backends](#certificate-storage-configuration) for details.

> **Backup Best Practices**: CertMate includes a unified backup system that creates atomic snapshots of both settings and certificates. After setup, create a **disaster-recovery** backup — `POST /api/backups/create` with `{"include_secrets": true}` and `CERTMATE_BACKUP_PASSPHRASE` set — and keep it off the host. The default (and every automatic) backup is *share-safe*: credentials masked and no private keys, so it cannot restore an instance on its own.

### 3. Deploy

```bash
# Start all services
docker-compose up -d

# Check status
docker-compose ps

# View logs
docker-compose logs -f certmate
```

### 4. Access CertMate

| Service                  | URL                          | Description                           |
| ------------------------ | ---------------------------- | ------------------------------------- |
| **Web Dashboard**        | http://localhost:8000        | Main certificate management interface |
| **API Documentation**    | http://localhost:8000/docs/  | Interactive Swagger/OpenAPI docs      |
| **Alternative API Docs** | http://localhost:8000/redoc/ | ReDoc documentation                   |
| **Health Check**         | http://localhost:8000/health | Service health monitoring             |

### 5. Create Your First Certificate

Using the Web Interface:
1. Navigate to http://localhost:8000
2. Go to Settings and configure your DNS provider
3. Add your domain (e.g. `example.com`)
4. Click "Create Certificate"

Using the API:
```bash
curl -X POST "http://localhost:8000/api/certificates/create" \
 -H "Authorization: Bearer your_api_token_here" \
 -H "Content-Type: application/json" \
 -d '{"domain": "example.com"}'
```

## Installation Methods

Choose the installation method that best fits your environment:

### Docker (Recommended)
Perfect for production deployments with isolation and easy scaling. **Supports multiple architectures**: AMD64 (Intel/AMD), ARM64 (Apple Silicon, ARM servers), and ARM v7 (Raspberry Pi).

```bash
# Quick start with Docker Compose
git clone https://github.com/fabriziosalmi/certmate.git
cd certmate
cp .env.example .env
# Edit .env with your configuration
docker-compose up -d
```

**Multi-Platform Support:**
```bash
# Build for multiple architectures (ARM64 + AMD64)
./build-multiplatform.sh

# Build and push to Docker Hub for all platforms
./build-multiplatform.sh -r YOUR_DOCKERHUB_USERNAME -p

# Use pre-built multi-platform image (bound to localhost; put it behind a
# reverse proxy and enable authentication before exposing it externally)
docker run --platform linux/arm64 -d --name certmate --env-file .env -p 127.0.0.1:8000:8000 fabriziosalmi/certmate:latest
```

> **Multi-Platform Guide**: See [Docker Guide](docs/docker.md) for comprehensive multi-architecture setup instructions.

### Python Virtual Environment
Ideal for development and testing environments.

```bash
# Create and activate virtual environment
python3 -m venv certmate-env
source certmate-env/bin/activate # On Windows: certmate-env\Scripts\activate

# Install dependencies
git clone https://github.com/fabriziosalmi/certmate.git
cd certmate
pip install -r requirements.txt

# Set environment variables
export API_BEARER_TOKEN="your_token_here"
export CLOUDFLARE_TOKEN="your_cloudflare_token"

# Run the application
python app.py
```

### Kubernetes
For container orchestration and high availability deployments.

```yaml
# Example Kubernetes deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: certmate
spec:
  replicas: 1
  selector:
    matchLabels:
      app: certmate
  template:
    metadata:
      labels:
        app: certmate
    spec:
      containers:
        - name: certmate
          image: certmate:latest
          ports:
            - containerPort: 8000
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: "1"
              memory: 1536Mi
          env:
            - name: API_BEARER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: certmate-secrets
                  key: api-token
            - name: CERTMATE_CERT_INFO_CACHE_TTL
              value: "60"
          volumeMounts:
            - name: certificates
              mountPath: /app/certificates
      volumes:
        - name: certificates
          persistentVolumeClaim:
            claimName: certmate-certificates
```

For production sizing, OOM troubleshooting, and a `kubectl patch` example, see
[Kubernetes Production Notes](docs/kubernetes.md).

### System Package Installation
For system-wide installation on Linux distributions.

```bash
# Install system dependencies (Ubuntu/Debian)
sudo apt update
sudo apt install python3 python3-pip python3-venv certbot openssl

# Clone and install
git clone https://github.com/fabriziosalmi/certmate.git
sudo mv certmate /opt/
cd /opt/certmate
sudo pip3 install -r requirements.txt

# Create systemd service (see Service Setup section below for detailed instructions)
sudo cp certmate.service /etc/systemd/system/
sudo systemctl enable certmate
sudo systemctl start certmate
```

> **Detailed Instructions**: See [Installation Guide](docs/installation.md) for complete setup guides for each method.

## Service Setup

For production deployments, CertMate should run as a system service. This section provides comprehensive instructions for setting up CertMate with systemd on Linux distributions.

### Prerequisites

- Linux system with systemd
- Python 3.12
- Root/sudo access

### 1. Create Dedicated System User

Create a dedicated user for running CertMate:

```bash
# Create system user and group
sudo useradd --system --shell /bin/false --home-dir /opt/certmate --create-home certmate

# Set proper ownership
sudo chown -R certmate:certmate /opt/certmate
```

### 2. Prepare Application Directory

Set up the application in `/opt/certmate`:

```bash
# If not already done, clone the repository
git clone https://github.com/fabriziosalmi/certmate.git
sudo mv certmate /opt/
cd /opt/certmate

# Create Python virtual environment
sudo -u certmate python3 -m venv venv
sudo -u certmate ./venv/bin/pip install -r requirements.txt

# Create necessary directories
sudo -u certmate mkdir -p certificates data
```

### 3. Configure Environment Variables

Create environment file for the service:

```bash
# Create environment file
sudo tee /opt/certmate/.env > /dev/null <<EOF
# SECURITY: Change this token!
API_BEARER_TOKEN=your_super_secure_api_token_here_change_this

# Optional: Set the port (the bind address is not configurable here —
# publish the container on 127.0.0.1 if you want loopback only)
PORT=8000
EOF

# Set proper permissions
sudo chown certmate:certmate /opt/certmate/.env
sudo chmod 600 /opt/certmate/.env
```

### 4. Install systemd Service

Install and configure the systemd service:

```bash
# Copy service file
sudo cp /opt/certmate/certmate.service /etc/systemd/system/

# Reload systemd configuration
sudo systemctl daemon-reload

# Enable service to start on boot
sudo systemctl enable certmate

# Start the service
sudo systemctl start certmate
```

### 5. Verify Service Status

Check that the service is running correctly:

```bash
# Check service status
sudo systemctl status certmate

# View recent logs
sudo journalctl -u certmate --lines=50

# Follow logs in real-time
sudo journalctl -u certmate -f
```

### 6. Service Management Commands

Common commands for managing the CertMate service:

```bash
# Start service
sudo systemctl start certmate

# Stop service
sudo systemctl stop certmate

# Restart service
sudo systemctl restart certmate

# Reload service configuration
sudo systemctl reload certmate

# Check if service is enabled
sudo systemctl is-enabled certmate

# Check if service is active
sudo systemctl is-active certmate

# Disable service from starting on boot
sudo systemctl disable certmate
```

### 7. File Permissions

Ensure proper file permissions for security:

```bash
# Set ownership
sudo chown -R certmate:certmate /opt/certmate

# Set directory permissions
sudo chmod 755 /opt/certmate
sudo chmod 750 /opt/certmate/certificates /opt/certmate/data

# Set file permissions
sudo chmod 644 /opt/certmate/*.py /opt/certmate/*.md
sudo chmod 600 /opt/certmate/.env
sudo chmod 755 /opt/certmate/venv/bin/*
```

### Security Notes

- **API Bearer Token**: Always change the default API bearer token in `/opt/certmate/.env`
- **File Permissions**: The service runs with restricted permissions and limited filesystem access
- **Network Access**: The service binds to `0.0.0.0:8000` by default - consider using a reverse proxy for production
- **Environment File**: The `.env` file contains sensitive data and should be readable only by the `certmate` user
- **Certificates**: Generated certificates are stored in `/opt/certmate/certificates` with restricted access

### Troubleshooting Service Setup

If the service fails to start:

1. **Check service status**: `sudo systemctl status certmate`
2. **View logs**: `sudo journalctl -u certmate --lines=100`
3. **Verify permissions**: Ensure the `certmate` user can read all necessary files
4. **Test manually**: `sudo -u certmate /opt/certmate/venv/bin/python /opt/certmate/app.py`
5. **Check dependencies**: `sudo -u certmate /opt/certmate/venv/bin/python validate_dependencies.py`

For more detailed installation instructions, see the [Installation Guide](docs/installation.md).

## Single Sign-On (OIDC/SSO)

CertMate supports authenticating users against an external OpenID Connect provider (Keycloak, Authentik, Okta, Google Workspace, Microsoft Entra, ...) using the Authorization Code + PKCE flow. SSO is **additive**: local username/password login and API keys keep working alongside it.

### Configuring an IdP

1. Open the CertMate UI as an admin → **Settings → SSO**.
2. Set the **Issuer URL** to the IdP's base URL (the path before `/.well-known/openid-configuration`). For example:
   - Keycloak: `https://idp.example.com/realms/main`
   - Authentik: `https://idp.example.com/application/o/certmate/`
   - Google: `https://accounts.google.com`
3. Fill in the **Client ID** and **Client Secret** issued by the IdP for the CertMate application.
4. In the IdP, register CertMate's callback URL as a valid redirect URI:

```
   https://your-certmate.example.com/api/auth/oidc/callback
   ```

5. Pick the claim names your IdP uses for username (`preferred_username` by default), email (`email`), and role (`groups`). Add **Role mappings** to translate IdP group/role claim values to CertMate roles — first match wins. Anything that doesn't match falls back to the configured **Default role** (`viewer` recommended).
6. Toggle **Enable OIDC/SSO** on and save. Visit `/login` in a new browser session to see the **Sign in with <Provider>** button.

### Role mapping example

For a Keycloak realm that exposes a `groups` claim, the configuration block in `settings.json` looks like:

```json
"oidc": {
  "enabled": true,
  "provider_name": "Keycloak",
  "issuer_url": "https://idp.example.com/realms/main",
  "client_id": "certmate",
  "client_secret": "********",
  "scopes": ["openid", "email", "profile", "groups"],
  "role_claim": "groups",
  "role_mappings": [
    { "claim_value": "certmate-admins",    "role": "admin" },
    { "claim_value": "certmate-operators", "role": "operator" }
  ],
  "default_role": "viewer",
  "auto_create_users": true,
  "link_by_email": true,
  "sync_role_on_login": true
}
```

### Provisioning and linking

- **Just-in-time provisioning** (`auto_create_users`) creates a CertMate user row on first login. The row has an empty password hash so JIT-provisioned SSO accounts cannot fall back to local login.
- **Email linking** (`link_by_email`) detects collisions with existing local users and merges identities — the user keeps their existing role and **their existing password hash**, so a local-then-linked account can still log in either way during a rollout. Disable `link_by_email` if you want JIT-only provisioning with no local-password fallback.
- Subject (`sub` + `iss`) lookup always wins over email matching, so an already-linked SSO user is never accidentally re-merged when their IdP email changes.
- **Role sync** (`sync_role_on_login`, default `true`) re-derives the role from the current claims on every login, so removing someone from an admin group in the IdP demotes them in CertMate too. Set it to `false` when the IdP only authenticates and roles are managed inside CertMate — an admin promoting someone by hand then survives their next login.
- A **disabled** CertMate user is refused at SSO login exactly as at local login: disabling an account locks it out regardless of how it authenticates.

### Security

- PKCE (S256) is enforced for every flow regardless of client type.
- The id_token's signature, audience, issuer, expiry and nonce are validated server-side by Authlib using the IdP's published JWKS.
- `client_secret` is masked (`********`) in every GET response and round-tripped safely through the Settings UI.
- The `oidc` settings block is on the bulk-POST reject list — only the dedicated `/api/auth/oidc/settings` endpoint can mutate it, with full audit.
- Failed callbacks count against the same per-IP rate limit as local login.

## API Usage

CertMate provides a comprehensive REST API for programmatic certificate management. All endpoints require Bearer token authentication.

### Authentication

Include the Authorization header in all API requests:

```bash
Authorization: Bearer your_api_token_here
```

### Core Endpoints

#### Health & Status
```bash
# Health check
GET /health

# API documentation
GET /docs/ # Swagger UI
GET /redoc/ # ReDoc documentation

# Prometheus/OpenMetrics monitoring
GET /metrics # Prometheus-compatible metrics
GET /api/metrics # JSON metrics summary
```

#### Settings Management
```bash
# Get current settings
GET /api/settings
Authorization: Bearer your_token_here

# Update settings
POST /api/settings
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "dns_provider": "cloudflare",
 "dns_providers": {
 "cloudflare": {
 "api_token": "your_cloudflare_token"
 }
 },
 "domains": [{
 "domain": "example.com",
 "dns_provider": "cloudflare"
 }
 ],
 "email": "admin@example.com",
 "auto_renew": true
}
```

#### Certificate Management
```bash
# List all certificates
GET /api/certificates
Authorization: Bearer your_token_here

# Create new certificate
POST /api/certificates/create
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "domain": "example.com",
 "dns_provider": "cloudflare", # Optional, uses default from settings
 "account_id": "production" # Optional, specify which account to use
}

# Create SAN certificate (multiple domains)
POST /api/certificates/create
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "domain": "example.com",
 "san_domains": ["www.example.com", "mail.example.com", "api.example.com"],
 "dns_provider": "cloudflare"
}
# This creates a single certificate covering all specified domains.
# The primary domain is "example.com" and san_domains are additional 
# Subject Alternative Names included in the certificate.
# Note: All domains must use the same DNS provider for validation.

# Create certificate with specific account
POST /api/certificates/create
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "domain": "staging.example.com",
 "dns_provider": "cloudflare",
 "account_id": "staging"
}

# Create certificate with DNS alias via CNAME delegation
POST /api/certificates/create
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "domain": "example.com",
 "dns_provider": "cloudflare",
 "domain_alias": "validation.example.org"
}
# DNS alias validation works via CNAME delegation. Before issuing, create
# a CNAME record in your DNS zone:
#
#   _acme-challenge.example.com  CNAME  _acme-challenge.validation.example.org
#
# CertMate creates the TXT record on the provider-managed alias name, and
# Let's Encrypt follows the CNAME chain during the DNS-01 challenge.
# Alias mode is supported for CertMate's first-class DNS providers; generic
# fallback providers are rejected until a dedicated adapter exists.
# This is useful when:
# - The primary domain's DNS does not support an API
# - You want to centralize ACME validations on a dedicated domain
# - There are DNS restrictions on the primary zone

# Renew certificate
POST /api/certificates/example.com/renew
Authorization: Bearer your_token_here

# Download certificate bundle as JSON
GET /api/certificates/example.com/download?format=json
Authorization: Bearer your_token_here

# Check certificate deployment status
GET /api/certificates/example.com/deployment-status
Authorization: Bearer your_token_here

# Scan filesystem for orphan ("zombie") certificates no longer tracked by Certbot
POST /api/certificates/zombies/scan
Authorization: Bearer your_token_here
```

#### Multi-Account Management
```bash
# Add multiple accounts for a provider
POST /api/dns/cloudflare/accounts
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "account_id": "production",
 "config": {
 "name": "Production Environment",
 "description": "Main production Cloudflare account",
 "api_token": "your_production_token_here"
 }
}

# List all accounts for a provider
GET /api/dns/cloudflare/accounts
Authorization: Bearer your_token_here

# Set default account for a provider — there is no dedicated endpoint:
# "set_as_default" travels with the account payload.
PUT /api/dns/cloudflare/accounts/production
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "set_as_default": true
}

# Update account configuration
PUT /api/dns/cloudflare/accounts/staging
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "config": {
 "name": "Staging & Testing",
 "description": "Updated staging environment",
 "api_token": "new_staging_token_here"
 }
}
```

#### Storage Backend Management
```bash
# Get current storage backend information
GET /api/storage/info
Authorization: Bearer your_token_here

# Update storage backend configuration
POST /api/storage/config
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "backend": "azure_keyvault",
 "azure_keyvault": {
 "vault_url": "https://yourvault.vault.azure.net/",
 "tenant_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
 "client_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
 "client_secret": "your_client_secret"
 }
}

# Test storage backend connectivity
POST /api/storage/test
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "backend": "aws_secrets_manager",
 "config": {
 "region": "us-east-1",
 "access_key_id": "AKIAIOSFODNN7EXAMPLE",
 "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
 }
}

# Migrate certificates between storage backends
POST /api/storage/migrate
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "source_backend": "local_filesystem",
 "target_backend": "azure_keyvault",
 "source_config": {
 "cert_dir": "certificates"
 },
 "target_config": {
 "vault_url": "https://yourvault.vault.azure.net/",
 "tenant_id": "...",
 "client_id": "...",
 "client_secret": "..."
 }
}
```

#### Backup Management
```bash
# List all available backups
GET /api/backups
Authorization: Bearer your_token_here

# Create new backup
POST /api/backups/create
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "reason": "manual_backup"
}

# Download specific backup
GET /api/backups/download/unified/{filename}
Authorization: Bearer your_token_here

# Example:
GET /api/backups/download/unified/unified_backup_20241225_120000.zip

# Restore from backup
POST /api/backups/restore/unified
Authorization: Bearer your_token_here
Content-Type: application/json

{
 "filename": "unified_backup_20241225_120000.zip",
 "create_backup_before_restore": true
}
```

### Automation-Friendly Download URL

**Certificate downloads for infrastructure automation:**

```bash
# Download every certificate file as one ZIP
GET /api/certificates/{domain}/download
Authorization: Bearer your_token_here
```

This endpoint returns a ZIP file containing all certificate files:
- `cert.pem` - Server certificate
- `chain.pem` - Intermediate certificate chain
- `fullchain.pem` - Full certificate chain (cert + chain)
- `privkey.pem` - Private key

For a single file, use the path form
`GET /api/certificates/{domain}/download/{cert|chain|fullchain|privkey|combined}`.
`privkey` and `combined` require operator; a viewer may pull the public
material only.

### Integration Examples

#### cURL Download
```bash
curl -H "Authorization: Bearer your_token_here" \
 -o example.com-tls.json \
 https://your-certmate-server.com/api/certificates/example.com/download?format=json
```

#### Python SDK Example
```python
import requests
from pathlib import Path

class CertMateClient:
 def __init__(self, base_url, token):
 self.base_url = base_url.rstrip('/')
 self.headers = {"Authorization": f"Bearer {token}"}
 
 def download_certificate(self, domain):
 """Download certificate bundle as JSON for domain"""
 url = f"{self.base_url}/api/certificates/{domain}/download?format=json"
 
 response = requests.get(url, headers=self.headers)
 response.raise_for_status()
 return response.json()
 
 def list_certificates(self):
 """List all managed certificates"""
 response = requests.get(f"{self.base_url}/api/certificates", 
 headers=self.headers)
 response.raise_for_status()
 return response.json()
 
 def create_certificate(self, domain, dns_provider=None):
 """Create new certificate for domain"""
 data = {"domain": domain}
 if dns_provider:
 data["dns_provider"] = dns_provider
 
 response = requests.post(f"{self.base_url}/api/certificates/create",
 json=data, headers=self.headers)
 response.raise_for_status()
 return response.json()
 
 def renew_certificate(self, domain):
 """Renew existing certificate"""
 response = requests.post(f"{self.base_url}/api/certificates/{domain}/renew",
 headers=self.headers)
 response.raise_for_status()
 return response.json()

# Usage example
client = CertMateClient("https://certmate.company.com", "your_token_here")

# List and download certificates
certs = client.list_certificates()
bundle = client.download_certificate("api.company.com")
Path("/etc/ssl/certs/api").mkdir(parents=True, exist_ok=True)
for name, key in {
    "cert.pem": "cert_pem",
    "chain.pem": "chain_pem",
    "fullchain.pem": "fullchain_pem",
    "privkey.pem": "private_key_pem",
}.items():
    Path("/etc/ssl/certs/api", name).write_text(bundle[key])
```

The same JSON response shape can be consumed directly by Ansible's `uri` module or Salt's HTTP helpers without unpacking an archive.

#### Infrastructure as Code Examples

**Terraform Provider Example:**
```hcl
# Configure the CertMate provider
terraform {
 required_providers {
 certmate = {
 source = "local/certmate"
 version = "~> 1.0"
 }
 }
}

provider "certmate" {
 endpoint = "https://certmate.company.com"
 token = var.certmate_token
}

# Create certificates for multiple domains with different accounts
resource "certmate_certificate" "api" {
 domain = "api.company.com"
 dns_provider = "cloudflare"
 account_id = "production"
}

resource "certmate_certificate" "web" {
 domain = "web.company.com" 
 dns_provider = "route53"
 account_id = "main-aws"
}

resource "certmate_certificate" "staging" {
 domain = "staging.company.com"
 dns_provider = "cloudflare"
 account_id = "staging"
}

# Download certificates to local files
data "certmate_certificate_download" "api" {
 domain = certmate_certificate.api.domain
}

# Use in nginx configuration
resource "kubernetes_secret" "api_tls" {
 metadata {
 name = "api-tls"
 namespace = "default"
 }
 
 type = "kubernetes.io/tls"
 
 data = {
 "tls.crt" = data.certmate_certificate_download.api.fullchain_pem
 "tls.key" = data.certmate_certificate_download.api.private_key_pem
 }
}
```
**Bash Automation Script:**
```bash
#!/bin/bash
set -euo pipefail

# Configuration
CERTMATE_URL="https://certmate.company.com"
API_TOKEN="${CERTMATE_TOKEN}"
DOMAIN="${1:-example.com}"
CERT_DIR="/etc/ssl/certs/${DOMAIN}"
BACKUP_DIR="/backup/certs/${DOMAIN}/$(date +%Y%m%d_%H%M%S)"

# Functions
log() {
 echo "[$(date +'%Y-%m-%d %H:%M:%S')] $*" >&2
}

create_backup() {
 if [[-d "$CERT_DIR" ]]; then
 log "Creating backup of existing certificates"
 mkdir -p "$BACKUP_DIR"
 cp -r "$CERT_DIR"/* "$BACKUP_DIR/" || true
 fi
}

download_certificate() {
 log "Downloading certificate for ${DOMAIN}"
 
 # Download with retry logic
 for i in {1..3}; do
 if curl -f -H "Authorization: Bearer $API_TOKEN" \
 -o "${DOMAIN}-tls.json" \
 "$CERTMATE_URL/api/certificates/$DOMAIN/download?format=json"; then
 log "Certificate downloaded successfully"
 return 0
 else
 log "Download attempt $i failed, retrying..."
 sleep 5
 fi
 done
 
 log "Failed to download certificate after 3 attempts"
 return 1
}

extract_certificate() {
 log "Extracting certificate to ${CERT_DIR}"
 mkdir -p "$CERT_DIR"
 jq -r '.cert_pem' "${DOMAIN}-tls.json" > "$CERT_DIR/cert.pem"
 jq -r '.chain_pem' "${DOMAIN}-tls.json" > "$CERT_DIR/chain.pem"
 jq -r '.fullchain_pem' "${DOMAIN}-tls.json" > "$CERT_DIR/fullchain.pem"
 jq -r '.private_key_pem' "${DOMAIN}-tls.json" > "$CERT_DIR/privkey.pem"
 
 # Set proper permissions
 chmod 600 "$CERT_DIR"/*.pem
 chown root:ssl-cert "$CERT_DIR"/*.pem
}

reload_services() {
 log "Reloading web services"
 systemctl reload nginx || log "Failed to reload nginx"
 systemctl reload apache2 || log "Failed to reload apache2"
 systemctl reload haproxy || log "Failed to reload haproxy"
}

cleanup() {
 rm -f "${DOMAIN}-tls.json"
}

# Main execution
main() {
 log "Starting certificate update for ${DOMAIN}"
 
 create_backup
 download_certificate
 extract_certificate
 reload_services
 cleanup
 
 log "Certificate update completed for ${DOMAIN}"
}

# Trap cleanup on exit
trap cleanup EXIT

# Run main function
main "$@"
```

**Advanced Ansible Playbook:**
```yaml
---
- name: Manage SSL certificates with CertMate multi-account support
 hosts: web_servers
 vars:
 certmate_url: "https://certmate.company.com"
 certmate_token: "{{ vault_certmate_token }}"
 
 tasks:
 - name: Configure Cloudflare accounts
 uri:
 url: "{{ certmate_url }}/api/dns/cloudflare/accounts"
 method: POST
 headers:
 Authorization: "Bearer {{ certmate_token }}"
 Content-Type: "application/json"
 body_format: json
 body:
 account_id: "{{ item.account_id }}"
 config:
 name: "{{ item.name }}"
 description: "{{ item.description }}"
 api_token: "{{ item.api_token }}"
 loop:
 - account_id: "production"
 name: "Production Environment"
 description: "Main production Cloudflare account"
 api_token: "{{ vault_cloudflare_prod_token }}"
 - account_id: "staging"
 name: "Staging Environment"
 description: "Development and testing account"
 api_token: "{{ vault_cloudflare_staging_token }}"
 
 - name: Create certificates with specific accounts
 uri:
 url: "{{ certmate_url }}/api/certificates/create"
 method: POST
 headers:
 Authorization: "Bearer {{ certmate_token }}"
 Content-Type: "application/json"
 body_format: json
 body:
 domain: "{{ item.domain }}"
 dns_provider: "{{ item.provider }}"
 account_id: "{{ item.account_id }}"
 loop:
 - domain: "api.company.com"
 provider: "cloudflare"
 account_id: "production"
 - domain: "staging.company.com"
 provider: "cloudflare"
 account_id: "staging"
 - domain: "test.company.com"
 provider: "route53"
 account_id: "backup-aws"
 
 - name: Download and deploy certificates
 block:
 - name: Download certificate bundle as JSON
 uri:
 url: "{{ certmate_url }}/api/certificates/{{ item }}/download?format=json"
 headers:
 Authorization: "Bearer {{ certmate_token }}"
 return_content: yes
 register: cert_bundle
 
 - name: Write certificate files
 copy:
 dest: "/etc/ssl/certs/{{ item.0.item }}/{{ item.1.name }}"
 content: "{{ item.0.json[item.1.key] }}"
 owner: root
 group: ssl-cert
 mode: "{{ item.1.mode }}"
 loop: "{{ cert_bundle.results | product(cert_files) | list }}"
 vars:
 cert_files:
 - { name: "cert.pem", key: "cert_pem", mode: "0644" }
 - { name: "chain.pem", key: "chain_pem", mode: "0644" }
 - { name: "fullchain.pem", key: "fullchain_pem", mode: "0644" }
 - { name: "privkey.pem", key: "private_key_pem", mode: "0600" }
 loop:
 - "api.company.com"
 - "staging.company.com"
 - "test.company.com"
```

**Production-Ready Ansible Playbook:**
```yaml
---
- name: Enterprise SSL certificate management with CertMate
 hosts: web_servers
 become: yes
 vars:
 certmate_url: "https://certmate.company.com"
 api_token: "{{ vault_certmate_token }}"
 certificate_domains:
 - name: "api.company.com"
 dns_provider: "cloudflare"
 nginx_sites: ["api"]
 services_to_reload: ["nginx"]
 - name: "web.company.com"
 dns_provider: "route53"
 nginx_sites: ["web", "admin"]
 services_to_reload: ["nginx", "haproxy"]
 
 tasks:
 - name: Create certificate directories
 file:
 path: "/etc/ssl/certs/{{ item.name }}"
 state: directory
 owner: root
 group: ssl-cert
 mode: '0750'
 loop: "{{ certificate_domains }}"
 
 - name: Check certificate expiry
 uri:
 url: "{{ certmate_url }}/api/certificates/{{ item.name }}/deployment-status"
 method: GET
 headers:
 Authorization: "Bearer {{ api_token }}"
 register: cert_status
 loop: "{{ certificate_domains }}"
 
 - name: Create new certificates if needed
 uri:
 url: "{{ certmate_url }}/api/certificates/create"
 method: POST
 headers:
 Authorization: "Bearer {{ api_token }}"
 Content-Type: "application/json"
 body_format: json
 body:
 domain: "{{ item.name }}"
 dns_provider: "{{ item.dns_provider }}"
 loop: "{{ certificate_domains }}"
 when: cert_status.results[ansible_loop.index0].json.needs_renewal | default(false)
 
 - name: Download certificates
 uri:
 url: "{{ certmate_url }}/api/certificates/{{ item.name }}/download?format=json"
 method: GET
 headers:
 Authorization: "Bearer {{ api_token }}"
 return_content: yes
 register: cert_bundle
 loop: "{{ certificate_domains }}"
 
 - name: Write certificates
 copy:
 dest: "/etc/ssl/certs/{{ item.0.item.name }}/{{ item.1.name }}"
 content: "{{ item.0.json[item.1.key] }}"
 owner: root
 group: ssl-cert
 mode: "{{ item.1.mode }}"
 loop: "{{ cert_bundle.results | product(cert_files) | list }}"
 vars:
 cert_files:
 - { name: "cert.pem", key: "cert_pem", mode: "0644" }
 - { name: "chain.pem", key: "chain_pem", mode: "0644" }
 - { name: "fullchain.pem", key: "fullchain_pem", mode: "0644" }
 - { name: "privkey.pem", key: "private_key_pem", mode: "0600" }
 notify: 
 - reload nginx
 - reload haproxy
 - restart services
 
 - name: Verify certificate installation
 openssl_certificate:
 path: "/etc/ssl/certs/{{ item.name }}/fullchain.pem"
 provider: assertonly
 has_expired: no
 valid_in: 86400 # Valid for at least 1 day
 loop: "{{ certificate_domains }}"
 
 - name: Update nginx SSL configuration
 template:
 src: "nginx-ssl.conf.j2"
 dest: "/etc/nginx/sites-available/{{ item.1 }}"
 backup: yes
 loop: "{{ certificate_domains | subelements('nginx_sites') }}"
 notify: reload nginx
 
 - name: Cleanup temporary files
 file:
 path: "/tmp/{{ item.name }}-tls.json"
 state: absent
 loop: "{{ certificate_domains }}"
 
 handlers:
 - name: reload nginx
 systemd:
 name: nginx
 state: reloaded
 
 - name: reload haproxy
 systemd:
 name: haproxy
 state: reloaded
 
 - name: restart services
 systemd:
 name: "{{ item }}"
 state: restarted
 loop: "{{ services_to_restart | default([]) }}"
```

## Configuration Guide

### Environment Variables

| Variable           | Required | Default        | Description                         |
| ------------------ | -------- | -------------- | ----------------------------------- |
| `API_BEARER_TOKEN`      |          | auto-generated | Bearer token for API authentication |
| `API_BEARER_TOKEN_FILE` |          | -              | Path to a file containing the API bearer token (takes precedence over `API_BEARER_TOKEN`) |
| `SECRET_KEY`            |          | auto-generated | Flask secret key for sessions       |
| `SECRET_KEY_FILE`       |          | -              | Path to a file containing the Flask secret key (takes precedence over `SECRET_KEY`) |
| `PORT`             |          | `8000`         | Server port (honoured by the container entrypoint) |
| `FLASK_ENV`        |          | `production`   | Flask environment. `production` refuses `--debug`  |
| `CERTMATE_LOG_FILE` |         | -              | Also write logs to this path. Off by default: the container logs to stdout, which is what `docker logs` and log shippers expect. Set it (e.g. `/app/logs/certmate.log`) to keep a file on the mounted volume — it is what the web UI's log stream reads |
| `CERTMATE_LOG_MAX_BYTES` |    | `10485760`     | Rotate the log file at this size (10 MB). File logging is always rotated — there is no way to configure an unbounded one |
| `CERTMATE_LOG_BACKUP_COUNT` | | `5`            | How many rotated files to keep (~60 MB ceiling with the default size) |
| `CERTMATE_AUDIT_LOG_MAX_BYTES` | | `10485760`  | Rotate the human-readable audit log (`logs/audit/certificate_audit.log`) at this size. `0` disables rotation. Does **not** apply to the tamper-evident hash chain in `data/audit/`, which is never rotated |
| `CERTMATE_AUDIT_LOG_BACKUP_COUNT` | | `5`       | How many rotated audit logs to keep. Note the Activity page tails only the active file, so it shows fewer entries immediately after a roll |
| `CERTMATE_LOG_LEVEL` |        | `INFO`         | DEBUG / INFO / WARNING / ERROR |
| `CERTMATE_LOG_JSON` |         | `true`         | JSON log lines (set `false` for human-readable) |

> **Bind address is not an environment variable.** The container always binds
> `0.0.0.0` inside its own network namespace; to expose it only on loopback,
> publish it that way — `-p 127.0.0.1:8000:8000`. Running `app.py` directly
> (development only) takes `--host` / `--port` / `--debug` as CLI flags.
> `HOST` and `FLASK_DEBUG` are read by nothing, and setting them has never
> had any effect (#429).

### DNS Provider Configuration

#### Cloudflare Setup
1. Go to [Cloudflare API Tokens](https://dash.cloudflare.com/profile/api-tokens)
2. Click "Create Token" → "Custom token"
3. Set permissions:
 - **Zone**: `DNS:Edit` + `Zone:Read`
 - **Zone Resources**: Include specific zones or all zones
4. Copy the generated token

```bash
# Environment variable
CLOUDFLARE_TOKEN=your_cloudflare_api_token_here
```

#### AWS Route53 Setup
1. Create IAM user with Route53 permissions
2. Attach policy: `Route53FullAccess` or custom policy:

```json
{
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": ["route53:ListHostedZones",
 "route53:GetChange",
 "route53:ChangeResourceRecordSets"
 ],
 "Resource": "*"
 }
 ]
}
```

```bash
# Environment variables
AWS_ACCESS_KEY_ID=your_access_key_id
AWS_SECRET_ACCESS_KEY=your_secret_access_key
AWS_DEFAULT_REGION=us-east-1
```

#### Azure DNS Setup
1. Create Service Principal:
```bash
az ad sp create-for-rbac --name "CertMate" --role "DNS Zone Contributor" --scopes "/subscriptions/{subscription-id}/resourceGroups/{resource-group}"
```

```bash
# Environment variables
AZURE_SUBSCRIPTION_ID=your_subscription_id
AZURE_RESOURCE_GROUP=your_resource_group_name
AZURE_TENANT_ID=your_tenant_id
AZURE_CLIENT_ID=your_client_id
AZURE_CLIENT_SECRET=your_client_secret
```

#### Google Cloud DNS Setup
1. Create service account with DNS Administrator role
2. Download JSON key file

```bash
# Environment variables
GOOGLE_PROJECT_ID=your_project_id
GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
```

#### PowerDNS Setup
```bash
# Environment variables
POWERDNS_API_URL=https://your-powerdns-server:8081
POWERDNS_API_KEY=your_api_key
```

### Certificate Storage Configuration

CertMate supports multiple storage backends for certificates, providing flexibility for different deployment scenarios and security requirements. By default, certificates are stored locally on the filesystem, but you can configure enterprise-grade storage backends for enhanced security and compliance.

> **Choosing the Right Storage Backend:**
> - **Local Filesystem**: Perfect for development, testing, and small deployments
> - **Azure Key Vault**: Best for Azure-native environments and Microsoft ecosystem integration
> - **AWS Secrets Manager**: Ideal for AWS infrastructure and cross-region deployments
> - **HashiCorp Vault**: Excellent for multi-cloud environments and advanced secret management
> - **Infisical**: Great for teams wanting open-source secret management with collaboration features

#### Local Filesystem (Default)
The default storage backend stores certificates in the local filesystem with secure permissions:

```bash
# Default certificate directory
certificates/
 example.com/
 cert.pem # Server certificate
 chain.pem # Certificate chain
 fullchain.pem # Full chain
 privkey.pem # Private key (600 permissions)
```

**Configuration:**
- **Directory**: `certificates` (configurable)
- **Permissions**: `600` for private keys, `644` for certificates
- **Backup**: Included in automatic backups
- **Use Cases**: Development, testing, single-server deployments

**Benefits:**
- Zero configuration required
- No external dependencies
- Fast access and operations
- Perfect for getting started

#### Azure Key Vault
Store certificates securely in Azure Key Vault for enterprise-grade secret management:

**Required Dependencies:**
```bash
pip install -r requirements-azure-storage.txt
```

**Configuration:**
```json
{
 "certificate_storage": {
 "backend": "azure_keyvault",
 "azure_keyvault": {
 "vault_url": "https://yourvault.vault.azure.net/",
 "tenant_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
 "client_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
 "client_secret": "your_client_secret"
 }
 }
}
```

**Benefits:**
- Azure-native secret management
- Compliance and audit capabilities (SOC 2, ISO 27001, FIPS 140-2)
- Hardware security module (HSM) protection
- Azure RBAC integration and managed identity support
- Automatic backup and disaster recovery

**Use Cases:**
- Azure-based infrastructure
- Enterprise compliance requirements
- Multi-region Azure deployments
- Integration with Azure DevOps and ARM templates

#### AWS Secrets Manager
Integrate with AWS Secrets Manager for scalable secret storage:

**Required Dependencies:**
```bash
pip install -r requirements-aws-storage.txt
```

**Configuration:**
```json
{
 "certificate_storage": {
 "backend": "aws_secrets_manager",
 "aws_secrets_manager": {
 "region": "us-east-1",
 "access_key_id": "AKIAIOSFODNN7EXAMPLE",
 "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
 }
 }
}
```

**Benefits:**
- AWS-native secret management
- Automatic encryption at rest with AWS KMS
- Cross-region replication for high availability
- IAM-based access control and fine-grained permissions
- Integration with AWS CloudTrail for audit logging
- Automatic rotation capabilities

**Use Cases:**
- AWS-based infrastructure
- Multi-region deployments
- Integration with ECS, EKS, Lambda
- Compliance with AWS security best practices

#### HashiCorp Vault
Use industry-standard HashiCorp Vault for advanced secret management:

**Required Dependencies:**
```bash
pip install -r requirements-vault-storage.txt
```

**Configuration:**
```json
{
 "certificate_storage": {
 "backend": "hashicorp_vault",
 "hashicorp_vault": {
 "vault_url": "https://vault.example.com:8200",
 "vault_token": "hvs.xxxxxxxxxxxxxxxxxxxx",
 "mount_point": "secret",
 "engine_version": "v2"
 }
 }
}
```

**Benefits:**
- Industry-standard secret management
- Secret versioning and rollback capabilities
- Comprehensive audit logging and monitoring
- Fine-grained access policies and dynamic secrets
- Multi-cloud and hybrid cloud support
- Advanced authentication methods (LDAP, Kubernetes, AWS IAM)

**Use Cases:**
- Multi-cloud environments
- Complex organizational security requirements
- Dynamic secret generation
- Integration with CI/CD pipelines and Kubernetes

#### Infisical
Modern open-source secret management with team collaboration:

**Required Dependencies:**
```bash
pip install -r requirements-infisical-storage.txt
```

**Configuration:**
```json
{
 "certificate_storage": {
 "backend": "infisical",
 "infisical": {
 "site_url": "https://app.infisical.com",
 "client_id": "your_client_id",
 "client_secret": "your_client_secret",
 "project_id": "your_project_id",
 "environment": "prod"
 }
 }
}
```

**Benefits:**
- Open-source secret management with transparency
- End-to-end encryption for maximum security
- Team collaboration features and role-based access
- Multi-environment support (dev, staging, prod)
- Git-like versioning for secrets
- Self-hostable for complete control

**Use Cases:**
- Team-based development workflows
- Open-source preference
- Self-hosted secret management
- Multi-environment certificate management

#### Quick Installation Guide

**Install All Storage Backends:**
```bash
# Install all storage backends at once
pip install -r requirements-storage-all.txt
```

**Install Individual Storage Backends:**
```bash
# Azure Key Vault only
pip install -r requirements-azure-storage.txt

# AWS Secrets Manager only 
pip install -r requirements-aws-storage.txt

# HashiCorp Vault only
pip install -r requirements-vault-storage.txt

# Infisical only
pip install -r requirements-infisical-storage.txt
```

**Requirements File Overview:**
- `requirements-storage-all.txt` - All storage backends (recommended for production)
- `requirements-azure-storage.txt` - Azure Key Vault dependencies
- `requirements-aws-storage.txt` - AWS Secrets Manager dependencies 
- `requirements-vault-storage.txt` - HashiCorp Vault dependencies
- `requirements-infisical-storage.txt` - Infisical dependencies
- `requirements-minimal.txt` - Base CertMate without storage backends

#### Configuring Storage Backends

**Via Web Interface:**
1. Navigate to Settings → Certificate Storage Backend
2. Select your preferred backend from the dropdown
3. Configure the required credentials and settings
4. Test the connection to verify configuration
5. Save settings and optionally migrate existing certificates

**Via API:**
```bash
# Test storage backend connectivity before switching
curl -X POST "http://localhost:8000/api/storage/test" \
 -H "Authorization: Bearer your_token" \
 -H "Content-Type: application/json" \
 -d '{
 "backend": "azure_keyvault",
 "config": {
 "vault_url": "https://yourvault.vault.azure.net/",
 "tenant_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
 "client_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
 "client_secret": "your_client_secret"
 }
 }'

# Get current storage backend information
curl -X GET "http://localhost:8000/api/storage/info" \
 -H "Authorization: Bearer your_token"

# Update storage backend configuration
curl -X POST "http://localhost:8000/api/storage/config" \
 -H "Authorization: Bearer your_token" \
 -H "Content-Type: application/json" \
 -d '{
 "backend": "hashicorp_vault",
 "config": {
 "vault_url": "https://vault.example.com:8200",
 "vault_token": "hvs.xxxxxxxxxxxxxxxxxxxx",
 "mount_point": "secret",
 "engine_version": "v2"
 }
 }'
```

** Migrating Between Backends:**

*Zero-Downtime Migration Process:*
1. Configure the new storage backend
2. Test connectivity and verify access
3. Use the migration tool in Settings or API
4. Verify all certificates are accessible in new backend
5. Optionally clean up old storage

*Migration via API:*
```bash
# Migrate all certificates from current backend to new backend
curl -X POST "http://localhost:8000/api/storage/migrate" \
 -H "Authorization: Bearer your_token" \
 -H "Content-Type: application/json" \
 -d '{
 "target_backend": "aws_secrets_manager",
 "target_config": {
 "region": "us-east-1",
 "access_key_id": "AKIAIOSFODNN7EXAMPLE", 
 "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
 },
 "verify_migration": true
 }'
```

*Migration Benefits:*
- Zero downtime during migration
- Automatic verification of migrated certificates
- Rollback capability if issues are detected
- Preservation of certificate metadata and permissions

**Backward Compatibility:**
- Existing installations continue working without changes
- New storage backends are opt-in
- Migration is non-destructive (copies certificates)
- Local filesystem remains the default backend

### Directory Structure

```
certmate/
 app.py # Main Flask application
 requirements.txt # Python dependencies
 docker-compose.yml # Docker Compose configuration
 Dockerfile # Container build instructions
 nginx.conf.example # Nginx reverse proxy config template (copy to nginx.conf)
 .env.example # Environment template
 README.md # This documentation
 CONTRIBUTING.md # Contribution guidelines
 docs/ # Comprehensive documentation
 certificates/ # Certificate storage
 {domain}/
 cert.pem # Server certificate
 chain.pem # Certificate chain
 fullchain.pem # Full chain
 privkey.pem # Private key
 data/ # Application data
 settings.json # Persistent settings
 logs/ # Application logs
 letsencrypt/ # Let's Encrypt working directory
 config/ # Certbot configuration
 work/ # Certbot working files
 logs/ # Certbot logs
 modules/ # Modular backend
 core/ # Core modules
 deployer.py # Deploy hooks manager
 notifier.py # Notification channels
 events.py # SSE event bus
 digest.py # Weekly digest
 audit.py # Audit logger
 shell.py # Shell executor
 web/ # Web routes
 api/ # API resources
 templates/ # Web interface templates
 index.html # Main dashboard
 settings.html # Settings page
 activity.html # Activity timeline
 help.html # Help & documentation
 static/js/ # Frontend modules
 dashboard.js # Dashboard logic
 settings.js # Settings components
 cmd-palette.js # Cmd+K palette
 shortcuts.js # Keyboard shortcuts
```

## Security & Best Practices

### Security Considerations

#### Authentication & Authorization
- **Role-Based Access Control**: Assign viewer, operator, or admin roles to each user
- **Scoped API Keys**: Create API keys with specific role permissions and optional expiration
- **Per-Domain Scoping (`allowed_domains`)**: Restrict a scoped API key to a list of domain patterns. Supports exact (`example.com`) and wildcard (`*.example.com`) forms; the wildcard matches subdomains only, not the apex. Out-of-scope requests return `403 DOMAIN_OUT_OF_SCOPE` and are recorded in the audit log. Leave the field empty for unrestricted access (legacy behavior).
- **HMAC-SHA256 Token Hashing**: API tokens are hashed with a server-side HMAC secret, preventing offline brute-force even if the settings file is leaked (backward compatible with pre-2.2.6 SHA-256 hashes)
- **Strong Bearer Tokens**: Use cryptographically secure tokens (32+ characters)
- **Token Rotation**: Regularly rotate API tokens and revoke unused keys
- **Environment Variables**: Never commit tokens to version control
- **HTTPS Only**: Always use HTTPS in production environments
- **IP Restrictions**: Implement firewall rules to restrict access

#### Settings API Hardening
- **Strict Field Whitelist on `POST /api/settings`**: only documented configuration keys are accepted. Sensitive fields (`api_bearer_token`, `deploy_hooks`, `users`, `api_keys`, `local_auth_enabled`) **cannot** be written through the generic settings endpoint — each has its own dedicated endpoint with its own audit:
  - Users → `POST /api/users`
  - API keys → `POST /api/keys`
  - Deploy hooks → `POST /api/deploy/config`
  - Local auth toggle → `POST /api/auth/config` (admin-only)
  Unknown or rejected keys are returned in a `400` response with a `hint` field pointing at the correct endpoint.
- **Audit Trail for Configuration Changes**: every mutation to settings, the auth-config toggle, users, scoped API keys, and deploy hooks is recorded with operator identity and source IP. Authorization denials (out-of-scope domain access, blocked field writes) are recorded too. Logs are written to `data/audit/certificate_audit.log` as one JSON object per line — pipeable into your SIEM of choice.

#### Certificate Security
- **File Permissions**: Private keys stored with `600` permissions
- **Directory Permissions**: Certificate directories with `700` permissions
- **Backup Encryption**: Encrypt certificate backups
- **Access Logging**: Monitor certificate access patterns

#### Secret Storage Hardening
CertMate stores DNS provider credentials, ACME account keys, and (legacy) bearer tokens inside `data/settings.json`. The bearer token itself is migrated to an HMAC-SHA256 hash on first save, but **DNS provider credentials remain in the file in their original form** so they can be passed to certbot plugins. The file is created with `0600` permissions, which is the first line of defense. Recommended hardening, in order of effort:

1. **Use an external secret backend** (already supported): point `certificate_storage.backend` at HashiCorp Vault, Infisical, AWS Secrets Manager, or Azure Key Vault. Issued certificates are stored there transparently, keeping the bearer token + DNS credentials in `settings.json` as the only on-disk secret surface.
2. **Encrypt the underlying volume**: run CertMate's `data/` directory on a LUKS-encrypted partition (Linux), an encrypted APFS volume (macOS), or a Kubernetes `Secret` mounted in a `tmpfs`. This protects credentials at rest even if the host disk is removed or imaged.
3. **Run as a dedicated non-root user** with `data/` owned by that user and `0700` mode. Verify with `ls -la data/` — only the CertMate process user should be able to read the file.
4. **Avoid bind-mounting `data/` from untrusted sources** in Docker. Use a named volume managed by Docker, or a CSI-provisioned volume in Kubernetes, rather than a bind from a multi-tenant host.
5. **Rotate credentials regularly**: DNS provider credentials can be rotated independently of CertMate — issue a new token in the provider console, update Settings, then revoke the old token. CertMate will pick up the new credentials on the next renewal.

> CertMate does **not** currently encrypt secrets at the application layer. If your threat model requires that the same operator who can read `settings.json` should still not be able to read DNS credentials, use option (1) above and configure the per-provider credentials in the external secret backend rather than in CertMate's settings.

#### Network Security
```bash
# Example firewall rules (iptables)
# Allow only specific IPs to access CertMate
iptables -A INPUT -p tcp --dport 8000 -s 10.0.0.0/8 -j ACCEPT
iptables -A INPUT -p tcp --dport 8000 -s 192.168.0.0/16 -j ACCEPT
iptables -A INPUT -p tcp --dport 8000 -j DROP
```

The rules above confine **inbound** access. You can also confine **outbound**
traffic: route CertMate's HTTP(S) egress through a forward proxy and deny it any
other route to the internet, so it can reach only your CA, DNS provider, object
storage, and notification endpoints. With [Secure Proxy Manager](https://github.com/fabriziosalmi/secure-proxy-manager)
v3.9.0+ this is a built-in default-deny egress allowlist. See
[Confining outbound traffic](docs/installation.md#confining-outbound-traffic-egress-hardening)
for a worked example.

### Performance Optimization

#### Production Deployment
```yaml
# docker-compose.prod.yml
version: '3.8'
services:
  certmate:
    image: certmate:latest
    deploy:
      # CertMate runs an in-process renewal scheduler and per-domain locks, so
      # it must run as a SINGLE writer. Multiple replicas (or >1 gunicorn
      # worker) each fire the renewal check and would issue duplicate ACME
      # orders and hit the CA's duplicate-certificate rate limit. A host-local
      # flock guards multiple workers/containers on a *shared* data volume, but
      # the safe default is one writer. For HA, front a single active instance
      # rather than scaling this service.
      replicas: 1
      resources:
        limits:
          cpus: '1.0'
          memory: 512M
        reservations:
          cpus: '0.5'
          memory: 256M
    environment:
      - FLASK_ENV=production
      # The worker and thread counts are fixed in the image
      # (`--workers 1 --threads 8`): one worker because the scheduler runs
      # in-process and a second would duplicate every renewal. GUNICORN_TIMEOUT
      # is the one that is read.
      - GUNICORN_TIMEOUT=300   # the image default; raise it for slow DNS providers
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
```

#### Load Balancing with Nginx
```nginx
upstream certmate_backend {
 server certmate1:8000;
 server certmate2:8000;
 server certmate3:8000;
}

server {
 listen 443 ssl http2;
 server_name certmate.company.com;
 
 ssl_certificate /etc/ssl/certs/certmate.company.com/fullchain.pem;
 ssl_certificate_key /etc/ssl/certs/certmate.company.com/privkey.pem;
 
 location / {
 proxy_pass http://certmate_backend;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 
 # API rate limiting
 limit_req zone=api burst=10 nodelay;
 }
}
```

### Backup and Recovery

CertMate provides comprehensive backup and recovery capabilities built directly into the application, ensuring your certificates and configuration data are always protected.

#### Unified Backup System

**What is Unified Backup?**
- **Atomic Operation**: Creates a single ZIP file containing both settings and certificates
- **Data Consistency**: Ensures settings and certificates are always in sync
- **Prevents Corruption**: Eliminates configuration/certificate mismatches
- **Simplified Management**: One file per snapshot. A disaster-recovery archive (`include_secrets=true`) contains everything needed for complete restoration; the default share-safe archive contains certificates, chains, metadata, the audit chain and the inventory — no credentials and no private keys

**Two kinds of archive:**
- **Share-safe** (the default, and what every automatic backup is): settings with every credential masked, and **no private keys** — no ACME `privkey.pem`, no ACME account key, no private-CA key, no `.pfx`. Certificates, chains, metadata, the audit chain and the inventory are all there. The manifest says so (`secrets_masked`, `key_material_excluded`). Such an archive cannot restore an instance on its own, and the restore path refuses to pretend otherwise.
- **Disaster recovery**: `POST /api/backups/create` with `{"include_secrets": true}` (admin, audited) — plaintext settings and every key. Set `CERTMATE_BACKUP_PASSPHRASE` so it is encrypted at rest; that is the archive to keep off-site.

**Automatic Backups:**
- **Unified Snapshots** - Automatically created when DNS providers, domains, certificates, or application settings are modified (share-safe, see above)
- **Retention** - Two rules, whichever hits first: the 50 most recent archives per type are kept (`MAX_BACKUPS_PER_TYPE`), **and any archive older than 30 days is deleted regardless of how few there are** (`BACKUP_RETENTION_DAYS`). Both are constants, not settings. This is why a disaster-recovery archive must be kept off the host: one left in `backups/unified/` is deleted after 30 days like any other. Logging in no longer writes a backup, so routine sign-ins do not consume restore points
- **Automatic Cleanup** - Pruning runs after every backup, applying both rules above

**Manual Backups:**
- **On-Demand Creation** - Create backups anytime via the web interface or API
- **Download Support** - Export backups for external storage and disaster recovery
- **Comprehensive Coverage** - Includes all DNS configurations, certificates, and application settings

#### Web Interface Backup Management

Access backup features from the Settings page:

```html

<button onclick="createBackup('unified', this)">Create Backup</button>

- Download backups for external storage
- Restore from any backup point with atomic consistency
- View backup contents and metadata
- Delete specific backups manually
```

#### API Backup Operations

**Create Backup:**
```bash
# Create backup (settings + certificates)
curl -X POST "http://localhost:8000/api/backups/create" \
 -H "Authorization: Bearer your_token" \
 -H "Content-Type: application/json" \
 -d '{"reason": "manual_backup"}'

# Response includes backup file information
{
 "success": true,
 "backup_file": "unified_backup_20241225_120000.zip",
 "size": "2.5MB",
 "contents": {
 "settings": true,
 "certificates": 15
 }
}
```

**List and Download Backups:**
```bash
# List all backups
curl -H "Authorization: Bearer your_token" \
 "http://localhost:8000/api/backups"

# Download backup
curl -H "Authorization: Bearer your_token" \
 "http://localhost:8000/api/backups/download/unified/unified_backup_20241225_120000.zip" \
 -o backup.zip
```

#### Backup File Structure

**Backup File (ZIP):**
```
backup_20260821_120000_000000_manual.zip
 backup_metadata.json  # manifest: secrets_masked, key_material_excluded, key_files_excluded, encrypted
 settings.json         # application settings — credentials masked in a share-safe archive, plaintext in a DR one
 certificates/
   domain1.com/
     cert.pem
     chain.pem
     fullchain.pem
     privkey.pem       # DR archive only (include_secrets=true); absent from a share-safe archive
     metadata.json
     live/ archive/ renewal/ accounts/   # certbot lineage (keys under accounts/ and live/ DR only)
 data/
   certs/ca/ca.crt      # ca.key: DR only
   certs/client/...     # client certificates (.key/.pfx: DR only)
   audit/ inventory/
```

#### Recovery Procedures

**Backup Restoration:**

*Web Interface:*
1. Navigate to Settings → Backup Management
2. Select the backup to restore from
3. Confirm restoration (restores both settings and certificates atomically)
4. Application will restart to apply new settings
5. Verify all certificates and configurations are working

*API Restoration:*
```bash
# Restore from backup
curl -X POST "http://localhost:8000/api/backups/restore/unified" \
 -H "Authorization: Bearer your_token" \
 -H "Content-Type: application/json" \
 -d '{"filename": "unified_backup_20241225_120000.zip", "create_backup_before_restore": true}'
```

#### External Backup Integration

For additional protection, integrate with external backup systems:

**Automated External Backup Script:**
```bash
#!/bin/bash
# /opt/scripts/backup-certmate-external.sh

BACKUP_DIR="/backup/certmate/$(date +%Y%m%d_%H%M%S)"
CERT_DIR="/opt/certmate/certificates"
DATA_DIR="/opt/certmate/data"
RETENTION_DAYS=30

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Download latest backup via API
curl -H "Authorization: Bearer $API_TOKEN" \
 "http://localhost:8000/api/backups/download/unified/latest" \
 -o "$BACKUP_DIR/certmate_backup.zip"

# Backup certificates directory
tar -czf "$BACKUP_DIR/certificates.tar.gz" "$CERT_DIR"

# Backup application data
tar -czf "$BACKUP_DIR/data.tar.gz" "$DATA_DIR"

# Encrypt backups (optional)
gpg --cipher-algo AES256 --compress-algo 1 --symmetric \
 --output "$BACKUP_DIR/certmate_backup.zip.gpg" \
 "$BACKUP_DIR/certmate_backup.zip"

# Cleanup old backups
find /backup/certmate -type d -mtime +$RETENTION_DAYS -exec rm -rf {} \;

echo "External backup completed: $BACKUP_DIR"
```

#### Recovery Procedure
```bash
#!/bin/bash
# Recovery from backup

BACKUP_DATE="20241225_120000"
BACKUP_DIR="/backup/certmate/$BACKUP_DATE"

# Stop services
docker-compose down

# Restore certificates
tar -xzf "$BACKUP_DIR/certificates.tar.gz" -C /opt/certmate/

# Restore data
tar -xzf "$BACKUP_DIR/data.tar.gz" -C /opt/certmate/

# Set permissions
chown -R 1000:1000 /opt/certmate/certificates
chmod -R 700 /opt/certmate/certificates

# Start services
docker-compose up -d

echo "Recovery completed from backup: $BACKUP_DATE"
```

## Monitoring & Observability

### Health Monitoring

#### Built-in Health Checks
```bash
# Basic health check
curl -f http://localhost:8000/health

# Detailed health with auth
curl -H "Authorization: Bearer your_token" \
 http://localhost:8000/api/certificates
```

#### Prometheus Metrics Integration
```python
# Add to app.py for Prometheus monitoring
from prometheus_client import Counter, Histogram, generate_latest

# Metrics
certificate_requests = Counter('certmate_certificate_requests_total', 
 'Total certificate requests', ['domain', 'status'])
certificate_expiry = Histogram('certmate_certificate_expiry_days',
 'Days until certificate expiry', ['domain'])

@app.route('/metrics')
def metrics():
 return generate_latest()
```

A ready-to-import **Grafana dashboard**, **Prometheus alert rules**, and an
authenticated **scrape config** ship in [`monitoring/`](monitoring/) — see
[monitoring/README.md](monitoring/README.md). The `/metrics` endpoint requires
the admin role, so scrape it with an admin-scoped API token (Bearer).

#### Log Aggregation
```yaml
# docker-compose.logging.yml
version: '3.8'
services:
 certmate:
 logging:
 driver: "fluentd"
 options:
 fluentd-address: localhost:24224
 tag: certmate
 fluentd-async-connect: "true"
 
 fluentd:
 image: fluent/fluentd:v1.14
 volumes:
 - ./fluentd/conf:/fluentd/etc
 - ./logs:/var/log/fluentd
 ports:
 - "24224:24224"
 - "24224:24224/udp"
```

### Grafana Dashboard Example
```json
{
 "dashboard": {
 "title": "CertMate SSL Certificate Monitoring",
 "panels": [{
 "title": "Certificate Expiry Status",
 "targets": [{
 "expr": "certmate_certificate_expiry_days < 30",
 "legendFormat": "Expiring Soon ({{domain}})"
 }
 ],
 "alert": {
 "conditions": [{
 "query": {"queryType": "", "refId": "A"},
 "reducer": {"type": "last", "params": []},
 "evaluator": {"params": [30], "type": "lt"}
 }
 ],
 "executionErrorState": "alerting",
 "frequency": "1h",
 "handler": 1,
 "name": "Certificate Expiring",
 "noDataState": "no_data",
 "notifications": []
 }
 }
 ]
 }
}
```

### Built-in Notifications

CertMate includes a built-in notification system configurable from Settings > Notifications:

- **Email (SMTP)** - Certificate expiry warnings and renewal confirmations
- **Slack** - Incoming webhook integration for team channels
- **Discord** - Webhook notifications for Discord servers
- **Telegram** - Bot API messages (bot token + chat ID)
- **ntfy** - Push to an [ntfy](https://ntfy.sh) topic (self-hostable); optional access token, per-message priority
- **Gotify** - Push to a self-hosted [Gotify](https://gotify.net) server (server URL + app token, numeric priority)
- **Generic Webhooks** - HTTP POST/PUT/PATCH with HMAC-SHA256 signed payloads, bearer/basic/header authentication, and a JSON **payload template** with `{{placeholders}}` so the body fits whatever receives it — see [docs/webhooks.md](docs/webhooks.md)
- **Weekly Digest** - Scheduled summary of certificate status and upcoming renewals

All notification channels support per-event filtering (created, renewed, expiring, failed) and can be tested from the settings UI.

> **Microsoft Teams:** no dedicated adapter is needed. Use a Teams channel's
> built-in email address (channel → ... → Get email address) as a recipient on
> the Email (SMTP) channel — Teams posts the message into the channel for you.

### Downgrades & Recovery

Downgrading to a version older than the one that wrote `settings.json` is not supported and may result in a broken configuration or loss of accounts. If you see a `DOWNGRADE DETECTED` message in the logs after rolling back, **restore the latest unified backup before using the UI**:

```bash
# List available backups inside the container
docker exec certmate ls -lt /app/backups/unified/

# Restore the most recent one
curl -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"filename": "backup_YYYYMMDD_HHMMSS.zip", "create_backup_before_restore": true}' \
  http://localhost:8000/api/backups/restore/unified
```

If you have lost the admin password and cannot log in, use the emergency reset script (requires container shell access):

```bash
docker exec -it certmate python scripts/reset_admin_password.py
```

## Troubleshooting Guide

### Common Issues & Solutions

#### Certificate Creation Failures

**Issue**: `DNS validation failed`
```bash
# Check DNS propagation
dig TXT _acme-challenge.example.com @8.8.8.8

# Verify DNS provider credentials
curl -H "Authorization: Bearer cf_token" \
 "https://api.cloudflare.com/client/v4/user/tokens/verify"
```

**Issue**: `Rate limit exceeded`
```bash
# Let's Encrypt rate limits:
# - 50 certificates per registered domain per week
# - 5 duplicate certificates per week
# - 300 new orders per account per 3 hours

# Check rate limit status
curl "https://crt.sh/?q=example.com&output=json" | jq length
```

**Issue**: `Permission denied accessing certificate files`
```bash
# Fix file permissions
sudo chown -R certmate:certmate /opt/certmate/certificates
sudo chmod -R 700 /opt/certmate/certificates
sudo chmod 600 /opt/certmate/certificates/*/privkey.pem
```

#### API Authentication Issues

**Issue**: `401 Unauthorized`
```bash
# Verify token format
curl -H "Authorization: Bearer your_token_here" \
 http://localhost:8000/api/certificates

# Check token in settings
docker exec certmate cat /app/data/settings.json | jq .api_bearer_token
```

**Issue**: `Token not found`
```bash
# Reset API token
docker exec -it certmate python -c "
import json
with open('/app/data/settings.json', 'r+') as f:
 data = json.load(f)
 data['api_bearer_token'] = 'new_secure_token_here'
 f.seek(0)
 json.dump(data, f, indent=2)
 f.truncate()
"
```

#### Docker & Container Issues

**Issue**: `Container won't start`
```bash
# Check logs
docker-compose logs certmate

# Verify environment variables
docker-compose config

# Check port conflicts
netstat -tulpn | grep :8000
```

**Issue**: `Volume mount issues`
```bash
# Fix volume permissions
sudo chown -R 1000:1000 ./certificates ./data ./logs

# Check volume mounts
docker inspect certmate | jq '.[0].Mounts'
```

#### Deployment Status Shows "Backend: Unreachable"

**Q: All my certificates show "Backend: Unreachable" even though they are issued, served, and downloadable. Is something broken?**

No. The deployment-status badge is an optional health indicator — it does not affect issuance, renewal, or download. CertMate's own process opens a TLS connection to `<domain>:<port>` and compares the served certificate's fingerprint against the stored one. The badge only answers whether CertMate itself can reach the service and see the expected certificate.

- **Deployed** — handshake succeeded and the fingerprint matches.
- **Wrong Cert** — handshake succeeded but a different certificate is served.
- **Unreachable** — CertMate could not open a TLS connection to the domain at all.

**Configuring the probe per certificate**

By default the probe connects on port 443 with a direct TLS handshake (HTTPS). You can change the port and protocol per certificate via the API or the dashboard:

| Protocol | Port (default) | Use case |
|---|---|---|
| `https-tls` | 443 | Standard HTTPS |
| `tls` | 465 | SMTPS, IMAPS, or any direct TLS service |
| `smtp-starttls` | 587 | SMTP with STARTTLS (port 25 also works) |

From the **dashboard**, open a certificate's detail panel and click **Configure Probe**, then enter the port and protocol.

From the **API**:

```bash
# Set probe to SMTP STARTTLS on port 587
curl -X PATCH https://certmate.example.com/api/certificates/example.com \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"deployment_port": 587, "deployment_protocol": "smtp-starttls"}'

# Reset to defaults (HTTPS on port 443)
curl -X PATCH https://certmate.example.com/api/certificates/example.com \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"deployment_port": null, "deployment_protocol": "https-tls"}'
```

The probe protocol defaults to the global timeout (3s) and can be tuned via the `CERTMATE_TLS_PROBE_TIMEOUT_SECONDS` environment variable.

**Global timeout tuning:**

```bash
# Accepts 1–30 seconds; default is 3
CERTMATE_TLS_PROBE_TIMEOUT_SECONDS=10
```

**"Unreachable" for every certificate** is common in Kubernetes/ingress setups where the CertMate pod cannot dial the public/ingress IP directly (split-horizon DNS, an egress `NetworkPolicy`, or TLS terminated by an ingress controller / load balancer). If the target is merely slow, raise the probe budget or configure the port/protocol to match your topology.

See [docs/kubernetes.md](docs/kubernetes.md#deployment-status-badge-shows-backend-unreachable) for the Kubernetes-specific note. (Reference: [#263](https://github.com/fabriziosalmi/certmate/issues/263).)

#### DNS Provider Specific Issues

**Cloudflare**:
```bash
# Verify API token permissions
curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
 -H "Authorization: Bearer your_token_here"

# Check zone access
curl -X GET "https://api.cloudflare.com/client/v4/zones" \
 -H "Authorization: Bearer your_token_here"
```

**AWS Route53**:
```bash
# Test AWS credentials
aws sts get-caller-identity

# Check Route53 permissions
aws route53 list-hosted-zones
```

**Azure DNS**:
```bash
# Verify service principal
az login --service-principal \
 -u $AZURE_CLIENT_ID \
 -p $AZURE_CLIENT_SECRET \
 --tenant $AZURE_TENANT_ID

# Check DNS zone access
az network dns zone list
```

### Debug Mode

Enable debug logging for troubleshooting:

```bash
# Development server: debug and log level are CLI flags, not env vars
python app.py --debug --log-level DEBUG

# FLASK_ENV=production makes --debug refuse to start, by design
FLASK_ENV=development

# Or in Docker Compose
docker-compose -f docker-compose.yml -f docker-compose.debug.yml up
```

### What's New in v2.0.0

CertMate 2.0 is a major release that adds enterprise-grade access control, a notification system, post-issuance automation, and a modernized UI.

**Highlights:**
- **Role-Based Access Control** - Three-tier RBAC (viewer / operator / admin) with per-user roles
- **Scoped API Key Management** - Create, list, revoke API keys with role scope and optional expiry
- **Notification System** - Email (SMTP), Slack, Discord, and webhook channels with HMAC signatures
- **Deploy Hooks** - Post-issuance shell commands with environment variables, dry-run testing, and execution history
- **Weekly Digest** - Scheduled email summary of certificate health and upcoming renewals
- **Setup Wizard** - Guided first-run flow for DNS, CA, and authentication configuration
- **Command Palette** - Cmd+K / Ctrl+K quick search and navigation
- **Keyboard Shortcuts** - Power-user shortcuts (`?` help, `/` search, `g+h` home, `g+s` settings, etc.)
- **Activity Timeline** - Chronological event log for all certificate and system operations
- **SSE Real-Time Events** - Live push notifications on the dashboard
- **Dark Mode Toggle** - System-aware theme switching
- **Mobile Bottom Tab Bar** - Responsive navigation on small screens
- **HTTP-01 Challenge Support** - Alternative to DNS-01 for simple setups

### Reporting Bugs

CertMate ships an in-app bug reporter that produces an actionable issue with one click. When an action fails and you're signed in as admin, the error toast surfaces a **Report this issue** button. Clicking it:

1. Calls `GET /api/diagnostics/snapshot` (admin-only) to collect sanitised operational state — CertMate version, Python version, OS, scheduler status, certificate count, DNS provider name, CA name, challenge type, storage backend, disk free, and the last 5 audit-log entries with all identifiers (resource_id, user, IP address, details, error) stripped.
2. Merges the snapshot with browser-side context (user agent, current page, viewport, the specific error envelope `endpoint` / `status` / `code` / `message` / `hint`).
3. Formats the result as Markdown and copies it to your clipboard.
4. Opens `github.com/fabriziosalmi/certmate/issues/new?template=bug_report.md` with the title pre-filled as `[Bug] <status> <code> on <method> <endpoint>`.

Paste the clipboard contents into the issue body, edit if you want, submit. The flow is fully manual — nothing leaves the install without your explicit click, and you can read everything before submitting (it's right there in the textarea). If your browser blocks the clipboard write or pop-ups, the reporter falls back to a modal with the markdown in an editable textarea and a clickable GitHub link.

### Support Checklist (manual reporting)

If the in-app reporter is unavailable (you're not signed in as admin, or you hit the bug before reaching the UI), please provide:

- [ ] CertMate version/commit hash
- [ ] DNS provider being used
- [ ] Error messages from logs
- [ ] Steps to reproduce the issue
- [ ] Environment details (Docker, Python version, OS)

```bash
# Collect system information manually (equivalent of the in-app snapshot)
echo "=== CertMate Debug Info ==="
echo "Version: $(docker exec certmate python -c 'import modules; print(modules.__version__)')"
echo "Python: $(docker exec certmate python --version)"
echo "OS: $(docker exec certmate cat /etc/os-release | head -2)"
echo "Certbot: $(docker exec certmate certbot --version)"
echo "DNS Plugins: $(docker exec certmate pip list | grep certbot-dns)"
# DO NOT paste settings.json verbatim into a public issue — it contains
# credentials. The /api/diagnostics/snapshot endpoint returns a redacted
# subset; use it instead:
curl -sS -H "Authorization: Bearer $TOKEN" http://localhost:8000/api/diagnostics/snapshot | jq .
```

## Documentation

### Complete Documentation Set

| Document                                           | Description                         | Target Audience       |
| -------------------------------------------------- | ----------------------------------- | --------------------- |
| **[README.md](README.md)**                         | Main documentation and quick start  | All users             |
| **[docs/installation.md](docs/installation.md)**   | Installation and deployment         | System administrators |
| **[docs/dns-providers.md](docs/dns-providers.md)** | DNS provider setup                  | DevOps engineers      |
| **[docs/ca-providers.md](docs/ca-providers.md)**   | Certificate Authority configuration | Enterprise users      |
| **[docs/docker.md](docs/docker.md)**               | Docker and multi-platform builds    | DevOps engineers      |
| **[docs/testing.md](docs/testing.md)**             | Testing framework and CI/CD         | Developers            |
| **[docs/architecture.md](docs/architecture.md)**   | System architecture                 | Developers            |
| **[docs/api.md](docs/api.md)**                     | Client certificates API reference   | Developers            |
| **[docs/mcp.md](docs/mcp.md)**                     | MCP server for AI agents: tools, auth, audit attribution | Developers, SRE |
| **[docs/guide.md](docs/guide.md)**                 | Step-by-step guide for common tasks | All users             |
| **[docs/discovery-inventory.md](docs/discovery-inventory.md)** | Discovery, inventory, adopt, crypto readiness | SRE, security |
| **[docs/deploy-hooks.md](docs/deploy-hooks.md)**   | Post-issuance deploy hooks          | DevOps engineers      |
| **[docs/webhooks.md](docs/webhooks.md)**           | Generic webhooks: payload templates, auth, signature verification | Integrators |
| **[docs/compliance.md](docs/compliance.md)**       | Audit chain, attribution, NIS2/eIDAS posture | Compliance, security |
| **[docs/kubernetes.md](docs/kubernetes.md)**       | Pod sizing, OOM troubleshooting, Helm chart | SRE              |
| **[docs/probes.en.md](docs/probes.en.md)**         | Deployment probe configuration      | DevOps engineers      |
| **[CONTRIBUTING.md](CONTRIBUTING.md)**             | Development and contribution guide  | Developers            |
| **[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)**       | Community guidelines                | Contributors          |

### Online Resources

- **API Documentation**: http://your-server:8000/docs/ (Swagger UI)
- **Alternative API Docs**: http://your-server:8000/redoc/ (ReDoc)
- **GitHub Repository**: https://github.com/fabriziosalmi/certmate
- **Docker Hub**: https://hub.docker.com/r/fabriziosalmi/certmate
- **Issue Tracker**: https://github.com/fabriziosalmi/certmate/issues

### Examples Repository

Check out our examples repository for:
- Production deployment configurations
- Integration scripts for popular tools
- Terraform modules
- Kubernetes manifests
- CI/CD pipeline examples

### Community Contributions

We welcome contributions! Areas where we need help:
- **Documentation** - Tutorials, use cases, translations
- **Testing** - DNS provider testing, edge cases
- **Integrations** - New DNS providers, monitoring tools
- **Features** - UI improvements, API enhancements

## Commercial Support & Consulting

Running CertMate in production? I offer paid support, custom development, and
security consulting — certificate/TLS automation, hardening, WAF, and cloud
detection & alerting. Reach out: **[fabrizio.salmi@gmail.com](mailto:fabrizio.salmi@gmail.com)**.

## Contributing

We love contributions! CertMate is an open-source project and we welcome:

### Types of Contributions
- **Bug Reports** - Help us identify and fix issues
- **Feature Requests** - Suggest new functionality
- **Documentation** - Improve guides and examples
- **Testing** - Test new features and edge cases
- **Code** - Submit pull requests with improvements

### Quick Start for Contributors

```bash
# Fork and clone the repository
git clone https://github.com/fabriziosalmi/certmate.git
cd certmate

# Create development environment
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-test.txt

# Lint, format, and security-scan (the same tools CI runs)
make lint
make format
make security

# Run tests
pytest

# Start development server
python app.py
```

### Contribution Guidelines

1. **Fork** the repository
2. **Create** a feature branch (`git checkout -b feature/amazing-feature`)
3. **Commit** your changes (`git commit -m 'Add amazing feature'`)
4. **Push** to the branch (`git push origin feature/amazing-feature`)
5. **Open** a Pull Request

## License

This project is licensed under the **MIT License** - see the [LICENSE](LICENSE) file for details.

---

**Made with ❤️ by Fabrizio Salmi**

[Star us on GitHub](https://github.com/fabriziosalmi/certmate) • [Report Bug](https://github.com/fabriziosalmi/certmate/issues) • [Request Feature](https://github.com/fabriziosalmi/certmate/issues/new?template=feature_request.md)

<br>

<a href="https://europeanopensource.eu/">
  <img src="https://raw.githubusercontent.com/European-OpenSource/media-kit/main/assets/logo/light/Logo%20-%20standard.svg" alt="European Open Source" height="28">
</a>
<br>
<sub>Listed in the European Open Source Catalogue</sub>

</div>

certmate

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

certmate

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README