zync

KO | EN

gitlite — search

#python #ai #golang #go #android #machine-learning #rust #docker #macos #windows #awesome-list #pytorch #openai #mcp

(★ 21)

Zync takes your 3scale data and pushes it somewhere else, reliably.

zync Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![CircleCI](https://circleci.com/gh/3scale/zync.svg?style=svg)](https://circleci.com/gh/3scale/zync)
[![Maintainability](https://qlty.sh/gh/3scale/projects/zync/maintainability.svg)](https://qlty.sh/gh/3scale/projects/zync)
[![Code Coverage](https://qlty.sh/gh/3scale/projects/zync/test_coverage.svg)](https://qlty.sh/gh/3scale/projects/zync)

# Zync - the sync tool

Zync is going to take your 3scale data and pushes it somewhere else, reliably. Offers only one directional sync (from 3scale to other systems).

## Building from Source

This component is deployed as part of the overall 3scale API Management solution.

Please see [these instructions](INSTALL.md) and [Quickstart guide](doc/Quickstart.md) on how to set up your development environment locally and [these instructions](INTEGRATE.md) to integrate it with [Porta](https://github.com/3scale/porta) and Keycloak.

## Terminology

Before describing the design it is important to define the terms.

* **Zync** - deployment of this project, Web API.

* **3scale** - 3scale API Manager (Porta).
* **Tenant** - mapping of Provider id registered in 3scale API Manager to the domain and access token.
* **Model** - relevant object in **3scale** like Application, Limit, Metric, Service.
* **Notification** - Message sent to **Zync** describing the **model** that changed and having all required properties to fetch it from the API later.
  * Example: Application 3456, tenant_id: 26
  * Example: Limit 4566, metric_id: 36, application_id: 46, tenant_id: 16
* **Update** - **Zync** fetches updated **Model** from the **Notifier** via the API using the information provided by the **Notification**.
* **Lock** - the mechanism that prevents concurrent data access to the same scope.
  * Example: **Tenant Lock** would mean only one can be running for one **Tenant**.
  * Example: **Model** **Lock** - only one per uniquely identified **Model**.
* **Entry** - The information from the API provided by the **Update**.
* **Log** - ordered list of **Entries** as they were received.
* **Push** - a call from **Zync** to external service updating **model** data
* **Integration** - code that **pushes** **Log** **entries** one-by-one for the same **Model** to some external service.
  * Integration can access the **entries** **log** to fetch more data and for example handle **model** dependencies by accessing all dependent **models** and delete them before deleting the parent.
  * Integration keeps **Status** of each **model** synchronization state. If a **push** fails **status** should be updated to reflect that and keep trying.
* **Integration Configuration** - configuration for each instance of **Integration** for each **Tenant**. It is populated the same way as **Model** - by **Notification** and **Fetch**.

## Design

**Zync** is meant to synchronize data from **3scale** to external systems (like IDPs). Some people use Web-hooks for this but without further logic, they can be unreliable and arrive out of order. This tool is meant to synchronize the final state to a different system.

The flow is defined as **3scale** -> **Zync** ( <- **3scale**) -> **Integration**. So **3scale** notifies **Zync** there was a change to a **model** but does not say more than the primary key and information required to fetch it from the **3scale** API. In some cases **model** needs just its primary key (**id**) and in some, it needs other metadata (usually primary keys of its parents) to compose the API call (service_id, metric_id, …).

**Zync** upon receiving the notification will acquire an **update model lock** and try to perform an **update**. Any information received this way is added as an **entry** to the **log** and the **model lock** is released. That **entry** can be either new data or information that the record is no longer there (404 from the API). If new **notification** came when the **model lock** was acquired, it is going to be processed after the lock is released.

After adding **entry** to the **log** an **integration** is triggered and acquires an **integration model lock** so it will process only one (latest) **entry** for a **model** at a time. After the **integration** finishes (with both failure or success) it will release the lock and trigger another run if failed.

## Properties

Given the locking on the **model** there will be some parallelization, but also updates to one object will be serialized. This needs to be done to limit the network issues and ensure the request was delivered before issuing a new one.

Because **Zync** will keep a **log** of **events** it will be able to replay changes and recover the last state just taking last revisions of each **model** and even remove the ones that have been created before but have been deleted.

## Data Model

**Record** types are for enforcing correctness of data on the database level and referential integrity. There is one relationship (Model -> Record) that can't have foreign constraints but can be recreated from other data.

### Tenant

| id (pk) | domain | access_token |
| ------- | ------ | ------------ |
| bigint  | string | string       |

### Notification

| id (pk) | model_class | model_id (fk) | data | tenant_id (fk) |
| ------- | ----------- | ------------- | ---- | -------------- |
| uuid    | string      | bigint        | json | bigint         |

### Application (Record)

| id (pk) | account_id | tenant_id (fk) |
| ------- | ---------- | -------------- |
| string  | bigint     | bigint         |

### Service (Record)

| id (pk) | tenant_id (fk) |
| ------- | -------------- |
| string  | bigint         |

### Metric (Record)

| id (pk) | service_id (fk) | tenant_id (fk) |
| ------- | --------------- | -------------- |
| string  | bigint          | bigint         |

### UsageLimit (Record)

| id (pk) | metric_id (fk) | plan_id | tenant_id (fk) |
| ------- | -------------- | ------- | -------------- |
| string  | bigint         | bigint  | bigint         |

### Model

| id (pk) | record_id (fk) | tenant_id (fk) |
| ------- | -------------- | -------------- |
| uuid    | bigint         | bigint         |

### Update

| id (pk) | model_id (fk) | tenant_id (fk) |
| ------- | ------------- | -------------- |
| uuid    | uuid          | bigint         |

### Entry

| id (pk) | update_id (fk) | data | tenant_id (fk) |
| ------- | -------------- | ---- | -------------- |
| uuid    | uuid           | json | bigint         |

### Status

| id (pk) | integration_id (fk) | model_id (fk) | tenant_id (fk) |
| ------- | ------------------- | ------------- | -------------- |
| uuid    | uuid                | uuid          | bigint         |

zync

zync Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

zync

zync Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README