masker

KO | EN

gitlite — search

#python #ai #llm #go #rust #linux #docker #windows #awesome #cli #kubernetes #pytorch #openai #open-source

(★ 9)

Composite data masking utility

masker 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# @qiwi/masker
> Composite data masking utility

[![CI](https://github.com/qiwi/masker/actions/workflows/ci.yaml/badge.svg?branch=master)](https://github.com/qiwi/masker/actions/workflows/ci.yaml)
[![Maintainability](https://api.codeclimate.com/v1/badges/6205424ac673cb3f2bb8/maintainability)](https://codeclimate.com/github/qiwi/masker/maintainability)
[![Test Coverage](https://api.codeclimate.com/v1/badges/6205424ac673cb3f2bb8/test_coverage)](https://codeclimate.com/github/qiwi/masker/test_coverage)

- [Digest](#digest)
    - [Purpose](#purpose)
    - [Status](#status)
    - [Roadmap](#roadmap)
    - [Key features](#key-features)
- [Getting started](#getting-started)
    - [Install](#install)
    - [Default preset](#default-preset)
    - [Custom pipeline](#custom-pipeline)
    - [Masking schema](#masking-schema)
    - [CLI](#cli)
    - [Playground](#playground)
- [Integration](#integration)
    - [Console](#console)
    - [Winston](#winston)
- [Design](#design)
- [Documentation](#documentation)
- [Contributing](#contributing)
- [Packages](#packages)
- [License](#license)

## Digest
### Purpose
Implement instruments, describe practices, contracts to solve sensitive data masking problem in JS/TS.
For secure logging, for public data output, for internal mimt-proxies (kuber sensitive-data-policy) and so on.

### Status
🚧 Work in progress / MVP#0 is available for testing  
⚠️ **Not ready for production yet**

### Roadmap
- [x] Implement masking composer/processor
- [x] Introduce (declarative?) masking directives: [schema](https://github.com/qiwi/masker/tree/master/packages/schema)
- [x] Describe masking strategies and add masking utils
- [x] Support logging tools integration

### Key features
* Both sync and async API
* Declarative configuration
* Deep customization
* TS and Flow typings

## Getting started
### Install
With npm:
```shell
npm install --save @qiwi/masker
```
or yarn:
```shell
yarn add @qiwi/masker
```

### Default preset
```ts
import {masker} from '@qiwi/masker'

// Suitable for most std cases: strings, objects, json strings, which may contain any standard secret keys/values or card PANs.
masker('411111111111111')       // Promise<4111 **** **** 1111>
masker.sync('4111111111111111') // 4111 **** **** 1111
```

### Custom pipeline
```ts
import {masker, registry} from '@qiwi/masker'

masker.sync({
  secret: 'foo',
  nested: {
    pans: [4111111111111111]
  },
  foo: 'str with printed password=foo and smth else',
  json: 'str with json inside {"secret":"bar"} {"4111111111111111":"bar"}',
}, {
  registry,           // plugin storage
  pipeline: [
    'split',          // to recursively process object's children. The origin `pipeline` will be applied to internal keys and values
    'pan',            // to mask card PANs
    'secret-key',     // to conceal sensitive fields like `secret` or `token` (pattern is configurable)
    'secret-value',   // to replace sensitive parts of strings like `token=foobar` (pattern is configurable)
    'json',           // to find jsons in strings
  ]
})

// result:
{
  secret: '***',      // secret-key
  nested: { // split
    pans: [ // split
      '4111 **** **** 1111' // pan
    ],  
  },
  foo: 'str with printed *** and smth else',  // secret-value
  // json
  // chunk#1: split, secret-key
  // chunk#2: split, pan (applied to key!)
  json: 'str with json inside {"secret":"***"} {"4111 **** **** 1111":"bar"}'
}
```

### Masking schema
Declare masker directives over [json-schema](https://json-schema.org/). See [@qiwi/masker-schema](https://github.com/qiwi/masker/tree/master/packages/schema) for details.
```ts
import {masker} from '@qiwi/masker';

masker.sync({
  fo: 'fo',
  foo: 'bar',
  foofoo: 'barbar',
  baz: 'qux',
  arr:  [4111111111111111, 1234123412341234]
}, {
  pipeline: ['schema'],
  schema: {
    type: 'object',
    properties: {
      fo: {
        type: 'string',
        maskKey: ['plain']
      },
      foo: {
        type: 'string',
        maskKey: ['plain']
      },
      foofoo: {
        type: 'string',
        maskKey: ['strike'],
        maskValue: ['plain']
      },
      arr: {
        type: 'array',
        items: {
          type: 'number',
          maskValue: ['pan']
        }
      }
    }
  }
})

// result:
{
  baz: 'qux',
  arr: [ '4111 **** **** 1111', '1234123412341234' ],
  '***': 'fo',
  '***(2)': 'bar',
  '******': '***',
}
```

### CLI
```shell script
npx masquer "4111 1111 1111 1111"
# returns 4111 **** **** 1111
```

### Playground
[codesandbox.io/s/qiwi-masker-sandbox-ngrnu](https://codesandbox.io/s/qiwi-masker-sandbox-ngrnu?file=/src/index.js)

## Integration
### Console
Override global console methods to print sensitive data free output to `stderr/stdout`:
```ts
import {masker} from '@qiwi/masker'

['log', 'info', 'error'].forEach(method => {
  const _method = console[method]
  console[method] = (...args: any[]) => _method(...args.map(masker))
})
```

### Winston
Create a [custom masker formatter](https://github.com/winstonjs/winston#formats), then attach it to your reporter / transport:
```ts
const winston = require('winston')
const {masker} = require('@qiwi/masker')

const logger = winston.createLogger({
  levels: winston.config.syslog.levels,
  transports: [
    new winston.transports.Console({
      format: winston.format.combine(
        winston.format((info) => Object.assign(info, masker.sync(info)))(),
        winston.format.json(),
      ),
    })
  ]
})

logger.log({
  level: 'info',
  message: {foo: 'bar', secret: 'foobar', pan: [4111111111111111, 1234123412341234]},
})

// stdout
{"level":"info","message":{"foo":"bar","secret":"***","pan":["4111 **** **** 1111","1234123412341234"]}}
```
[stackoverflow.com/how-to-make-a-custom-json-formatter-for-winston3-logger](https://stackoverflow.com/questions/51454523/how-to-make-a-custom-json-formatter-for-winston3-logger)

## Design
### Middleware
The masker bases on [the middleware pattern](https://www.google.com/search?q=middleware+pattern+javascript): it takes a piece of data and pushes it forward the `pipeline`. 
The output of each `pipe` is the input for the next one. Each pipe is a dual interface data processor:
```ts
export interface IMaskerPipe {
  name: IMaskerPipeName
  exec: IMaskerPipeAsync | IMaskerPipeDual
  execSync: IMaskerPipeSync | IMaskerPipeDual,
  opts?: IMaskerPipeOpts
}
```

### True dynamic
During the execution, every pipe handler takes full control of the `context`. It can override next steps, change the `executor` impl (replace, append hook, etc), 
create internal masker threads, parallelize invocation queues and sync them back together, and so on.

### Context
Each pipe is fed with a normalized context which consists of:
```ts
export interface IMaskerPipeInput {
  value: any                // value to process
  _value?: any              // pipe result
  id: IContextId            // ctx unique key
  context: IMaskerPipeInput // ctx self ref
  parentId?: IContextId     // parent ctx id
  registry: IMaskerRegistry       // pipe registry attached to ctx
  execute: IExecutor              // executor 
  sync: boolean                   // sync / async switch
  mode: IExecutionMode            // lagacy sync switch
  opts: IMaskerOpts               // current pipe options
  pipe?: IMaskerPipeNormalized              // current pipe ref
  pipeline: IMaskerPipelineNormalized       // actual pipeline
  originPipeline: IMaskerPipelineNormalized // origin pipeline
  [key: string]: any
}
```

### Sync / async
Both. In different situations, each approach has pros and cons.
For this reason, the masker provides a choice:
```ts
masker(data)                // async
masker.sync(data)           // sync
masker(data, {sync: true})  // sync
```

### Documentation
* [JS/TS API](https://github.com/qiwi/masker/blob/master/API.md)
* [CLI](https://github.com/qiwi/masker/blob/master/CLI.md)

## Packages
There is also a bunch of plugins, that extend the available masking scenarios. Please follow their internal docs.

| Package | Description | Version
|---|---|---
|[@qiwi/masker](https://github.com/qiwi/masker/tree/master/packages/facade)| Composite data masking utility with common pipeline preset | [![npm](https://img.shields.io/npm/v/@qiwi/masker/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker)
|[masquer](https://github.com/qiwi/masker/tree/master/packages/cli)| CLI for [@qiwi/masker](https://github.com/qiwi/masker/tree/master/packages/facade) | [![npm](https://img.shields.io/npm/v/masquer/latest.svg?label=&color=09e)](https://www.npmjs.com/package/masquer)
|[@qiwi/masker-common](https://github.com/qiwi/masker/tree/master/packages/common)| Masker common components: interfaces, executor, utils | [![npm](https://img.shields.io/npm/v/@qiwi/masker-common/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-common)
|[@qiwi/masker-debug](https://github.com/qiwi/masker/tree/master/packages/debug)| Debug plugin to observe pipe effects | [![npm](https://img.shields.io/npm/v/@qiwi/masker-debug/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-debug)
|[@qiwi/masker-infra](https://github.com/qiwi/masker/tree/master/packages/infra)| Infra package: build configs, tools, etc | [![npm](https://img.shields.io/npm/v/@qiwi/masker-infra/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-infra)
|[@qiwi/masker-json](https://github.com/qiwi/masker/tree/master/packages/json)| Plugin to search and parse JSONs chunks in strings | [![npm](https://img.shields.io/npm/v/@qiwi/masker-json/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-json)
|[@qiwi/masker-limiter](https://github.com/qiwi/masker/tree/master/packages/limiter)| Plugin to limit masking steps count and duration | [![npm](https://img.shields.io/npm/v/@qiwi/masker-limiter/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-limiter)
|[@qiwi/masker-pan](https://github.com/qiwi/masker/tree/master/packages/pan)| Plugin to search and conceal [PANs](https://en.wikipedia.org/wiki/Payment_card_number) | [![npm](https://img.shields.io/npm/v/@qiwi/masker-pan/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-pan)
|[@qiwi/masker-plain](https://github.com/qiwi/masker/tree/master/packages/plain)| Plugin to substitute any kind of data with `***` | [![npm](https://img.shields.io/npm/v/@qiwi/masker-plain/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-plain)
|[@qiwi/masker-schema](https://github.com/qiwi/masker/tree/master/packages/schema)| Masker schema builder and executor | [![npm](https://img.shields.io/npm/v/@qiwi/masker-schema/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-schema)
|[@qiwi/masker-secret-key](https://github.com/qiwi/masker/tree/master/packages/secret-key)| Plugin to hide sensitive data by key/path pattern match | [![npm](https://img.shields.io/npm/v/@qiwi/masker-secret-key/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-secret-key)
|[@qiwi/masker-secret-value](https://github.com/qiwi/masker/tree/master/packages/secret-value)| Plugin to conceal substrings by pattern match | [![npm](https://img.shields.io/npm/v/@qiwi/masker-secret-value/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-secret-value)
|[@qiwi/masker-split](https://github.com/qiwi/masker/tree/master/packages/split)| Executor hook to recursively process any object inners | [![npm](https://img.shields.io/npm/v/@qiwi/masker-split/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-split)
|[@qiwi/masker-strike](https://github.com/qiwi/masker/tree/master/packages/strike)| Plugin to ~~strikethough~~ any non-space string chars | [![npm](https://img.shields.io/npm/v/@qiwi/masker-strike/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-strike)
|[@qiwi/masker-trycatch](https://github.com/qiwi/masker/tree/master/packages/trycatch)| Executor hook to capture and handle exceptions | [![npm](https://img.shields.io/npm/v/@qiwi/masker-trycatch/latest.svg?label=&color=09e)](https://www.npmjs.com/package/@qiwi/masker-trycatch)

## Contributing
Feel free to open any issues: for bugs, feature requests or questions. 
You're always welcome to suggest a PR. Just fork this repo, write some code, add some tests and push your changes. 
Any feedback is appreciated.

## License
[MIT](https://github.com/qiwi/masker/blob/master/LICENSE)

masker

masker 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

masker

masker 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README