RounderDB.js

KO | EN

gitlite — search

#javascript #python #react #ai #llm #go #golang #deep-learning #java #awesome #kubernetes #security #ios #open-source

(★ 8)

A low-I/O, fixed size, round robin db with in-mem support. Store in RAM and sync periodically to disk.

RounderDB.js Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# RounderDB.js
## A low-I/O, fixed size, round robin db with in-mem support. Stores in RAM and syncs periodically to disk.

Build status: [![Build Status](https://travis-ci.org/aweijnitz/RounderDB.js.png)](https://travis-ci.org/aweijnitz/RounderDB.js)

## Description
RounderDB is a data logger. Each log, like for example CPU load reading, is called an *archive*. An *archive* keeps a number of *buckets* to store data in. The first bucket receives the added data points and data is then *aggregated up* to the remaining buckets in a chain, first -> second -> third bucket, etc. The aggregation method is configurable, and support is included for average, min, max and sum.

Each bucket is a fixed-size ring buffer of size N, where N is a configurable number. Aggregation up the chain occurs when N entries have been added to the a buffer, so going "full circle" triggers buffer aggregation.

Data is stored as tuples `[<val>, <timestamp>]`. The timestamp is either set explicitly when adding points, or will be added automatically. Out-of-sequence adding is currently not supported.

## Installation
RounderDB is a Node.js project, so you need to have Node.js setup before. There are lots of instructions out there, so google it if you don't have it already.

Provided Node.js is in place, either clone this repository, or in your project folder, do

npm install --save rounderdb

## Usage
The intended use for RounderDB is to be embedded in a host application The way to use it is something like this:

1. Create an instance of RounderDB (explicitly, or from config in a file)

2. Add a couple of Archives, each with a couple of DataBuckets (you don't need to worry about this step if you do the config file)

3. Add data to archives as it comes in, either with explicit time stamps, or using the default (timestamp created by RounderDB from server time).

### Examples

__Creating an instance__
```javascript
var db = new RounderDB();
var archive = new Archive();

archive.addBucket(new DataBucket(60, 'average'));
db.addArchive('cpuLoad', archive);

assert(db.getArchive('cpuLoad') instanceof Archive);
assert(db.getNrArchives() == 1);
```
__Loading a saved instance, or creating a new from config__
```javascript
var DBConf = require('./conf/default.conf.js');
var db = null;

if (fs.existsSync(DBconf.persistenceConf.dbFile)) {
    var obj = JSON.parse(fs.readFileSync(DBconf.persistenceConf.dbFile));
    db = RounderDB.loadInstance(obj);
} else
    db = RounderDB.createInstance(DBconf);
```

__Adding and reading data to/from an archive__
```javascript
db.add('loadAvg', 1.2);         // No timestamp. Use the local server time
db.add('loadAvg', 1.3, 1234);   // Explicit timestamp provided as last arg

// Get data for the first bucket in the archive 'loadAvg'
var arr = db.getArchive('loadAvg').getDataForBucket(0);
```

See the file `./test/test-RounderDB.js` and the example config file in `./test/fixutures/testConfig.persist.json` to get an idea about how to use it. Looking in `RounderDB.js` is of course also useful.

## Motivation
I wrote this to serve as the backing part of a small monitoring package I am building to monitor the status a Raspberry Pi. I am running the Pi as a headless server and need to monitor server status without too much impact on the other processes.

IO is really slow in the Raspberry Pi and after realizing that the RRDTOOL-based program I had downloaded took so much IO and CPU, that it was the major contribution to some monitoring graphs, I decided to build something that trades risk of data loss for reduced IO and CPU needs.

To achieve the goal of reduced IO I need a datastore that doesn't store so frequently. On the Pi, especially when storing on the SD card, it is better to write a large file infrequently, than writing a small amounts of data frequently and this is exactly what the RounderDB does; It will keep data archives in memory and store all archives in one go at regular intervals. Data that has not been stored is of course lost, but considering that the intended use is monitoring of server status, infrequent loss of some data points is acceptable.

Also, [see this short presentation](http://mildly-interesting.info/slides/MUCNodeJsUG/10-Oct-2013-RounderDB.html) I made for the Node.js UserGroup in Munich

## Running the tests
**Unit tests:**

npm test

**Integration tests**

mocha -R spec ./test/integration/*js

*If you don't a have Mocha installed, it's included as a dev dependency, so instead you can run*

./node_modules/mocha/bin/mocha -R spec ./test/

and

./node_modules/mocha/bin/mocha -R spec ./test/integration/*js

## Known issues
The major "issue", is probably the inefficient storage format (JSON serialization of the whole object structure to disk). As long as you have limited data amounts, this should not be a problem, but it could become one with large data sets. At that point, switching to something like RRDTOOL, Ganglia, Nagios, Graphite or Munin is probably a good thing. RounderDB is not an enterprise application.

Adding support for out of sequence messages would be nice, but since I currently don't need it, I haven't implemented it.

There are no known bugs at this point. Please let me know if you find one, or even better: Fork, fix it and send me a pull request (including a test of course ;-).

RounderDB.js

RounderDB.js Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

RounderDB.js

RounderDB.js Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README