stockdb

KO | EN

gitlite — search

#python #react #javascript #nodejs #deep-learning #python3 #css #covid-19 #hacktoberfest #typescript #php #reinforcement-learning

★ 92 Open GitHub ↗

Database for storing Stock Exchange quotes in erlang

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Stockdb
=======

This library is a storage for Stock Exchange quotes.

It is an append-only online-compress storage, that supports failover, indexing and simple lookups of stored quotes.

It can compress 400 000 daily quotes into 4 MB of disk storage.

Usage
=====

First install and compile it. Include it as a rebar dependency.

It can be used either as an appender, either as reader. You cannot mix these two modes now.

Writing database: Appender
==========================

Typical workflow when appending data to DB:

{ok, Appender} = stockdb:open_append('NASDAQ.AAPL', "2012-01-15", [{depth, 2}]),
    {ok, Appender1} = stockdb:append({md, 1326601810453, [{450.1,100},{449.56,1000}], [{452.43,20},{454.15,40}]}, Appender),
    stockdb:close(Appender1).

Now lets explain, what is happening.

* Open appender. Stock name should be a symbol, date should either erlang date `{YYYY,MM,DD}`, either a string `"YYYY-MM-DD"`.
* Don't forget to specify proper depth. If you skip it, default depth is 1 and you will save only best bid and best ask
* Specify also `{scale, 1000}` option, if you want to store quotes with precision less than 1 cent. Stockdb stores your prices as int: `round(Price*Scale)`
* Now append market data.
* Market data is following: ```{md, UtcMilliseconds, [{L1BidPrice,L1BidSize},{L2BidPrice,L2BidSize}..], [{L1AskPrice,L2AskSize}..]}```
* You can include ```-include_lib("stockdb/include/stockdb.hrl").``` to use ```#md{}``` and ```#trade{}``` records

Now take a look at db/stock folder. There you can see new file `db/stock/NASDAQ.AAPL-2012-01-15.stock` and now you can read back stocks from it.

Reading database
================

Read whole DB
-------------

The most simple way is just to read all daily events to replaying them

{ok, Events} = stockdb:events('NASDAQ.AAPL', "2012-01-15").

Get candle for whole day or specified time range:

DayCandleEvents = stockdb:events('NASDAQ.AAPL', "2012-08-10", [{filter, candle, [{period, undefined}]}]).
    RangeCandleEvents = stockdb:events('NASDAQ.AAPL', "2012-08-10", [{range, {15,0,0}, {16,0,0}}, {filter, candle, [{period, undefined}]}]).

But there are possible more enhanced ways of limiting amount of loaded data.

Iterator
--------

If you need something more complex than just getting all data from DB for stock/date pair, you can use iterators.
Iterator is database opened for read-only with (optionally) filters applied on it.
You can read iterator's events one-by-one, saving memory by not keeping extracted data.

Basic iterator is created as follows:

{ok, Iterator} = stockdb:init_reader('NASDAQ.AAPL', {2012, 8, 7}, []).

Here first argument is stock, second is date, third is list of filters (empty for basic case).

You can read events one-by-one using `stockdb:read_event/1` function:

{Event1, Iterator1} = stockdb:read_event(Iterator),
    {Event2, Iterator2} = stockdb:read_event(Iterator1).

When there are no more events, `eof` event is returned. Make sure your code handles it well!

Also, you can call `stockdb:events(Iterator)` to get all events from it.

Iterator filters
----------------

Iterator filters currently may be `{range, Start, End}` or `{filter, FilterFun, FilterState0}`.
`FilterFun` may be function name from module `stockdb_filters` (currently only `candle`) or
any function with arity 2 which returns list of emitted events. Filter must accept events from
previous filter (or `#md{}` and `trade{}` from DB) and `eof` to handle end of underlying source.
Return value is tuple with list of emitted events on first place and next state on second.

For example, this simple function drops every second event:

FilterFun = fun
        (eof, _State) -> {[], eof};
        (Event, true) -> {[Event], false};
        (_Event, _Other) -> {[], true}
      end.

And, we can see it is working:

25> length(stockdb:events(Iterator)).          
    20703
    27> {ok, FIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{filter, FilterFun, false}]),
    27> length(stockdb:events(FIterator)).
    10351

To use pre-defined filters you can just specify filter name:

28> {ok, CIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{filter, candle, [{period, 120000}]}]),
    28> length(stockdb:events(CIterator)).
    2242

StockDB index is optimized for fast timestamp seeking, so you can use `{range, Start, End}` pseudo-filter. Start and End (if defined)
are both millisecond timestamps or erlang-style `{HH, MM, SS}` tuples (tuples will work only over DB source, not over other iterator). `undefined` for `Start` or `End` means the very beginning or the very end respectively. Example:

31> {ok, RIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{range, {14,0,0}, {15,0,0}}]),
    31> length(stockdb:events(RIterator)).
    5139
    49> {ok, HIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{range, undefined, 1344348900451}]),
    49> length(stockdb:events(HIterator)).                                                                      
    1954

Of course, you may specify multiple filters:

32> {ok, RFCIterator} = stockdb:init_reader('NASDAQ.AAPL', "2012-08-07", [{range, {14,0,0}, {15,0,0}}, {filter, FilterFun, false}, {filter, candle, [{period, 120000}]}]),
    32> length(stockdb:events(RFCIterator)).
    372

Also, iterators may cascade:

35> {ok, RIterator_F} = stockdb:init_reader(RIterator, [{filter, FilterFun, false}]),
    35> {ok, RIterator_F_C} = stockdb:init_reader(RIterator_F, [{filter, candle, [{period, 120000}]}]),
    35> length(stockdb:events(RIterator_F_C)).
    372
    36> stockdb:events(RIterator_F_C) == stockdb:events(RFCIterator).
    true

Self-sufficient read-only state
-------------------------------

Function `stockdb:init_reader/3` currently accesses file directly. If you have distributed setup, it will fail. Stockdb is able to bypass this by using a lower-level `stockdb:open_read/2`.
`open_read/2` returns in-memory read-only database state with full buffer and file descriptor closed. Actually, `stockdb:init_reader/3` first opens DB using `stockdb:open_read/2` and then calls `stockdb:init_reader/2` on it. So does `stockdb:events/2`. You can do the same:

42> {ok, S} = stockdb:open_read('NASDAQ.AAPL', {2012, 8,7}),
    42> {ok, Iterator} = stockdb:init_reader(S, []),
    42> stockdb:events(S) == stockdb:events(Iterator).
    true

Note that we still use the same Iterator which matches perfectly. `stockdb:init_reader(S, [])` can be called when original file is unavailable allowing to minimize network load when DB content is needed on other node.

Querying existing data
======================

There are simple functions which let you know what data you have.
* To list all stocks having any data in database, use `stockdb:stocks()`
* To list dates when some stock has any data, use `stockdb:dates(Stock)`
* To get date intersection between multiple stocks, use `stockdb:common_dates([Stock1, Stock2, ...])`
* To get some information about file, stockdb instance or stock/date pair, use `stockdb:info(Stockdb)`, `stockdb:info(Filename)`, `stockdb:info(Stock, Date)`, `stockdb:info(Stock, Date, [Key1, Key2, ...])`. Key can be one of `path, stock, date, version, scale, depth, chunk_size, presence`. Return value is tuplelist. Presence is `{ChunkCount, [ChunkNumber1, ChunkNumber2, ...]}`, representing some internal report.

stockdb

Was this content helpful?

Recent Feedback

Download README