heka-stream-aggregator

KO | EN

gitlite — search

#javascript #python #react #ai #llm #android #docker #kubernetes #vue #pytorch #security #ios #mcp #open-source

heka-stream-aggregator

(★ 8)

No description available.

heka-stream-aggregator Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

heka-stream-aggregator
======================
Stream aggregation plugin for [Mozilla Heka](http://hekad.readthedocs.org/)

StreamAggregatorFilter
===========
This is a filter that can be used for aggregating multiple payloads into a single message.
It accepts an encoder option that will be used prior to aggregating each payload.
Each payload is separated by the delimiter config value.

Config:

- stream_aggregator_tag:
	Optional tagging for identifying new pack down the pipeline since you will lose any Fields previously held.
	This setting creates a new Heka Field called "StreamAggregatorTag" and is given the value of this option. Defaults to "aggregated"

- flush_interval: 
	Interval at which accumulated payloads should be compressed in milliseconds.
	Defaults to 1000 (i.e. one second)

- flush_bytes:
	Number of payloads that, if processed, will trigger them to be compressed.
	Defaults to 10.

- encoder:
	This option will run each Payload through the specified encoder prior to aggregating.(required)

Example:

[filter_stream_aggregator]
	type = "StreamAggregatorFilter"
	message_matcher = "Fields[decoded] == 'True'"
	stream_aggregator_tag = "aggregated"
	flush_interval = 30000
	flush_bytes = 1000000
	encoder = "ESLogstashV0Encoder"
	delimiter = ""
	
	[ESLogstashV0Encoder]
	index = "logstash-%{program}-%{2006.01.02}"
	type_name = "%{program}"
	es_index_from_timestamp = true
	id = "%{id}"
	
HTTP Output heka instance

[HttpOutput]
	message_matcher = "Fields[StreamAggregatorTag] == 'aggregated'"
	address = "http://es01.foo.bar:9200/_bulk"
	encoder	= "encoder_payload"

StreamSplitterDecoder
===========

Used inside of MultiDecoder for splitting a payload generated by a StreamAggregatorFilter

Example:

[multi_decoder]
	type = "MultiDecoder"
	order = ['zlib_decoder', 'split_decoder', 'json_decoder']

[multi_decoder.subs.zlib_decoder]
	type = "ZlibDecoder"

[multi_decoder.subs.split_decoder]
	type = "StreamSplitterDecoder"
	delimiter = "\n"

[multi_decoder.subs.json_decoder]
	type = "SandboxDecoder"
	script_type = "lua"
	filename = "/usr/share/heka/lua_decoders/json_decoder.lua"
	preserve_data = true

To Build
========

See [Building *hekad* with External Plugins](http://hekad.readthedocs.org/en/latest/installing.html#build-include-externals)
for compiling in plugins.

Edit cmake/plugin_loader.cmake file and add

add_external_plugin(git https://github.com/michaelgibson/heka-stream-aggregator master)

Build Heka:
	. ./build.sh

heka-stream-aggregator

heka-stream-aggregator Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

heka-stream-aggregator

heka-stream-aggregator Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README