KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #developer-tools #cli #codex #claude-code #dsh-plugin #mcp #react #agent #dsh #deepseek-harness

log4stash

★ 60 Open GitHub ↗

Module to Log log4net Messages to ElasticSearch

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

log4stash
=====================

> **NOTE:** This package is no longer maintained, I will be happy to get help with maintaing it.

[![Build status][AzureDevOpsImg]][azure-devops]

log4stash is a [log4net](http://logging.apache.org/log4net/) appender to log messages to the [ElasticSearch](http://www.elasticsearch.org) document database. ElasticSearch offers robust full-text search engine and analyzation so that errors and messages can be indexed quickly and searched easily.

log4stash provides a few logging filters similar to the filters on [logstash](http://logstash.net).

The origin of log4stash is [@jptoto](https://github.com/jptoto)'s [log4net.ElasticSearch](https://github.com/jptoto/log4net.ElasticSearch) repository.

log4stash is based on RestSharp and Newtonsoft.Json but uses ILRepack to avoid nuget dependencies.

**log4stash is fully open source, MIT licensed.**

### Features:
* Supports .NET 4.5.2+ and .NET Core 2.0+
* Easy installation and setup via [Nuget](https://nuget.org/packages/log4stash/)
* Ability to analyze the log event before sending it to ElasticSearch using built-in filters and custom filters similar to [logstash](http://logstash.net/docs/1.4.2/).

### Breaking Changes:
Navigate to breaking changes page [here](https://github.com/urielha/log4stash/blob/master/docs/breaking.md). See also [Version notes](https://github.com/urielha/log4stash/blob/master/docs/version_notes.md) page.

### Filters:
* [**Add**][docs-filters-add] - add new key and value to the event.
* [**Remove**][docs-filters-remove] - remove key from the event.
* [**Rename**][docs-filters-rename] - rename key to another name.
* **Kv** - analyze value (default is to analyze the 'Message' value) and export key-value pairs using regex (similar to logstash's kv filter).
* **Grok** - analyze value (default is 'Message') using custom regex and saved patterns (similar to logstash's grok filter).
* **ConvertToArray** - split raw string to an array by given seperators. 
* **Json** - convert json string to an object (so it will be parsed as object in elasticsearch).
* **Convert** - Available convertors: `ToString`, `ToLower`, `ToUpper`, `ToInt` and `ToArray`. See [config example][config-example] for more information. 
* **Xml** - Parse xml into an object.

#### Custom filter:
To add your own filters you just need to implement the interface IElasticAppenderFilter on your assembly and configure it on the log4net configuration file.

### Issues:
I do my best to reply to issues or questions ASAP. Please use the [ISSUES](https://github.com/urielha/log4stash/issues) page to submit questions or errors.

### Configuration Examples:

Almost all the parameters are optional, to see the default values check the [c'tor](https://github.com/urielha/log4stash/blob/master/src/log4stash/ElasticSearchAppender.cs#L86) of the appender and the c'tor of every filter. 
You can also set any public property in the appender/filter which didn't appear in the example.

##### Simple configuration:
```xml
<appender name="ElasticSearchAppender" type="log4stash.ElasticSearchAppender, log4stash">
    <Server>localhost</Server>
    <Port>9200</Port>
    <ElasticFilters>
      
      <kv /> 
    </ElasticFilters>
</appender>
```

##### (Almost) Full configuration:
```xml
<appender name="ElasticSearchAppender" type="log4stash.ElasticSearchAppender, log4stash">
	<Server>localhost</Server>
	<Port>9200</Port>
	
	<Path>/es5</Path>
	
	<IndexName>log_test_%{+yyyy-MM-dd}</IndexName>
	
	<IndexType>LogEvent</IndexType>
	<BulkSize>2000</BulkSize>
	<BulkIdleTimeout>10000</BulkIdleTimeout>
	<IndexAsync>False</IndexAsync>
	<DropEventsOverBulkLimit>False</DropEventsOverBulkLimit>

<SerializeObjects>True</SerializeObjects>

<ElasticSearchTimeout>10000</ElasticSearchTimeout>

<Ssl>False</Ssl>
	<AllowSelfSignedServerCert>False</AllowSelfSignedServerCert>

<IndexOperationParams>
		<Parameter>
			<Key>_id</Key>
			<Value>%{IdSource}</Value>
		</Parameter>
		<Parameter>
			<Key>key</Key>
			<Value>value</Value>
		</Parameter>
	</IndexOperationParams>

<FixedFields>Partial</FixedFields>

<Template>
		<Name>templateName</Name>
		<FileName>path2template.json</FileName>
	</Template>

<AuthenticationMethod>
		
		<Basic>
			<Username>Username</Username>
			<Password>Password</Password>
		</Basic>
		
		<Aws>
			<Aws4SignerSecretKey>Secret</Aws4SignerSecretKey>
			<Aws4SignerAccessKey>AccessKey</Aws4SignerAccessKey>
			<Aws4SignerRegion>Region</Aws4SignerRegion>
		</Aws>
		
		<ApiKey>
			
			<ApiKeyBase64>aWQ6YXBpa2V5</ApiKey>
			
			<Id>id</Id>
			<ApiKey>apikey</ApiKey>
		</ApiKey>
	</AuthenticationMethod>

<ElasticFilters>
		<Add>
			<Key>@type</Key>
			<Value>Special</Value>
		</Add>

<Add>
			<Key>SmartValue</Key>
			<Value>the type is %{@type}</Value>
		</Add>

<Filter type="log4stash.Filters.RenameKeyFilter, log4stash">
			<Key>SmartValue</Key>
			<RenameTo>SmartValue2</RenameTo>
		</Filter>

<Json>
			<SourceKey>JsonRaw</SourceKey>
			<FlattenJson>false</FlattenJson>
			
			<Separator>_</Separator> 
		</Json>

<Xml>
			<SourceKey>XmlRaw</SourceKey>
			<FlattenXml>false</FlattenXml>
		</Xml>

<Kv>
			<SourceKey>Message</SourceKey>
			<ValueSplit>:=</ValueSplit>
			<FieldSplit> ,</FieldSplit>
		</kv>

<Grok>
			<SourceKey>Message</SourceKey>
			<Pattern>the message is %{WORD:Message} and guid %{UUID:the_guid}</Pattern>
			<Overwrite>true</Overwrite>
		</Grok>

<ConvertToArray>
			<SourceKey>someIds</SourceKey>
			
			<Seperators>, </Seperators> 
		</ConvertToArray>

<Convert>
			
			<ToString>shouldBeString</ToString>

<ToArray>
				<SourceKey>anotherIds</SourceKey>
			</ToArray>
		</Convert>
	</ElasticFilters>
</appender>
```

Note that the filters got called by the order they appeared in the config (as shown in the example).

### Templates:
To get to know the [ElasticSearch templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html) follow the link.

Sample template could be found in: [log-index-spec.json](https://github.com/urielha/log4stash/blob/master/scripts/log-index-spec.json). And more complex template with dynamic mappings can be found in the tests template: [template.json](https://github.com/urielha/log4stash/blob/master/src/log4stash.Tests/template.json)

You can follow the link to read more about [dynamic mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/default-mapping.html).

### License:
[MIT License](https://github.com/urielha/log4stash/blob/master/LICENSE)

### Thanks:

Thanks to [@eran-gil](https://github.com/eran-gil) for helping me updating this package to support newer versions of ES, creating continous deployment and fixing issues.

Thanks to [@jptoto](https://github.com/jptoto) for the idea and the first working ElasticAppender.
Many thanks to [@mpdreamz](https://github.com/Mpdreamz) and the team for their great work on the NEST library!
The inspiration to the filters and style had taken from [elasticsearch/logstash](https://github.com/elasticsearch/logstash) project.

### Build status:

The CI is running on Azure DevOps and tested against ElasticSearch 5,6,7 every time.
Support for lower ElasticSearch versions is no longer maintained.

[AzureDevOpsImg]:https://erangil.visualstudio.com/log4stash/_apis/build/status/log4stash%20CI?branchName=master

[azure-devops]:https://azure.microsoft.com/en-us/services/devops/

[config-example]:https://github.com/urielha/log4stash#almost-full-configuration
[filters-section]:https://github.com/urielha/log4stash#filters

[docs-filters-add]:https://github.com/urielha/log4stash/blob/master/docs/Filters/Add.md
[docs-filters-remove]:https://github.com/urielha/log4stash/blob/master/docs/Filters/Remove.md
[docs-filters-rename]:https://github.com/urielha/log4stash/blob/master/docs/Filters/Rename.md

[erangil2]:https://github.com/erangil2
[ignasv]:https://github.com/ignasv
[mfpalladino]:https://github.com/mfpalladino

log4stash

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README