elastic-grok-script-plugin

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

elastic-grok-script-plugin

★ 12 Open GitHub ↗

Elastic-Grok-Script-Plugin is a provider of Grok ElasticSearch plug-in

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Introduction

Elastic-Grok-Script-Plugin is a provider of Grok ElasticSearch plug-in

# Install

* fork this project and change the `elasticsearch` in maven ,default is 2.3.3
* run `mvn assembly:assembly` ,you can find the release in `target/release`
* copy the `pattern` folder and `plugin-descriptor.properties` file to your plugin folder
* you can put your own pattern to pattern folder
* restart your elasticsearch
* Have Fun (○’ω’○)

# Params

Name|Desc
----|----
fieldName|which field in source do you want to grok
pattern| your grok pattern
groupkeys|you can use this param to get the field after grok or the file in source
,if you want to get multi fields ,you can use like this `a,b,c`
isHashMap|(optional) you can use this param to set how to return the result,string or hashmap,`true` or `false`

# Sample

* Use Grok to Extract Fields Extract Fields

{
          "query": {
            "match_all": {}
          },
          "_source": {
            "includes": [],
            "excludes": []
          },
          "script_fields": {
            "test2": {
              "params": {
                "fieldName": "message",
                "pattern": "%{WORD:wd}",
                "groupkeys": "wd"
              },
              "script": "grokscript",
              "lang": "native"
            }
          },
          "size": 1
        }

* Do Some Aggregation

{
          "query": {
            "match_all": {}
          },
          "_source": {
            "includes": [],
            "excludes": []
          },
          "aggs": {
            "counts_by_my_field": {
              "stats": {
                "params": {
                  "fieldName": "message",
                  "pattern": "%{WORD:wd}",
                  "groupkeys": "wd"
                },
                "script": "grokscript",
                "lang": "native"
              }
            }
          },
          "size": 1
        }

* GroupBy Extract Field and Do More Aggregation

{
            "query": {
                "match_all": {}
            },
            "_source": {
                "includes": [],
                "excludes": []
            },
            "aggs": {
                "counts_by_my_field": {
                    "terms": {
                        "params": {
                            "fieldName": "message",
                            "pattern": "%{WORD:wd}",
                            "groupkeys": "wd"
                        },
                        "script": "grokscript",
                        "lang": "native"
                    },
                    "aggs": {
                        "agg1": {
                            "terms": {
                                "field": "http_method"
                            }
                        }
                    }
                }
            },
            "size": 1
        }

# Metrics

Use ApacheAccessLog and use COMMONAPACHELOG Pattern

number of lines| cost
---------|-------
100 |0.2s
1000 |0.5s
10000 |19s
100000 |3m17s

Use ApacheAccessLog and use WORD Pattern

number of lines| cost
---------|-------
100 |0.03s
1000|0.26s
10000|2.9s
100000|34s

So, I Think The Performance bottlenecks is the grok lib ,and the grok performance is depency with your Gork Pattern

# Howto

* if you want to load the Grok Pattern from other place,you can implement the `IGrokPatternLoader`

elastic-grok-script-plugin

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README