KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agent #developer-tools #codex #typescript #cli #deepseek-harness #claude-code #ai-agents #mcp #rust

ingest_eval_examples

★ 34 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# INGEST_EVAL examples

A Splunk app containing worked examples of index-time data transformation using `INGEST_EVAL`, `CLONE_SOURCETYPE`, `REGEX`, `ADD_META`, `REPEAT_MATCH`, and related mechanisms. Each example ships with sample data, a commented `props.conf` / `transforms.conf` snippet, and a writeup explaining the problem, the approach, and how to search the result.

This is a reference / teaching app — not something you run standalone.

## Installing

1. Copy this repository into your Splunk apps directory as `ingest_eval_examples`:

```
   cp -r ingest_eval_examples $SPLUNK_HOME/etc/apps/
   ```

The directory name matters — `inputs.conf` uses absolute `$SPLUNK_HOME/etc/apps/ingest_eval_examples/...` paths to monitor the bundled sample data.

2. Restart Splunk:

```
   $SPLUNK_HOME/bin/splunk restart
   ```

3. Open the app from the Splunk nav ("INGEST_EVAL examples"). The dashboard lists each example; click through to read the description and view the underlying config.

The app creates three indexes on install (`ingest_eval_examples_1`, `ingest_eval_examples_2`, `shard_data_with_splitbyindexkeys`) plus a `detritus` last-chance index. See `default/indexes.conf`.

## Examples

Grouped by category (from `appserver/static/exampleInfo.json`):

### Datetime problems
- **conflicting_datetime_formats** — multiple datetime formats on a single sourcetype; worked around with `INGEST_EVAL` + `strptime`.
- **compound_datetimes** — date lives in the filename, time lives in the event; stitched together at ingest time.

### Enrichment
- **auto_extract_indexed_fields** — discover `attribute='value'` pairs dynamically using `REPEAT_MATCH=true` + `ADD_META=true`.
- **add_event_length** — attach `length(_raw)` to every event as indexed metadata for back-billing license usage.
- **drop_indexed_fields** — blend `INDEXED_CSV` with search-time extraction to keep `tstats` useful without bloating storage.
- **enrich_splunkd_component_level_thread** — extract `log_level`, `component`, and `thread` as indexed fields on `splunkd.log`.
- **enrich_splunkd_access_log** — extract IP, wait time, URL, and REST parameters from `splunkd_access.log`.
- **json_docker** — pre-process JSON-formatted Docker logs for better search performance.
- **find_ip_addresses** — pull likely-looking IPs out of arbitrary text via `REGEX` into indexed fields.

### Security
- **mask_and_clone** — simple double-indexing (one masked copy, one unmasked) for selective visibility.
- **mask_data_and_map** — SHA1-based masking with a separate "map" event for reversible decoding by privileged users.

### Platform extension
- **metricify_license** — clone `license_usage.log` into `_metrics` via `CLONE_SOURCETYPE`.
- **load_into_indexes** — export/import tool for moving data between Splunk instances.
- **shard_data_with_splitbyindexkeys** — group like data into buckets to improve bucket elimination.
- **data_estimation_pipeline** — measure ingestion rates before onboarding new data.

### Complex forwarding
- **split_forwarding** — break past the ~20 MB/s per-pipeline ceiling by spreading forwarder output across multiple TCP streams.
- **selective_routing** — fine-grained per-target routing using `CLONE_SOURCETYPE`.

## Documentation

- Per-example writeups live in [`appserver/static/description/`](appserver/static/description/). Each covers the problem, the approach, the config, and a sample SPL search.
- [`extended_documentation.md`](appserver/static/description/extended_documentation.md) covers `INGEST_EVAL` details that are absent from the public Splunk docs: `_time` vs `$pd:_time$`, multivalue handling (`$mv:field$`), `:=` vs `=`, type coercion (`[int]`, `[float32-sf]`), and guidance on high-cardinality string pitfalls.
- Sample data is generated by [`create_sample_data_sets.ipynb`](create_sample_data_sets.ipynb).

## Layout

```
default/        app.conf, props.conf, transforms.conf, indexes.conf, inputs.conf, ...
local/          local overrides
metadata/       ACLs (local.meta)
appserver/
  static/
    description/    per-example markdown writeups
    sample_data/    sample log files used by the examples
    icons/          per-example icons shown in the dashboard
    exampleInfo.json  index of examples rendered by the in-app dashboard
create_sample_data_sets.ipynb   notebook that regenerates sample data
```

## Author

rmorgan — app version 0.1.

ingest_eval_examples

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README