stealthwatch-data-exporter

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

stealthwatch-data-exporter

★ 13 Open GitHub ↗

Reference implementation of a Stealthwatch Data Exporter

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Data Exporter Client
====================

This project is the Java reference implementation of a Data Exporter Client.

The Flow Forwarder Docker Container on the Flow Collector listens to Flow
events from the Engine via ZeroMQ. It sends stitched, de-duplicated flow records 
over a secure web socket to registered clients.

![Overview](https://raw.githubusercontent.com/CiscoDevNet/stealthwatch-data-exporter/master/doc/dex.svg)

The flow records are sent once per minute as a `java.io.ByteBuffer` containing 
[Google Protocol Buffer](https://developers.google.com/protocol-buffers) representations 
of de-duplicated, stitched netflows.

Dependencies
------------

* Java 8 JDK
* Maven 3.3.9
* bash
* openssl

Java Cryptography Extension (JCE)
---------------------------------

Download the [JCE](http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html).

Unzip the downloaded zip.

Copy `local_policy.jar` and `US_export_policy.jar` to the `$JAVA_HOME/jre/lib/security` directory.

(Note: these jars will be already there so you have to overwrite them).

Setup Instructions
------------------

Build the Data Exporter Application

mvn clean package

The Flow Collector requires clients to connect over a secure web socket connection.
This means you need to:
 
1. Create a Certificate Authority
2. Create Data Exporter keys.
3. Prepare a Certificate Signing Request for the Data Exporter Keys. 
4. Sign the Data Exporter public key by the Certificate Authority.
5. Create a Trust Store with the Flow Collector's certificate for the Data Exporter.
6. Provision the Flow Collector with the Certificate Authority's certificate.

The `bin/generate-certs-and-keystore` script accomplishes steps 1-5.  All created certs
appear in the `./certs` directory.

$ bin/generate-certs-and-keystore flow-collector-ip-or-hostname
    
Import `certs/data-exporter-certificate-authority.crt` into the Flow Collector via the 
`Configuration -> Certificate Authority Certificates` menu.

The script generates these files:

* `data-exporter-certificate-authority.key` – the certificate authority key

* `data-exporter-certificate-authority.crt` – the certificate authority certificate 
    imported into Stealthwatch.

* `data-exporter.key` – the key of the data exporter application

* `data-exporter.csr` – the certificate signing request used to ask the certificate 
    authority for a certificate

* `data-exporter.crt` – the data exporter certificate signed by the certificate authority

* `data-exporter.pkcs12` – the data exporter’s key and certificate stored in pkcs12 format

* `{flow-collector-alias}.crt` – the flow collector's certificate

* `data-exporter-truststore.pkcs12` – the trust store holding the flow collector’s certificate.

The `data-exporter.pkcs12` and `data-exporter-truststore.pkcs12` are used by log-flows to setup 
the secure web socket connection.

Stealthwatch prior to 7.0
-------------------------

Import

certs/data-exporter-certificate-authority.crt
    
into the Flow Collector via the `Configuration -> Certificate Authority Certificates` menu.

Stealthwatch 7.0 and after
--------------------------

Import

certs/data-exporter-certificate-authority.crt
    certs/data-exporter.crt

into the Flow Collector via the Central Management.

Running the application
-----------------------

The `bin/log-flows` script runs the application using the certificates created above.
It requires the Flow Collector host name or IP address:

bin/log-flows 10.0.37.14

Once the Data Exporter is running the `./log` directory is created, and two log files 
are created:
    * `system.log` holds operational and error messages.
    * `messages.log` holds all flow record messages from the flow collectors(s).

Both the system and message log output appear in the console. To change this behavior,
modify the `./src/main/resources/log4j.properties` file.

Protocol Buffers
----------------

The protocol buffers used for reading flows are stored in src/main/resources/protos.
These get compiled into java classes during the package life cycle via 
com.google.protobuf.protoc for the local operating system..

The code generated by the protocol buffer compiler is under the target directory.

Flow Collector Notes
--------------------

The Flow Collector has a Docker Container Service named "Data Exporter", that is is
responsible for for exporting flows.  Use the Flow Collector `admin` UI to start or 
stop the container.

The actual container name is `flow-forwarder`.  Its logs are found in
`/lancope/var/logs/containers/docker/flow-forwarder.log`.

The `flow-forwarder` waits for websocket connection attempts on 
`wss://<hostname-or-ip>/flowforwarder/websocket` or
`ws://<hostname-or-ip>:8092/websocket`.

NOTE: port 8092 is closed by default on the flow collector - only connections
originating from the Flow Collector itself are supported.

stealthwatch-data-exporter

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README