The input to this topology is the normalized Metron JSON produced by the Parser/Normalizing Topology. The output of this topology is written to a number of data stores supported by Metron. There are two streams: a message stream and an enrichment stream(s). The message stream carries the original message, while the enrichment stream tack on additional enrichments or pieces of threat intelligence to the message.
| Bolt Name | Functionality | References |
|---|---|---|
| Enrichment Splitter | This bolt extracts fields and values from a message that can be enriched and sends them to the appropriate enrichment bolt. The configuration for which fields have an associated enrichment is stored in Zookeper. | |
| Enrichment Bolt | This bolt takes the enrichment information from the splitter bolt (key + value), extracts the value, cross references the value against the enrichment store, and then sends the value of the enrichment to the joiner bolt. There can be n enrichment bolts and each enrichment bolt has to be associated with a back end store (which is primarily Hbase). These bolts also use an in-memory cache so they don't thrash the back end reference store. There is a corresponding bulk loader provided per enrichment to be able to bootstrap the enrichment store | Enrichments |
| Enrichment Joiner Bolt | Join the enrichments with the original message. The bolt waits for all the enrichments to come in prior to joining. If an enrichment part does not come in, then the bolt times out that enrichment and sends the message down the topology without that enrichment part. | |
| Threat Intel Splitter Bolt | Splits the message the same way the Enrichment Splitter does. Based on Zookeeper configs, parts of the message that can be enriched get passed to the Threat Intel bolt, which checks for threat data on that element. | |
| Threat Intel Bolt | This bolt takes the enrichment information from the splitter bolt (key + value), extracts the value, cross references the value against the Threat Intel store, and then sends the value of the enrichment to the joiner bolt. There can be n enrichment bolts and each enrichment bolt has to be associated with a back end store (which is primarily Hbase). These bolts also use an in-memory cache so they don't thrash the back end reference store. There is a corresponding bulk loader provided per enrichment to be able to bootstrap the enrichment store | Threat Intel |
| Threat Intel Joiner Bolt | Join the enrichments with the original message. The bolt waits for all the enrichments to come in prior to joining. If an enrichment part does not come in, then the bolt times out that enrichment and sends the message down the topology without that enrichment part. If there is threat intel for the message the bolt will add an element alert=true | |
| Writer Bolt | Extensible writer bolt to write Metron's (enriched) telemetry to a data store. See the references descriptions for the number of sources supported by Metron. More than one writer can be run in a topology, allowing multiple data stores to be populated with Metron data | See Supported Data Stores for a list of available extensions |