Metrics

Logprep provides a prometheus exporter with certain processing and connector metrics, e.g. logprep_number_of_processed_events_total or logprep_processing_time_per_event_sum.

Examples of grafana dashboards can be found in the logprep github repo

Configuration

Example

metrics:
  enabled: true
  port: 8000

The metrics configuration offers some options regarding the metrics export. Because logprep utilizes the prometheus python client the environment variable PROMETHEUS_MULTIPROC_DIR is required to be set by the user. This is a temporary directory where logprep will store files needed for in-between process communication. This folder has to be provided by the user because logprep won’t create it.

enabled

Use true or false to activate or deactivate the metrics exporter. Defaults to false.

port

Specifies the port which should be used for the prometheus exporter endpoint. Defaults to 8000.

Processing Times in Events

It is also possible to add processing times of each processor to the event itself. The processing times can then be found in the field processing_time of each processed event. Additionally, the hostname of the machine on which Logprep runs is listed. To activate this feature you have to set the environment variable LOGPREP_APPEND_MEASUREMENT_TO_EVENT with any value. This divergence of the usual configuration pattern is needed due to performance reasons.

Metrics Overview

General Metrics

class logprep.runner.Runner.Metrics

Metrics for the Logprep Runner.

number_of_events_in_error_queue: GaugeMetric: Current size of the error queue.

class logprep.framework.pipeline.Pipeline.Metrics

Tracks statistics about a pipeline

number_of_failed_events: CounterMetric: Number of failed events

processing_time_per_event: HistogramMetric: Time in seconds that it took to process an event

class logprep.abc.connector.Connector.Metrics

Tracks statistics about this connector

number_of_processed_events: CounterMetric: Number of successful events

processing_time_per_event: HistogramMetric: Time in seconds that it took to process an event

number_of_warnings: CounterMetric: Number of warnings that occurred while processing events

number_of_errors: CounterMetric: Number of errors that occurred while processing events

class logprep.processor.base.rule.Rule.Metrics

Tracks statistics about the current rule

number_of_processed_events: CounterMetric: Number of events that were processed

processing_time_per_event: HistogramMetric: Time in seconds that it took to process an event

number_of_warnings: CounterMetric: Number of warnings that occurred while processing events

number_of_errors: CounterMetric: Number of errors that occurred while processing events

Connector Specific

class logprep.connector.confluent_kafka.input.ConfluentKafkaInput.Metrics

Metrics for ConfluentKafkaInput

number_of_processed_events: CounterMetric: Number of successful events

processing_time_per_event: HistogramMetric: Time in seconds that it took to process an event

number_of_warnings: CounterMetric: Number of warnings that occurred while processing events

number_of_errors: CounterMetric: Number of errors that occurred while processing events

commit_failures: CounterMetric: count of failed commits. Is filled by _commit_callback

commit_success: CounterMetric: count of successful commits. Is filled by _commit_callback

current_offsets: GaugeMetric: current offsets of the consumer. Is filled by _get_raw_event

committed_offsets: GaugeMetric: committed offsets of the consumer. Is filled by _commit_callback

librdkafka_age: GaugeMetric: Time since this client instance was created (microseconds)

librdkafka_replyq: GaugeMetric: Number of ops (callbacks, events, etc) waiting in queue for application to serve with rd_kafka_poll()

librdkafka_tx: GaugeMetric: Total number of requests sent to Kafka brokers

librdkafka_tx_bytes: GaugeMetric: Total number of bytes transmitted to Kafka brokers

librdkafka_rx: GaugeMetric: Total number of responses received from Kafka brokers

librdkafka_rx_bytes: GaugeMetric: Total number of bytes received from Kafka brokers

librdkafka_rxmsgs: GaugeMetric: Total number of messages consumed, not including ignored messages (due to offset, etc), from Kafka brokers.

librdkafka_rxmsg_bytes: GaugeMetric: Total number of message bytes (including framing) received from Kafka brokers

librdkafka_cgrp_stateage: GaugeMetric: Time elapsed since last state change (milliseconds).

librdkafka_cgrp_rebalance_age: GaugeMetric: Time elapsed since last rebalance (assign or revoke) (milliseconds).

librdkafka_cgrp_rebalance_cnt: GaugeMetric: Total number of rebalance (assign or revoke).

librdkafka_cgrp_assignment_size: GaugeMetric: Current assignment’s partition count.

class logprep.connector.confluent_kafka.output.ConfluentKafkaOutput.Metrics

Metrics for ConfluentKafkaOutput

number_of_errors: CounterMetric: Number of errors that occurred while processing events

number_of_processed_events: CounterMetric: Number of successful events

number_of_warnings: CounterMetric: Number of warnings that occurred while processing events

processing_time_per_event: HistogramMetric: Time in seconds that it took to process an event

librdkafka_age: GaugeMetric: Time since this client instance was created (microseconds)

librdkafka_msg_cnt: GaugeMetric: Current number of messages in producer queues

librdkafka_msg_size: GaugeMetric: Current total size of messages in producer queues

librdkafka_msg_max: GaugeMetric: Threshold - maximum number of messages allowed allowed on the producer queues

librdkafka_msg_size_max: GaugeMetric: Threshold - maximum total size of messages allowed on the producer queues

librdkafka_tx: GaugeMetric: Total number of requests sent to Kafka brokers

librdkafka_tx_bytes: GaugeMetric: Total number of bytes transmitted to Kafka brokers

librdkafka_rx: GaugeMetric: Total number of responses received from Kafka brokers

librdkafka_rx_bytes: GaugeMetric: Total number of bytes received from Kafka brokers

librdkafka_txmsgs: GaugeMetric: Total number of messages transmitted (produced) to Kafka brokers

librdkafka_txmsg_bytes: GaugeMetric: Total number of message bytes (including framing, such as per-Message framing and MessageSet/batch framing) transmitted to Kafka brokers

Processor Specific Metrics

class logprep.processor.amides.processor.Amides.Metrics

Track statistics specific for Amides processor instances.

total_cmdlines: CounterMetric: Total number of command lines processed.

new_results: GaugeMetric: Number of command lines that triggered detection and rule attribution.

cached_results: GaugeMetric: Number of command lines that could be resolved from cache.

num_cache_entries: GaugeMetric: Absolute number of current cache entries.

cache_load: GaugeMetric: Relative cache load.

mean_misuse_detection_time: HistogramMetric: Mean processing time of command lines classified by the misuse detector.

mean_rule_attribution_time: HistogramMetric: Mean processing time of command lines attributed by the rule attributor.

class logprep.processor.domain_resolver.processor.DomainResolver.Metrics

Tracks statistics about the DomainResolver

total_urls: CounterMetric: Number of all resolved urls

resolved_new: CounterMetric: Number of urls that had to be resolved newly

resolved_cached: CounterMetric: Number of urls that were resolved from cache

timeouts: CounterMetric: Number of timeouts that occurred while resolving a url

invalid_domains: CounterMetric: Number of invalid domains that were trying to be resolved

unknown_domains: CounterMetric: Number of unknown domains that were trying to be resolved

class logprep.processor.pseudonymizer.processor.Pseudonymizer.Metrics

Tracks statistics about the Pseudonymizer

pseudonymized_urls: CounterMetric: Number urls that were pseudonymized

new_results: GaugeMetric: Number of new pseudodonyms

cached_results: GaugeMetric: Number of resolved from cache pseudonyms

num_cache_entries: GaugeMetric: Number of pseudonyms in cache

cache_load: GaugeMetric: Relative cache load.