Neblic Kafka Sampler¶

Configuration¶

By default, kafka-sampler* will look for a configuration file at */etc/neblic/kafka-sampler/config.yaml. This path can be changed using the --config flag when executing the service.

All the options defined in the configuration file can be configured/overridden using environment variables. The environment variable name needs to be written in all caps and use _ to divide nested objects. For example, to configure the Kafka server URL you would need to use the env variable KAFKA_SERVERS.

Internally, kafka-sampler uses the Sarama Go library to interact with Kafka and all its options can be configured under the kafka.sarama key. See the following examples section to see advanced configurations.

Examples¶

Topic monitoring selection¶

The maximum number of topics to monitor is defined by the key kafka.topics.max and by default, it will monitor the first max number of topics that match the filter rules (in no particular order), the rest will be ignored.

To configure what topics are selected you can use the key kafka.topics.filter.allow* or the key *kafka.topics.filter.deny, using both options at the same time is not supported. The value should follow regex RE2 syntax as described in here. For example, to only monitor topic1* and *topic2 topics:

Or to monitor all topics but topic3:

Take into account that kafka-sampler automatically discovers new topics so if the configuration is not too restrictive it will automatically monitor new topics as they are created.

Apache Kafka authentication¶

Kafka supports many authentication methods, since its configuration is not straightforward you can use these examples to get started:

SASL/PLAIN¶

SASL/SCRAM¶

Default configuration¶

logging:
  level: info

kafka:
  # required: Kafka bootstrap addresses, format "url1:port,url2:port".
  servers: kafka:9092

  # Consumer group that the Kafka consumers will use
  # consumergroup: neblic-kafka-sampler

  # Kafka configuration as defined in the Sarama library
  # https://pkg.go.dev/github.com/IBM/sarama#Config
  # Convert field names to all lowercase and nested structs to nested fields
  # sarama:  (...)

  # topics:
  #   # Safeguard to avoid consuming an unexpectedly large number of topics.
  #   # The first max number of topics that match the filter rules (in no particular order) will be monitored, the rest will be ignored.
  #   max: 25
  #
  #   # Topic list refresh period. In each refresh it will create/delete `Samplers` based on the cluster existing topics
  #   refreshperiod: 1m
  #
  #   # If unset, it will create a `Sampler` per each topic found in the Kafka cluster,
  #   # supports regex RE2 syntax as described at https://github.com/google/re2/wiki/Syntax, except for `\C`.
  #   # It always ignores internal topics like: `__consumer_offsets` and `__transaction_state`.
  #   filter:
  #     # Topics matching `allow` will be monitored, and topics matching `deny` will be ignored.
  #     # `allow` and `deny` options can't be set at the same time.
  #     allow: ^(topic1|topic2)$
  #     deny: ^topic3$

neblic:
  # `Sampler` resource name set to created `Samplers`
  # resourcename: kafka-sampler

  # `Control Plane` server address
  # controlserveraddr: localhost:8899

  # `Data Plane` server address
  # dataserveraddr: localhost:4317