Get data from Apache Kafka¶

Neblic provides a standalone service called kafka-sampler* capable of monitoring your *Apache Kafka* topics by automatically creating *Samplers that will read the data that flows through them.

Deployment¶

See the releases page to download the latest binary or the packages page to see the available containers. It is recommended to use the provided container image to deploy kafka-sampler. The following section describes how to deploy it using the container image.

Container¶

Supported architectures¶

For now, only x86-64 builds are offered. If you need another architecture you can build your own container using the files found in here.

Examples¶

docker-compose¶

version: "2.1"
services:
  neblic-kafka-sampler:
    image: ghcr.io/neblic/kafka-sampler:latest
    container_name: neblic-kafka-sampler
    environment:
      - KAFKA_SERVERS=kafka01:9092;kafka02:9092
      - NEBLIC_CONTROLSERVERADDR=neblic-otelcol:8899
      - NEBLIC_DATASERVERADDR=neblic-itelcol:4317
    restart: unless-stopped

kubernetes¶

apiVersion: apps/v1
kind: Deployment
metadata:
  name: neblic-kafka-sampler
  labels:
    app: neblic-kafka-sampler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: neblic-kafka-sampler
  template:
    metadata:
      labels:
        app: neblic-kafka-sampler
    spec:
      containers:
      - name: neblic-kafka-sampler
        image: ghcr.io/neblic/kafka-sampler:latest
        env:
          - name: KAFKA_SERVERS
            value: kafka.default.svc.cluster.local:9092
          - name: NEBLIC_CONTROLSERVERADDR
            value: neblic-otelcol.default.svc.cluster.local:8899
          - name: NEBLIC_DATASERVERADDR
            value: neblic-otelcol.default.svc.cluster.local:4317

Usage¶

On startup, it will subscribe to all or a subset (based on your configuration, see the reference page) of your Kafka topics and create a Sampler per each one. No other actions are required since it will automatically register the Samplers with Neblic's Control Plane server and keep the list of Samplers updated if topics are added or removed.

Configuration¶

To see all the configuration options available, check the kafka-sampler reference page.