Storage¶

All the Data Samples are represented as semi-structured data, such as JSON documents, and may not have a fixed schema. Therefore, any document database capable of efficiently storing and, more importantly, querying this data format can be used as a Data Sample storage.

Of course, you need some way to insert these Data Samples into the database, but since those are encoded as OpenTelemetry (OTLP) logs, you have two main options to do that:

• If the database supports ingestion of OTLP logs, use the OTLP/gRPC to send those.
• Use one of the supported contrib exporters to automatically transform OTLP logs to the representation supported by your database (for example clickhouse, elasticsearch, influx, etc.). It's possible that the provided Neblic collector does not include the exporter you want, if that's the case you will need to build your own collector.

You need to take into account that once you have stored them in a database, you want to be able to explore them easily and efficiently. For example, you want to be able to perform searches using expressions that can target semi-structured nested objects without a fixed schema (for example, if you have Data Sample with these contents: {id: "1", product_name: "T-Shirt", price: -10 }, you should be able to create an expression that targets this data similar to sample.price < 0). You may think that this is pretty common, but in practice, not many open-source databases allow you to easily explore data in this way.