Message passing

The online data pipeline uses a data broker as the central storage for active data. In this case the broker is Redpanda, an Apache Kafka compatible message broker.

Data transported in the online data pipeline is encoded using flatbuffers. Where possible, the same schemas as used by the ESS have been used, otherwise pipeline specific ones have been created.

The single source of truth for schemas used in the pipeline is here in the repository.