Event Streams, Brokers, Pub/Sub and Topics

A event stream basically is a stream of messages that carry payloads of data. Brokers are a piece of server software that use the pub/sub concept to manage event streams. Pub/Sub is the short form for publish and subscribe. Topics are unique strings that serve as a id and help to group the data in the event stream.

Software components connect to one or multiple topics in order to send and receive messages. To send a message, a software component called “publisher” simply pushes a message to a topic. Other software components called “subscribers” are connected to this topic. Every message that is published to one certain topic will automatically be received by every other software component that has subscribed to this topic. See the following diagram.

In this scenario the blockchain app has subscribed to two topics: “RFID Readings” and “Temperature”.
The Microservice has subscribed to the topic “Temperature” only.

The broker keeps a list of all topics and of all subscribers and whenever a publisher publishes a message to a certain topic the broker forwards this message to all subscribers that have subscribed to this topic. The message has a payload (usually encoded in the JSON format).

But the other important information that a message carries is the event itself. So if a motion sensor publishes data to a topic (e.g. to a topic named “motionsensor”) the payload will contain data about the sensor reading. But the pure fact that the message has been published is also an information. Keep in mind that an event stream carries the message payload plus the fact that the event happend just now.

So pub/sub messaging can be used to build event-driven real-time applications. In addition it also allows to decouple applications in order to increase performance, reliability and scalability.

Another very important point: A broker can run in the cloud (secured by TLS and digital certificates) while the publishing and subscribing software components run on-premises in a local environment. The broker architecture is firewall friendly because both subscriber and publisher initiate the connection to the broker from inside the firewall.

Meanwhile there are many broker based pub/sub event stream protocols on the market. Very important in my opinion are MQTT and Apache Kafka. But also have a look at the others.

Streamsheets come bundled with the Open Source MQTT Broker “Mosquitto”. Cedalo is proud to be the official commercial sponsor of this project.