Tag: kafka

ricardo gomez angel j5gCOKZdm6I unsplash

Apache Avro – Effective Big Data Serialization Solution for Kafka

November 15, 2020 / RainerGewalt / 0 Comments

In this article we will explain everything you need to know about Apache Avro, an open source big data serialization solution and why you should not do without it.

You can serialize data objects, i.e. put them into a sequential representation, in order to store or send them independent of the programming language. The text structure reflects your data hierarchy. Known serialization formats are for example XML and JSON. If you want to know more about both formats, read our articles on the topics. To read, you have to deserialize the text, i.e. convert it back into an object.

In times of Big Data, every computing process must be optimized. Even small computing delays can lead to long delays with a correspondingly large data throughput, and large data formats can block too many resources. The decisive factors are therefore speed and the smallest possible data formats that are stored. Avro is developed by the Apache community and is optimized for Big Data use. It offers you a fast and space-saving open source solution. If you don’t know what Apache means, look here. Here we have summarized everything you need to know about it and introduce you to some other Apache open source projects you should know about.

Apache Avro – Open Source Big Data Serialization Solution

With Apache Avro, you get not only a remote procedure call framework, but also a data serialization framework. So on the one hand you can call functions in other address spaces and on the other hand you can convert data into a more compact binary or text format. This duality gives you some advantages when you have cross-network data pipelines and is justified by its development history.

Avro was released back in 2011 as a part of Apache Hadoop. Here, Avro was supposed to provide a serialization format for data persistence as well as a data transfer format for communication between Hadoop nodes. To provide functionality in a Hadoop cluster, Avro needed to be able to access other address spaces. Due to its ability to serialize large amounts of data, cost-efficiently, Avro can now be used Hadoop-independently.

You can access Avro via special API’s with many common programming languages (Java, C#, C, C++, Python and Ruby). So you can implement it very flexible.

In the following figure we have summarized some reasons what makes the framework so ingenious. But what really makes Avro so fast?

The schema clearly shows all the features that Apache Avro offers the user and why he should use it — Features Apache Avro

What makes Avro so fast?

The trick is that a schema is used for serialization and deserialization. About that the data hierarchy, i.e. the metadata, is stored separately in a file. The data types and protocols are defined via a JSON format. These are to be assigned unambiguously by ID to the actual values and can be called for the further data processing constantly. This schema is sent along with the data exchange via RPC calls.

Creating a schema registry is especially useful when processing data streams with Apache Kafka.

Apache Avro and Apache Kafka

Here you can save a lot of performance if you store the metadata separately and call it only when you really need it. In the following figure we have shown you this process schematically.

When you let Avro manage your schema registration, it provides you with comprehensive, flexible and automatic schema development. This means that you can add additional fields and delete fields. Even renaming is allowed within certain limits. At the same time, Avro schema is backward and forward compatible. This means that the schema versions of the Reader and Writer can differ. Schema registration management solutions exist, with Google Protocol Buffers and Apache Thrift, among others. However, the JSON data structure makes Avro the most popular choice.

ksqlDB – Efficient real-time stream transformation of data within Kafka’s data pipelines

November 1, 2020 / RainerGewalt / 0 Comments

ksqlDB vs Kafka streams – Data streams are all the rage right now. A technique to move and process huge amounts of data simultaneously without caching it.

What is Apache Kafka?

With the messagebroker Kafka, the data can be stored resource-efficiently in so-called topics as so-called logs. These topics can then be subscribed to and rewritten by any number of clients, primarily microservices.
The metadata information is stored externally in a schemaregistry and assigned to the data again via an ID when it is read. In this way, each microservice can be developed independently of technology and programming languages. The data structure remains the same.

However, if a microservice wants to access the data streams from two or more topics and these arrive with different frequencies, then the correct allocation of the data is often difficult. The so-called data stream position can be controlled with event streaming databases.

What is ksqlDB?

Especially for Apache Kafka, ksqlDB allows easy transformation of data within Kafka’s data pipelines.

The following figure shows how a software architecture with Apache Kafka and ksqlDB could look like. It is still possible to subscribe to the data streams from the messagebroker, or indirectly via ksqlDB using pulls and pushs. The communication between table and kafka is done directly via the eventstreaming platform Confluent.

The figure shows how a software architecture with Apache Kafka and ksqlDB could look like. — software architecture with Apache Kafka and ksqlDB

It can be used to materialize views asynchronously using interactive SQL queries.
So with this, microservices can enrich the data and transform it in real time.
This enables anomaly detection, real-time monitoring, and real-time data format conversion.

Event Streaming

ksqlDB is an event streaming database. Thus, it is based on continuous streams of structured event data that can be published to multiple applications in real time. The following figure shows such an event stream schematically.

ksqlDB vs Kafka streams- The figure shows such an event stream schematically. — event stream

Each individual record always consists of an event and a unique key for identification.
These event streams can be combined with streaming analytics and is a way to offload work to back-end processing applications. If you want to know more about messaging patterns and how a message is transmitted between sender and receiver, read our article.

Window-based Query Processing

ksqlDB allows continuous stream queries. These are based on window-based aggregation of events.

Windows are polling intervals that are continuously executed over the data streams. These windows can be expanded and moved as needed to handle new incoming data items.
Several window types are shown in the figure below. They differ in their composition to each other.

ksqlDB - Several window types are shown in the figure. They differ in their composition to each other. — window types

The “Tumbling” type repeats a non-overlapping interval, while the “Bouncing” type allows overlaps. In a “Session” the elements are grouped by activity sessions without allowing overlaps. The session is terminated when no elements are received for a certain time.

ksqlDB Features

In addition to continuous queries through window-based aggregation of events, ksqlDB offers many other features that are helpful in dealing with streams. For example, the last value of a column can be tracked when aggregating events from a stream into a table.

Multiple streams can be merged by real-time joins or transformed in real-time. In doing so, the database is Distributed, Fault Tolerant and Scalable.
The Kafka Connect connectors can be executed and controlled directly.
Push and pull queries are applicable to the flows. Thus, subscribers get the constantly updated results of a query, or can retrieve data in request/response flows at a specific time.

Conclusion

With Confluent’s event streaming database ksqlDB, a service is provided that offers an absolutely compatible solution for real-time data stream processing with Kafka. Kafka in particular lends itself as a central element in a microservice-based software architecture. Microservices run as separate processes and consume in parallel from the message broker. Aligning these processes remains a challenge. However, ksqlDB ensures real-time stream processing within the services.