EXPERT KNOWLEDGE AT A GLANCE

Category: query

Apache Flink

Overview

– Open source stream processor framework developed by the Apache Software Foundation (2016)
– Data streams with high data volume can be processed and analyzed with low delay and high speed

flink analytics
Flink provides various tools for efficient real-time processing of continuous data streams and batch data

Core functions

– diverse, specialized APIs:
→ DataStream API (Stream Processing)
→ ProcessFunctions (control of states and time; event states can be saved and timers can be added for future calculations)
→ Table API
→ SQL API
→ provides a rich set of connectors to various storage systems such as Kafka, Kinesis, Kubernetes, YARN, HDFS, Elasticsearch, and JDBC database systems
→ REST API

Stream Processing

pexels pixabay 2438
How to handle this flood of data?

== Data is processed continuously with a short delay
→ without intermediate storage of the data in separate databases
– several data streams can be processed in parallel
– Each stream can be used to derive own follow-up actions and analyses

Architecture

Data can be processed as unbounded or bounded streams:

  • Unbounded stream

    • have a start but no defined end

    • must be continuously processed

  • Bounded stream

    • have a defined start and end

    • can be processed by ingesting all data before performing any computations(== batch processing)

– Flink automatically identifies the required resources based on the application’s configured parallelism and requests them from the resource manager.

In case of a failure, Flink replaces the failed container by requesting new resources.

– Stateful Flink applications are optimized for local state access





What is Microsoft’s .NET?

.NET

== collective term for several software platforms published by Microsoft (2000)
– for development and execution of application programs

Main components:

.NET Framework

.NET Core

Mono from Xamarin

overview dotNET

.NET Framework

is a run-time execution environment that manages apps that target

overview net frameworks

Tools

– Command line compiler (for Visual Basic.NET, C#, Jscript.NET)
– Software Development Kit (SDK) with tools, documentation and examples

Common Language Runtime (CLR)

– Runtime environment for the execution of a .NET application
– provides the just-in-time compiler
– provides numerous other basic services (garbage collector, exception handling, a security system and interoperability with non.NET applications)
– When a .NET application is started, Windows does not call the CLR itself directly, but first calls a so-called Runtime Host. This host loads the CLR and passes the entry point for the application to the CLR.

Runtime Hosts:
– Shell Runtime Host
– Internet Explorer Runtime Host
– ASP.NET

Language Integration

– enables the cooperation of different languages
→ Calling a program code written in another programming language
→ object-oriented languages can inherit classes written in another object-oriented language

Uniform class library


– NET Framework Class Library (FCL)
– extensive class library that can be used from all .NET languages
– implemented as a set of DLLs (Managed Code)

Component concept

– not only object-oriented, but also component-oriented
– At the center of the component concept are the so-called Assemblies (EXE, DLL)
== a composite of one or more MSIL files, where at least one of the files is a DLL or EXE/ non-MSIL files, so called resource files (database, graphic or sound files)
DLL assembly:
– a reusable software component that can be used by another assembly
EXE assembly:
– can be started as an independent application
– but can provide services for others

.NET Core

== universal open source development platform
– replaces the old .NET Framework (mixture of new implementation and redesign/refactoring of .NET Framework 4.x)
– faster than .NET Framework
– modularly constructed
– allows you to create .NET Core Apps, cross-platform, for Windows, macOS and Linux for x64, x86, ARM32 and ARM64 processors
→ different programming languages are supported (C#, F#, or Visual Basic)
→ Frameworks and APIs for the cloud, for IoT, for client user interfaces and for machine learning
– Modern language constructs like generics, Language Integrated Query (LINQ), and asynchronous programming

Language Integrated Query (LINQ)

Query
== an expression that retrieves data from a data source
– are usually written in a specialized query language
– a new query language for each type of data source or data format they need to support
→ LINQ provides a consistent model for working with data in different types of data sources and formats
– Queries are always object based


LINQ query operations consist of three actions:
– Retrieving the data source
– Creating the query
– Executing the query

overview linq

Mono

== alternative, open source implementation of Microsoft’s .NET Framework
→ Development of a platform independent software based on the standards of the Common Language Infrastructure, ECMA, and the programming language C# shall be enabled

Additional functions for .Net

– Interfaces for operating system related functions under Unix
– Comprehensive Technology Coverage
– Binary Compatible
– Multi-Language(VB 8, Java, Python, Ruby, Eiffel, F#, Oxygene …)
– Changes can be made to already compiled code
– easy generation of native code

Components

– C# Compiler

– Mono Runtime

– .NET Framework Class Library

– Mono Class Library

.NET 5.0

→ In autumn 2020 all three platforms (.NET Core, .NET Framework, Mono) will be merged

overview net5