EXPERT KNOWLEDGE AT A GLANCE

Tag: microsoft

Apache Hive Architecture – Data Warehouse System for free

Apache Hive Architecture – On the way to Industry 4.0, companies are trying to record all business processes as far as possible in order to subsequently optimize them through analysis.
Data warehouse systems provide central data management. Thus, only one data truth exists. In addition to persistence, these information systems take care of sorting, preprocessing, translation and data analysis.
If you want to know more about what a data warehouse system is, check out our article on the subject.

What is Apache Hive

Hive is a data warehousing software project and part of Apache, an open source and free web server software. Learn more about Apache here.
It is built on the Big Data framework Apache Hadoop and was released in 2010. Since then it has been continuously improved and extended by an industrious community.

hive
Apache Hive Architecture – Built on top of Hadoop

The query language used by Hive, called HiveQL, is SQL based and allows querying, aggregation and analysis of unstructured data. Hive does not work with the schema-on-write (SoW) approach like relational databases, but uses the so-called schema-on-read (SoR) approach.

What are the biggest advantages of Hive?

Data from relational databases is automatically converted into MapReduce or Tez or Spark jobs. Hadoopclusters are based on MapReduce, a Google programming model for concurrent computation on computer clusters, and powerful stream-based data analysis pipelines can be created with Apache Spark. This ensures full compatibility with the Apache ecosystem, which can be modularly tailored to the needs of an application.

The figure shows the main Apache Hive features
Apache Hive Features

Another advantage of Hive is that the tables are similar to the tables in a relational database. Data is queried using HiveQL. A declarative SQL-like language.
HiveQL allows multiple users to query data simultaneously. Hive supports a variety of data formats and provides a lightweight but powerful translation feature.
For data analysis, custom MapReduce processes can be written and run on clusters in parallel for high performance.

Apache Hive Architecture

Basically, the architecture of Hive can be divided into three core areas. Hive communicates with other applications via the client area. The integration is then executed via the service area. In the last layer, Hive stores the metadata, for example, or computes the data via Hadoop.

The figure shows the basic three-part core architecture of Apache Hive.
Apache Hive Architecture

Hive Clients

Apache Hive can be accessed via different clients. In addition to Open Database Connectivity (ODBC), an SQL-based application programming interface (API) created by Microsoft, there is Java Database Connectivity (JDBC), an SQL-based API developed by Sun Microsystems to allow Java applications to use SQL for database access. Hive also provides a high-performance Apache Thrift connection.

Hive Services

The core and central control of the Hive Services is the so-called driver. This
receives HiveQL commands and is responsible for their execution against the Hadoop system. It typically consists of a compiler that translates HiveQL requests into abstract syntax and executable tasks, an optimizer that aggregates, splits, and optimizes for better performance and scalability, and an executor that interacts with Hadoop’s job tracker and passes tasks to the system for execution.

Apache Hive also provides the ability to submit these tasks directly to the driver. Using the Command Line and User Interface (CLI + UI), it is possible to directly influence the process.

Metadata about persistent relational entities, i.e. databases, tables, columns and partitions are managed by the metastore.

Hive Storage and Computer

The metadata is stored here in a persistence. The results of the query and the data loaded into the tables are stored on HDFS in the Hadoop cluster.

Microsoft Power Platform – Microsoft’s Web tool to automate your workflows

In this article, we will show you everything about the cloud-based web tool Microsoft Power Platform and why you shouldn’t do without it.

Cloud-based web development has gained popularity in the web development industry in recent years. The globalization of the workforce and the diversification of the work process have significantly driven the development of cloud-based services.

What is Microsoft Power Platform?

With Power Platform you get an integrated application platform consisting of a group of different Microsoft products with which you can develop complex business solutions. This way you can make your business processes more efficient and productive. The platform can also take care of data storage, entry and processing. Data analysis via complex visualizations and predictions can also be handled by various services.

Microsoft Power Platform Services

Schematic representation of the Microsoft Power Platform Services
Power Platform Services

Since the Power Platform is a collection of different Microsoft services, we want to give you an overview of the individual parts.

Build your own Apps

With Power Apps you get a user suite for mapping custom apps. All apps are independent of data sources and can be extended by you as you wish via drag & drop. This way you can adapt them to your needs.

Automate your tasks

Power Automate was still called Microsoft Flow until 2019. This web tool lets you automate recurring tasks and simple cross-platform workflows. You can connect to over 100 third-party systems via connectors. This allows you to automate processes outside the Microsoft environment and across applications.

Schematic representation of the Principle Microsoft Power Automate
Principle Microsoft Power Automate

Analyze your business data

With Power BI you get a business intelligence tool. With this you can access different data sources. An advantage to other BI tools is the deep integration with Excel. So you can create user-friendly, data connections and visualize.

Should I choose Microsoft Power Platform?

You need more and more scalable and secure solutions at low cost. Device-independent access to important planning and evaluation software will increasingly become the focus of attention and change global corporate structures. To turn your company into an Industry 4.0 enterprise, you can no longer avoid cloud solutions.

The question is not: if, but when you will choose a corresponding service. The question is not: if, but when you will choose a corresponding service. Microsoft, as one of the largest IT companies, now presents you with a solution. Whether it fits your needs, however, you must ultimately decide for yourself.

What is Microsoft’s .NET?

.NET

== collective term for several software platforms published by Microsoft (2000)
– for development and execution of application programs

Main components:

.NET Framework

.NET Core

Mono from Xamarin

overview dotNET

.NET Framework

is a run-time execution environment that manages apps that target

overview net frameworks

Tools

– Command line compiler (for Visual Basic.NET, C#, Jscript.NET)
– Software Development Kit (SDK) with tools, documentation and examples

Common Language Runtime (CLR)

– Runtime environment for the execution of a .NET application
– provides the just-in-time compiler
– provides numerous other basic services (garbage collector, exception handling, a security system and interoperability with non.NET applications)
– When a .NET application is started, Windows does not call the CLR itself directly, but first calls a so-called Runtime Host. This host loads the CLR and passes the entry point for the application to the CLR.

Runtime Hosts:
– Shell Runtime Host
– Internet Explorer Runtime Host
– ASP.NET

Language Integration

– enables the cooperation of different languages
→ Calling a program code written in another programming language
→ object-oriented languages can inherit classes written in another object-oriented language

Uniform class library


– NET Framework Class Library (FCL)
– extensive class library that can be used from all .NET languages
– implemented as a set of DLLs (Managed Code)

Component concept

– not only object-oriented, but also component-oriented
– At the center of the component concept are the so-called Assemblies (EXE, DLL)
== a composite of one or more MSIL files, where at least one of the files is a DLL or EXE/ non-MSIL files, so called resource files (database, graphic or sound files)
DLL assembly:
– a reusable software component that can be used by another assembly
EXE assembly:
– can be started as an independent application
– but can provide services for others

.NET Core

== universal open source development platform
– replaces the old .NET Framework (mixture of new implementation and redesign/refactoring of .NET Framework 4.x)
– faster than .NET Framework
– modularly constructed
– allows you to create .NET Core Apps, cross-platform, for Windows, macOS and Linux for x64, x86, ARM32 and ARM64 processors
→ different programming languages are supported (C#, F#, or Visual Basic)
→ Frameworks and APIs for the cloud, for IoT, for client user interfaces and for machine learning
– Modern language constructs like generics, Language Integrated Query (LINQ), and asynchronous programming

Language Integrated Query (LINQ)

Query
== an expression that retrieves data from a data source
– are usually written in a specialized query language
– a new query language for each type of data source or data format they need to support
→ LINQ provides a consistent model for working with data in different types of data sources and formats
– Queries are always object based


LINQ query operations consist of three actions:
– Retrieving the data source
– Creating the query
– Executing the query

overview linq

Mono

== alternative, open source implementation of Microsoft’s .NET Framework
→ Development of a platform independent software based on the standards of the Common Language Infrastructure, ECMA, and the programming language C# shall be enabled

Additional functions for .Net

– Interfaces for operating system related functions under Unix
– Comprehensive Technology Coverage
– Binary Compatible
– Multi-Language(VB 8, Java, Python, Ruby, Eiffel, F#, Oxygene …)
– Changes can be made to already compiled code
– easy generation of native code

Components

– C# Compiler

– Mono Runtime

– .NET Framework Class Library

– Mono Class Library

.NET 5.0

→ In autumn 2020 all three platforms (.NET Core, .NET Framework, Mono) will be merged

overview net5