Challenge

The Federal Employment Agency is one of the central public institutions of the Federal Republic of Germany. It is therefore their responsibility to the citizens and the ministries to collect, analyze, and manage their data according to the highest possible standards. There is an important distinction between user data and monitoring data from business-critical information systems, which have similar requirements for high availability and durability (i.e., permanent storage). In order to monitor queries and perform analyses within the required timeframe while maintaining productive operation, the task of monitoring data should be separated from the actual user data. Unfortunately, this could not be achieved with the relational databases in place. In addition, the constantly growing volume of data records required scalability that the existing solutions could not provide.

As a result, the IT system department needed a new solution for real-time monitoring of applications and business processes, and to be able to quickly counteract any negative influences. Further, the department wanted to use extensive analysis functions and the resulting information to enable them to optimize applications and the associated processes before certain bottlenecks occur.

Improved performance was also on the wish list. In contrast to their relational databases based on Oracle and Microsoft architectures, which had difficulty processing a high influx of data, prompt monitoring with high ingest rates was needed in order to react quickly to potential incidents and generally make software development and operations more agile for the more than 2,000 developers and almost 1,000 infrastructure managers.

The challenge for the IT system department was to build an architecture stack that could process exponentially growing digital information and requests in a timely manner and with a degree of scalability that guaranteed future security, as well as a solution for real-time monitoring that could meet the high requirements for durability, high availability, and analytical performance.

Solution

Open source Apache Cassandra was selected as the core of the new solution because of its ability to be tailored to BA’s needs and its flexibility in development. Additional focus was on the further processing of data, which is where DataStax Enterprise fit the bill. DSE is based on Cassandra, but also easily integrates other components. Alongside the advantages of Cassandra, users have access to other tools such as DSE Search (based on the open source solution Solr) and Grafana, used for the visualization of data from a single source. DataStax also offers comprehensive consulting as well as remote or on-site service support—during and after project completion.

The always-on and parallel insert functionalities bypass the limitations of relational databases. The DataStax stack is designed for the insert and evaluation of large volumes of long-term stored data and thus simplifies further processing for the BA, even during many simultaneous analysis processes. The new solution shoulders the workloads that arise at the Federal Employment Agency during monitoring in real time. For the IT system department, this means that they can now react in real time thanks to direct monitoring, instead of recording and analyzing incidents after the fact.

This also promotes a proactive strategy that identifies potential sources of problems at an early stage and allows developers to take preventive countermeasures. A classic example is increasing latencies in the network to the employment agencies. With the new solution, one can quickly see whether only individual agencies are affected and if the problem is, for example, in a dedicated network line or whether the increase is due to an error in the source code of a particular application. In this case, the real-time monitoring allows the application to be identified and corrected before the resulting delays affect users. “Thanks to the DataStax solution, the delay in our infrastructure monitoring has shrunk to a low level of seconds. This is an enormous help for the entire IT team—from developers to operations,” says Matthias Sessler, Lead Architect for Data Driven Services at the Federal Employment Agency. “DSE combines all the components we need in one solution and offers the scalability we need to be prepared for the future. That was one of the decisive factors for us.”

In addition, it was important that the new solution could be integrated into the existing, system-wide processes of the authority. BA therefore decided to include DSE in its technology portfolio as a standard database alongside Oracle DB and Microsoft SQL Server. Over a period of almost 18 months, the solution was implemented in cooperation with DataStax partners Fujitsu and Capgemini SE.

Results

The new architecture stack includes all the tools that are important to the IT system department at BA. In addition, the individual components in the DataStax solution are already coordinated with each other, and users benefit immediately without having to worry about adjustments. Moreover, the stack fits seamlessly into the existing systems and networks of BA and noticeably facilitates the work of several hundred employees of the IT system department. DSE enables BA to monitor large volumes of data more efficiently and quickly in a complex IT infrastructure of several manufacturers. Information from the applications is associated with information from the infrastructure as well as business processes, which allows the most diverse findings to be derived, for example, where a problem has its origin. Such conclusions, which were previously only visible after a very long period of time, can now be identified immediately through real-time monitoring.

In addition, DSE Search and Grafana make it easier than ever to categorize, search, and display the information. The user can intuitively visualize complex relationships or time series with Grafana, and comprehensive filtering options can also provide important insights for strategic planning. In this way, the latency times or access rates of applications can be compared in different periods of the week, and as a result, well-founded decisions can be made for the allocation of resources or employees.

Thanks to intensive cooperation and communication—remote and on-site—between the Federal Employment Agency, DataStax, and its partners, the solution was integrated into the authority’s network without major difficulties. Now it is used by over 3,000 IT employees. “We had been in contact with DataStax for quite some time before the project started, and during the implementation we were always pleased with the well-founded support and maximum commitment of DataStax and its partners. This also made it easy to master the organizational challenges of a large authority,” summarized Matthias Sessler.