AWS
BIG Industries has developed a modern AWS cloud platform that supports the monitoring of the Covid-19 pandemic in Flanders. With the sudden outbreak the Flemish Government's Agency for Care and Health (AZG - Agentschap Zorg en Gezondheid) needed a monitoring system fast. Thomas De Block, Data Engineer at Big Industries, assisted with the development of that platform. He tells us how data from hospitals, vaccination centres and contact tracing provides a picture of where and when infections are being transmitted. The new platform also makes it possible to investigate and communicate clusters of infections.
AZG intervenes in epidemics
Het Vlaams Agentschap Zorg & Gezondheid (AZG) is responsible for the recognition, licensing and subsidizing of various providers and services for health care, elderly care, home care, residential care, and other forms of care, for facilitating the cooperation and data sharing between these services, for the organization of preventive health services such as vaccinations and screening programs, for the control of infectious diseases, for environmental health monitoring, and for the organization of the Flemish social protection programs.
The AZG was critical in the fight against the transmission of Covid-19 infections – collecting data from hospitals, care institutions, vaccination centres and other available sources that provide details about infections and deceased. It shares this information with federal and local governments to ensure that appropriate actions can be taken.
Need for a new modern Cloud Data Lake
Thomas De Block explains: the predecessor to the new AWS platform was used for storing and updating documents and lists for monitoring the status of various infectious diseases. But a pandemic like Covid19 is a completely different story. There was so much data to process that this legacy system was no longer up to the job.'
"The first step was to re-design and build new data pipelines. We already had a cloud infrastructure available, but we needed to implement a new solution fast while continuing to develop a system that could cope with the higher data loads. In the original system, all the data had to be processed manually in daily lists, so we started by replacing this step with an Apache NiFi ingestion tool to feed the data into a data warehouse based on Snowflake.
Big Data approach
"The amounts of data that suddenly needed processing demanded a modern Big Data approach,’ explains Thomas. ‘For the new system, we worked with AWS Cloud with managed services, AWS S3 data lake and Apache Kafka as an event broker. The technologies that were chosen are the most up-to-date solutions available. The AZG is clearly committed to ensuring security and speed.’
Tracing outbreak clusters
'The new platform goes beyond simply increasing the speed of data processing' continues Thomas. 'The new system doesn't just monitor outbreaks more efficiently compared to the legacy application; it also highlights possible outbreak clusters. We use this information to alert the Infectious Diseases Team who are in contact with the local authorities'
Thomas explains the additional requirements that the platform must meet; 'the customer wanted real-time data input to track potential outbreak clusters.' This information needs to be made available immediately through a web application to enable authorised users to monitor the outbreak clusters and share details with external parties. 'The authorities also need to receive notifications about new events, so we introduced a message broker and a streaming platform based on a microservices architecture.'
Act fast on insights
source: Sciensano
''The new platform is an extremely useful tool for the Infectious Diseases Team at the AZG. They share information with external infectious diseases experts, the Sciensano research institute and relevant authorities in cities and other municipalities,' explains Thomas. 'These authorities feed information back to doctors and inform organisations where there have been cluster of outbreaks, for example.'
'The monitoring platform facilitates collaboration, follows up contact tracing and tracks clusters to help relevant authorities make better decisions faster. Speed is essential in times of crisis.'
Managing a crisis
Thomas explains the project's unique context: 'We have implemented major changes with a relatively small team. We have a couple of people working on BI, a Data warehouse team and an AWS platform team working all together towards a common goal. The societal importance of this project meant we had to work towards very strict deadlines with lots of sudden and unexpected changes. New measures and recommendations always result in modifications to the platform or in the back end, which is why we are still releasing updated versions every week.'
'The project was very interesting from a personal perspective,' adds Thomas. 'As an IT professional, being able to collaborate on something that can help people so directly has helped me to grow on both a personal and professional level. Learning to handle this responsibility has definitely contributed to my development.'
Source Overview Image: El Pais
