Organised data archival and analysis for prediction and improving productivity is a challenge for many industries. The architecture referenced can be used for live data ingestion and breakdown into reports with machine-learning capabilities. For a traditional customer premises system when migrating into this massive scale data storage and processing this architecture requires some modifications which will also be explored as case studies.
ETL Edge though depicted with an AWS Snowball Edge device, this could even be swapped with IOT sensors and scanners polled in through a Raspberry PI which has the AWS IOT Core libraries or ulitizes the message broking MQTT to dynamically send telemetry data, to be ingested into the warehouse and to be processed by machine learning process in Amazon SageMaker. This is just a reference architecture which requires further polishing to adapt it to any real-world situations. We can explore some use cases.
Continue reading “Generic Reference Architecture for massive Data Lake”