Preparation for AI: From Raw Data to Reliable Models

After completing this course, participants will be able to:

• structure raw and curated data in an object-storage lakehouse using Bronze, Silver, and Gold layers.
• transform and query data at scale using SQL-centric workflows, producing training-ready datasets.
• learn practical patterns to store and organise multimedia training assets (images, audio, video).
• automate repeatable data preparation and model-training steps with a workflow orchestrator.
• validate data and model outputs through exploratory analysis, dashboards, and metrics.

This one-day workshop focuses on the practical data engineering work that makes AI projects succeed. We start from the foundations of a modern lakehouse: object storage, Parquet, and an open table format, mapped to the Bronze, Silver, and Gold layering model. You learn how to structure both structured data and ML-relevant assets such as images, audio, and video, by storing binaries in object storage and managing metadata, labels, and dataset manifests as queryable tables.

In the hands-on labs, you build and run scalable transformations using Spark with a SQL-first workflow and generate curated datasets ready for training. Next, we introduce automation with a workflow orchestrator, showing how data preparation steps are scheduled, retried, backfilled, and integrated with a first lightweight model training and evaluation step. After the guest session, we use business intelligence style exploration and dashboards to validate data quality and model outputs, and we close with reproducible ML handoff patterns, including metrics and artifacts stored in the data platform.

After lunch, a guest speaker from Renson presents how they organise data collection and processing for connected products in an Azure-based platform. The session covers their lakehouse approach with Bronze, Silver, and Gold layers, and how they separate hot and cold processing paths for time-series data and metadata. They discuss the key building blocks they use in practice for ingestion, scheduling, computation, storage, and analysis, followed by Q&A linking the real-world architecture to the concepts explored in the workshop labs.