Álvaro Martín Lozano
Implementing continuous delivery in a data processing pipeline
#1about 4 minutes
From research concepts to production-ready data products
The Volkswagen Data Lab shifted its focus from demonstrating proof-of-concepts to building and deploying real-world data solutions for its clients.
#2about 7 minutes
Core concepts of continuous delivery for data
Continuous delivery for data pipelines requires adapting standard CI/CD principles, where data is the deliverable, by progressing through version control, integration, and deployment stages.
#3about 11 minutes
Implementing a pipeline with immutable, versioned data
The five-step pipeline relies on treating data as immutable, creating a new versioned output for each run to enable simple rollbacks and reproducibility.
#4about 6 minutes
The challenge of orchestrating chained data jobs
Managing dependencies between jobs becomes complex when each job consumes versioned, immutable data inputs from upstream processes.
#5about 5 minutes
Pros and cons of the immutable data approach
While this method offers powerful benefits like reproducibility and instant rollbacks, it introduces challenges in orchestration complexity and increased storage costs.
Related jobs
Jobs that call for the skills explored in this talk.
Matching moments
00:36 MIN
The distinct roles of CI and CD pipelines
#90DaysOfDevOps - The DevOps Learning Journey
06:04 MIN
Using continuous delivery to enable business agility
The Affordances of Quality
03:47 MIN
Applying software engineering practices to data pipelines
Enjoying SQL data pipelines with dbt
00:51 MIN
Defining continuous integration, delivery, and deployment
CI/CD with Github Actions
05:34 MIN
Tracing the evolution of DevOps from silos to superhighways
Navigating the AI Wave in DevOps
04:11 MIN
Adopting trunk-based development and continuous delivery
100 times more frequent deployments: How did we create a high performance team?
01:21 MIN
Testing against production with continuous deployment
How to Destroy a Monolith?
09:45 MIN
Implementing an AI-in-the-loop continuous learning cycle
A solution to embed container technologies into automotive environments
Featured Partners
Related Videos
30:46Charting the Journey to Continuous Deployment with a Value Stream Map
Josh Armitage
43:47Industrializing your Data Science capabilities
Dubravko Dolic & Hüdaverdi Cakir
29:36CI/CD Patterns and Antipatterns - Things your Pipeline Should (Not) Do
Daniel Raniz Raneland
39:04Python-Based Data Streaming Pipelines Within Minutes
Bobur Umurzokov
27:52Progressive Delivery in Kubernetes
Carlos Sanchez
50:28Practical tips and tricks for CI/CD success
Zan Markan
56:19Plan CI/CD on the Enterprise level!
Pawel Piwosz
32:29GitLab CI pipelines for a whole company
Martin Beránek
Related Articles
View all articles
From learning to earning
Jobs that call for the skills explored in this talk.
ADMIRAL Technologies
Remote
€55K
Linux
Apache Kafka
Microsoft SQL Server