Álvaro Martín Lozano

Implementing continuous delivery in a data processing pipeline

What if you could roll back a data pipeline instantly, with zero state migration? See how treating data as an immutable, versioned artifact makes this a reality.

Implementing continuous delivery in a data processing pipeline
#1about 4 minutes

From research concepts to production-ready data products

The Volkswagen Data Lab shifted its focus from demonstrating proof-of-concepts to building and deploying real-world data solutions for its clients.

#2about 7 minutes

Core concepts of continuous delivery for data

Continuous delivery for data pipelines requires adapting standard CI/CD principles, where data is the deliverable, by progressing through version control, integration, and deployment stages.

#3about 11 minutes

Implementing a pipeline with immutable, versioned data

The five-step pipeline relies on treating data as immutable, creating a new versioned output for each run to enable simple rollbacks and reproducibility.

#4about 6 minutes

The challenge of orchestrating chained data jobs

Managing dependencies between jobs becomes complex when each job consumes versioned, immutable data inputs from upstream processes.

#5about 5 minutes

Pros and cons of the immutable data approach

While this method offers powerful benefits like reproducibility and instant rollbacks, it introduces challenges in orchestration complexity and increased storage costs.

Related jobs
Jobs that call for the skills explored in this talk.

test

Milly
Vienna, Austria

Intermediate

test

Milly
Vienna, Austria

Intermediate

Matching moments

The distinct roles of CI and CD pipelines
28:19 MIN

The distinct roles of CI and CD pipelines

#90DaysOfDevOps - The DevOps Learning Journey

Using continuous delivery to enable business agility
12:28 MIN

Using continuous delivery to enable business agility

The Affordances of Quality

Applying software engineering practices to data pipelines
14:52 MIN

Applying software engineering practices to data pipelines

Enjoying SQL data pipelines with dbt

Defining continuous integration, delivery, and deployment
02:53 MIN

Defining continuous integration, delivery, and deployment

CI/CD with Github Actions

Tracing the evolution of DevOps from silos to superhighways
01:16 MIN

Tracing the evolution of DevOps from silos to superhighways

Navigating the AI Wave in DevOps

Adopting trunk-based development and continuous delivery
11:32 MIN

Adopting trunk-based development and continuous delivery

100 times more frequent deployments: How did we create a high performance team?

Testing against production with continuous deployment
23:19 MIN

Testing against production with continuous deployment

How to Destroy a Monolith?

Implementing an AI-in-the-loop continuous learning cycle
18:08 MIN

Implementing an AI-in-the-loop continuous learning cycle

A solution to embed container technologies into automotive environments

Featured Partners

Related Videos

See all videos
Python-Based Data Streaming Pipelines Within Minutes39:04

Python-Based Data Streaming Pipelines Within Minutes

Bobur Umurzokov

Charting the Journey to Continuous Deployment with a Value Stream Map30:46

Charting the Journey to Continuous Deployment with a Value Stream Map

Josh Armitage

Enabling automated 1-click customer deployments with built-in quality and security44:40

Enabling automated 1-click customer deployments with built-in quality and security

Christoph Ruggenthaler

CI/CD Patterns and Antipatterns - Things your Pipeline Should (Not) Do29:36

CI/CD Patterns and Antipatterns - Things your Pipeline Should (Not) Do

Daniel Raniz Raneland

Industrializing your Data Science capabilities43:47

Industrializing your Data Science capabilities

Dubravko Dolic & Hüdaverdi Cakir

Progressive Delivery in Kubernetes27:52

Progressive Delivery in Kubernetes

Carlos Sanchez

Practical tips and tricks for CI/CD success50:28

Practical tips and tricks for CI/CD success

Zan Markan

Plan CI/CD on the Enterprise level!56:19

Plan CI/CD on the Enterprise level!

Pawel Piwosz

Related Articles

View all articles
CH
Chris Heilmann
Dev Digest 134 - Where pixels sing?
News and ArticlesWeAreDevelopers LIVE Data and Security Day is on Wednesday, 25/09/2024. Learn about OPC UA Updates, Best Practices for Using GitHub Secrets, Passwordless Web 1.5, Emerging AI Security Risks, Data Privacy in LLMs and get a chance to t...
Dev Digest 134 - Where pixels sing?
DD
Dilek Demir
Data Science & more: The Lopez dilemma
Catwalk, Data Science, Hollywood, Google Images, Haute Couture, StackOverflow, Comfort Zone, Dota 2 and Versace – all these topics are connected and influenced by each other. Read here how and why!In 2000 Jennifer Lopez's green Versace dress went vi...
Data Science & more: The Lopez dilemma

From learning to earning

Jobs that call for the skills explored in this talk.