Mario Valderrama

Operating etcd for Managed Kubernetes

To achieve zero-downtime migrations, we modified etcd snapshots to artificially inflate the revision number. Discover the surprising challenges of operating etcd at massive scale.

Operating etcd for Managed Kubernetes
#1about 3 minutes

The journey to managed Kubernetes at IONOS

From its first release in 2019 to managing over 20,000 clusters, IONOS scaled its Kubernetes service by building on a massive etcd foundation.

#2about 4 minutes

Evolving etcd deployment strategies over time

The team progressed from the CoreOS operator and Bitnami Helm charts to a simplified custom Helm chart for better control and stability.

#3about 3 minutes

Understanding multi-tenancy and its performance impact

Using a shared etcd with client-side prefixes reduces cost but creates noisy neighbor problems, requiring careful tuning like compaction and defragmentation.

#4about 3 minutes

Iterating on etcd cluster layouts for reliability

Initial cross-location clusters suffered from latency and revision drift, leading to a more stable single data center layout using availability zones.

#5about 3 minutes

A zero-downtime control plane migration strategy

A live migration process using `etcdctl mirror` allows moving a Kubernetes control plane to a new etcd cluster without global downtime or data loss.

#6about 3 minutes

Manipulating etcd revisions for seamless migration

By modifying an etcd snapshot to insert a high revision number, clients like kubelet continue watching for changes without needing a restart after migration.

#7about 2 minutes

Future plans for etcd management and automation

The team is working on automating the migration process, offering dedicated etcd clusters, and contributing their migration learnings to the Kaji project.

Related jobs
Jobs that call for the skills explored in this talk.

test

Milly
Vienna, Austria

Intermediate

test

Milly
Vienna, Austria

Intermediate

job ad

Saby Company
Delebio, Italy

Intermediate

Matching moments

Understanding the challenges of scaling Kubernetes with confidence
00:25 MIN

Understanding the challenges of scaling Kubernetes with confidence

5 steps for running a Kubernetes environment at scale

Managing containers at scale with Kubernetes
26:00 MIN

Managing containers at scale with Kubernetes

#90DaysOfDevOps - The DevOps Learning Journey

Managing a complex mix of old and new infrastructure
03:02 MIN

Managing a complex mix of old and new infrastructure

Hosting a modern justice system

Addressing unique data protection challenges in Kubernetes
07:14 MIN

Addressing unique data protection challenges in Kubernetes

It's all about the Data

Building end-to-end AI solutions in European data centers
11:33 MIN

Building end-to-end AI solutions in European data centers

From foundation model to hosted AI solution in minutes

Why teams move from monoliths to Kubernetes
03:52 MIN

Why teams move from monoliths to Kubernetes

Get ready for operations by pull requests

Using Kubernetes as an extensible control plane
07:26 MIN

Using Kubernetes as an extensible control plane

Chaos in Containers - Unleashing Resilience

Comparing managed DBaaS with databases on Kubernetes
37:18 MIN

Comparing managed DBaaS with databases on Kubernetes

Databases on Kubernetes: Why you should care

Featured Partners

Related Videos

See all videos
Fifty Shades of Kubernetes Autoscaling30:26

Fifty Shades of Kubernetes Autoscaling

Mario-Leander Reimer

Chaos in Containers - Unleashing Resilience27:52

Chaos in Containers - Unleashing Resilience

Maish Saidel-Keesing

Kubernetes Maestro: Dive Deep into Custom Resources to Unleash Next-Level Orchestration Power!29:21

Kubernetes Maestro: Dive Deep into Custom Resources to Unleash Next-Level Orchestration Power!

Um e Habiba

From Factory Floor to Kubernetes Core: Building an Edge Platform One Step at a Time17:34

From Factory Floor to Kubernetes Core: Building an Edge Platform One Step at a Time

Dean Oren & Stefan Belsch

Mastering Kubernetes – Beginner Edition57:24

Mastering Kubernetes – Beginner Edition

Hannes Norbert Göring

Kubernetes Security Best Practices23:08

Kubernetes Security Best Practices

Rico Komenda

Winning the Hybrid Cloud26:19

Winning the Hybrid Cloud

Alex Soto

Scaling: from 0 to 20 million users28:12

Scaling: from 0 to 20 million users

Josip Stuhli

Related Articles

View all articles
Learning Kubernetes made easy with KubeCampus
Learning to use Kubernetes? KubeCampus by Kasten offers free educational content for all skill levels to get you started!Kubernetes is an open-source system for deploying, scaling and managing containerized applications. It allows you to deploy your ...
Learning Kubernetes made easy with KubeCampus
BB
Benedikt Bischof
MLOps – What’s the deal behind it?
Welcome to this issue of the WeAreDevelopers Live Talk series. This article recaps an interesting talk by Nico Axtmann who introduced us to MLOpsAbout the speaker:Nico Axtmann is a seasoned machine learning veteran. Starting back in 2014 he observed ...
MLOps – What’s the deal behind it?
CH
Chris Heilmann
All the videos of Halfstack London 2024!
Last month was Halfstack London, a conference about the web, JavaScript and half a dozen other things. We were there to deliver a talk, but also to record all the sessions and we're happy to share them with you. It took a bit as we had to wait for th...
All the videos of Halfstack London 2024!

From learning to earning

Jobs that call for the skills explored in this talk.