George Asafev

Swapping Low Latency Data Storage Under High Load

Miro's Redis instance couldn't scale for 700 engineers. Learn how they moved 140 million accounts to DynamoDB while handling 100,000 requests per second.

Swapping Low Latency Data Storage Under High Load
#1about 4 minutes

Migrating 140M users from a Redis-based monolith

Miro's user registry team faced migrating 140 million users from Redis, which was unsuitable as a persistent store and was tied to a difficult-to-maintain monolith.

#2about 3 minutes

Evaluating PostgreSQL vs DynamoDB for the new database

The team compared PostgreSQL's flexibility with DynamoDB's managed scaling, ultimately choosing DynamoDB paired with DAX for its performance on read-heavy workloads.

#3about 2 minutes

Load testing the database solutions under production load

To validate their choice, they load-tested PostgreSQL using pgbench and DynamoDB with a k6 and EC2 setup to simulate production traffic.

#4about 2 minutes

Understanding the limitations of DynamoDB and DAX

Adopting DynamoDB with DAX required navigating limitations like primary key access patterns, separate item and query caches, and difficult vertical scaling.

#5about 4 minutes

The phased migration strategy from Redis to DynamoDB

The migration was executed in controlled phases using feature flags, including dual writes, a background data copy, shadow calls for validation, and a gradual rollout.

#6about 2 minutes

Unexpected DAX performance issues and learnings

After going live, the team discovered that rapid connection spikes caused DAX throttling and that adding a fourth node to their cluster unexpectedly increased replication lag.

#7about 4 minutes

Results of the migration and Q&A

The successful migration reduced endpoint latency up to four times and enabled the team to move forward with their microservice architecture, followed by a Q&A on alternative solutions.

Related jobs
Jobs that call for the skills explored in this talk.

d

Saby Company
Delebio, Italy

Junior

test

Milly
Vienna, Austria

Intermediate

Matching moments

Achieving massive scale on AWS for Prime Day
04:17 MIN

Achieving massive scale on AWS for Prime Day

Reliable scalability: How Amazon.com scales on AWS

Answering questions on data volume, challenges, and databases
28:34 MIN

Answering questions on data volume, challenges, and databases

Remote Driving on Plant Grounds with State-of-the-Art Cloud Technologies

Q&A on migration strategy and stakeholder management
26:29 MIN

Q&A on migration strategy and stakeholder management

AWS Migration within 3 months

Building a dynamic LRU cache with DynamoDB and CDC
12:04 MIN

Building a dynamic LRU cache with DynamoDB and CDC

Advanced Caching Patterns used by 2000 microservices

Merging data engineering and DevOps for scalability
02:48 MIN

Merging data engineering and DevOps for scalability

Software Engineering Social Connection: Yubo’s lean approach to scaling an 80M-user infrastructure

Supporting hyperscale workloads with a single database
18:32 MIN

Supporting hyperscale workloads with a single database

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Live demo and tracing errors to DynamoDB
31:23 MIN

Live demo and tracing errors to DynamoDB

Building your way to a serverless powered IOT Buzzwire game

Key learnings and results from the MLOps transformation
22:47 MIN

Key learnings and results from the MLOps transformation

The Road to MLOps: How Verivox Transitioned to AWS

Featured Partners

Related Videos

See all videos
Database Magic behind 40 Million operations/s24:22

Database Magic behind 40 Million operations/s

Jürgen Pilz

Scaling: from 0 to 20 million users28:12

Scaling: from 0 to 20 million users

Josip Stuhli

Single Server, Global Reach: Running a Worldwide Marketplace on Bare Metal in a Cloud-Dominated World50:06

Single Server, Global Reach: Running a Worldwide Marketplace on Bare Metal in a Cloud-Dominated World

Jens Happe

In-Memory Computing - The Big Picture30:36

In-Memory Computing - The Big Picture

Markus Kett

Making Data Warehouses fast. A developer's story.42:16

Making Data Warehouses fast. A developer's story.

Adnan Rahic

Reliable scalability: How Amazon.com scales on AWS22:24

Reliable scalability: How Amazon.com scales on AWS

Florian Mair

The Road to MLOps: How Verivox Transitioned to AWS25:48

The Road to MLOps: How Verivox Transitioned to AWS

Elisabeth Günther

AWS Migration within 3 months37:08

AWS Migration within 3 months

Steffen Heilmann

Related Articles

View all articles
BB
Benedikt Bischof
Making Data Warehouses Fast: A Developer’s Story
Welcome to this issue of the WeAreDevelopers Live Talk series. This article recaps an interesting talk by Adnan Rahic who teaches the audience how to make data warehouses.About the Speaker: Adnan is senior developers advocate at Cube. His passion lie...
Making Data Warehouses Fast: A Developer’s Story
CH
Chris Heilmann
Dev Digest 134 - Where pixels sing?
News and ArticlesWeAreDevelopers LIVE Data and Security Day is on Wednesday, 25/09/2024. Learn about OPC UA Updates, Best Practices for Using GitHub Secrets, Passwordless Web 1.5, Emerging AI Security Risks, Data Privacy in LLMs and get a chance to t...
Dev Digest 134 - Where pixels sing?
CH
Chris Heilmann
All the videos of Halfstack London 2024!
Last month was Halfstack London, a conference about the web, JavaScript and half a dozen other things. We were there to deliver a talk, but also to record all the sessions and we're happy to share them with you. It took a bit as we had to wait for th...
All the videos of Halfstack London 2024!

From learning to earning

Jobs that call for the skills explored in this talk.