Nathaniel Okenwa
Performant Architecture for a Fast Gen AI User Experience
#1about 2 minutes
Building a real-time translator inspired by sci-fi
The Babel fish from "Hitchhiker's Guide to the Galaxy" serves as the inspiration for a real-time audio translation project.
#2about 4 minutes
Analyzing the latency of a basic AI architecture
A demonstration of the initial 2019 architecture using GCloud reveals a significant latency of over ten seconds for a simple translation.
#3about 2 minutes
Reducing latency by upgrading the AI service stack
Switching to modern, specialized APIs like Deepgram and 11 Labs significantly cuts the total processing time from twelve to five seconds.
#4about 2 minutes
Implementing streaming to reduce response wait times
Adopting a streaming approach provides a major performance boost, but a naive implementation results in chaotic and low-quality audio output.
#5about 2 minutes
Using chunking to balance streaming speed and quality
Chunking data based on sentence punctuation controls the streaming waterfall, improving the quality of generated audio without sacrificing speed.
#6about 6 minutes
Eliminating network latency with local and edge models
Running a smaller, local AI model like Whisper on the edge eliminates cross-continental network latency and provides near-instantaneous results.
#7about 3 minutes
Using caching to serve pre-generated AI responses
Implementing caching, from simple request matching to semantic search with vector databases, avoids redundant generation and speeds up common queries.
#8about 2 minutes
Optimizing prompts and user experience for speed
Fine-tuning performance involves optimizing prompts to generate fewer tokens and improving perceived speed with clear loading states for the user.
#9about 2 minutes
Summary of key performance optimization techniques
A final recap covers the essential strategies for building fast Gen AI experiences, including streaming, edge computing, caching, and prompt optimization.
Related jobs
Jobs that call for the skills explored in this talk.
Matching moments
01:43 MIN
Key architectural challenges in building GenAI apps
Chatbots are going to destroy infrastructures and your cloud bills
02:24 MIN
Navigating the overwhelming wave of generative AI adoption
Developer Experience, Platform Engineering and AI powered Apps
00:54 MIN
Generative AI use cases and cloud provider limitations
Generative AI power on the web: making web apps smarter with WebGPU and WebNN
02:33 MIN
Panelists' wishes for future AI capabilities
The Future of Developer Experience with GenAI: Driving Engineering Excellence
02:09 MIN
The future of translation and human-AI collaboration
Fireside Chat: Deep Learning, Deep Impact: Harnessing AI for Language Innovation
02:20 MIN
The technology behind in-browser AI execution
Generative AI power on the web: making web apps smarter with WebGPU and WebNN
08:56 MIN
Exploring new frontiers in coding and computer interaction
WeAreDevelopers LIVE - Dapr / Pixels and Generative Art / Open Source and Communities / and more
03:59 MIN
Predicting the next era of generative AI
Closing Keynote by Joel Spolsky
Featured Partners
Related Videos
30:36Prompt API & WebNN: The AI Revolution Right in Your Browser
Christian Liebel
21:21Chatbots are going to destroy infrastructures and your cloud bills
Stanislas Girard
32:27Generative AI power on the web: making web apps smarter with WebGPU and WebNN
Christian Liebel
30:13Privacy-first in-browser Generative AI web apps: offline-ready, future-proof, standards-based
Maxim Salnikov
26:54Make it simple, using generative AI to accelerate learning
Duan Lightfoot
32:13Supercharge your cloud-native applications with Generative AI
Cedric Clyburn
25:14The shadows of reasoning – new design paradigms for a gen AI world
Jonas Andrulis
28:13Hello JARVIS - Building Voice Interfaces for Your LLMS
Nathaniel Okenwa
Related Articles
View all articles
From learning to earning
Jobs that call for the skills explored in this talk.
Conrad Electronic SE