Sandra Ahlgrimm & Kevin Lewis
Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models
#1about 3 minutes
The challenge of testing non-deterministic AI features
Traditional development relies on rigorous testing, but AI features are often implemented based on intuition without a structured evaluation process.
#2about 5 minutes
Managing prompts as code with GitHub Models
GitHub Models integrates AI development into your repository by defining prompts, models, and parameters in a version-controlled YAML file.
#3about 6 minutes
Using evaluators to compare AI model variants
The platform allows you to run multiple prompt and model variations against a test dataset to compare outputs on metrics like latency, coherence, and similarity.
#4about 5 minutes
Consuming prompt files in your application code
Use the GitHub Models inference API or the Azure AI Inference SDK to load your version-controlled prompt files and integrate AI calls directly into your application.
#5about 2 minutes
Local development and testing with the CLI
The GitHub CLI extension allows you to run prompts and execute model evaluations directly from your terminal for rapid, local iteration before committing changes.
#6about 4 minutes
Automating repository tasks with AI-powered actions
Use GitHub Actions to automate common repository tasks like generating changelogs from pull requests, triaging bug reports, or creating weekly issue summaries.
#7about 1 minute
Implementing CI/CD for AI prompt changes
Integrate prompt evaluations into your CI/CD pipeline using GitHub Actions to automatically run tests and block pull requests that degrade model performance.
#8about 2 minutes
Adopting GitHub Models in existing projects
You can quickly convert existing prompt files to the GitHub Models format to gain access to powerful evaluation, comparison, and automation capabilities.
Related jobs
Jobs that call for the skills explored in this talk.
Matching moments
08:29 MIN
Experimenting with prompts and models in GitHub
Agentic AI - From Theory to Practice: Developing Multi-Agent AI Systems on Azure
09:55 MIN
Shifting from traditional code to AI-powered logic
WWC24 - Ankit Patel - Unlocking the Future Breakthrough Application Performance and Capabilities with NVIDIA
20:21 MIN
A practical workflow for AI application developers
How AI Models Get Smarter
20:05 MIN
The limitations and potential of AI models
Coffee with Developers - Cassidy Williams -
16:56 MIN
How AI agents automate development with GitHub Copilot
Agentic DevOps: How AI-Powered Automation Transforms Software Delivery on GitHub and Azure
20:07 MIN
The current era of AI-assisted development
From Punch Cards to AI-assisted Development
14:00 MIN
The impact of AI coding assistants on developer productivity
Fireside Chat with Sir Tim Berners-Lee
00:07 MIN
Investigating the future of software development at GitHub Next
Innovating Developer Tools with AI: Insights from GitHub Next
Featured Partners
Related Videos
25:17AI: Superhero or Supervillain? How and Why with Scott Hanselman
Scott Hanselman
31:39Livecoding with AI
Rainer Stropek
28:53Agentic DevOps: How AI-Powered Automation Transforms Software Delivery on GitHub and Azure
Mike
24:11You are not my model anymore - understanding LLM model behavior
Andreas Erben
33:28Prompt Engineering - an Art, a Science, or your next Job Title?
Maxim Salnikov
32:26Bringing the power of AI to your application.
Krzysztof Cieślak
42:38Innovating Developer Tools with AI: Insights from GitHub Next
Krzystof Czieslak
35:16How AI Models Get Smarter
Ankit Patel
Related Articles
View all articles
From learning to earning
Jobs that call for the skills explored in this talk.
Front End Engineering Manager ( Generative AI experience )
Accenture
GraphQL
React Native
Continuous Integration