Jan Curn

How to scrape modern websites to feed AI agents

What if your AI could find and use new tools on its own? See how dynamic tool discovery creates powerful agents that can scrape the modern web.

How to scrape modern websites to feed AI agents
#1about 1 minute

Why web data is essential for training large language models

LLMs are trained on massive web datasets like Common Crawl, but this leads to knowledge cutoffs and hallucinations.

#2about 2 minutes

How RAG provides LLMs with up-to-date context

Retrieval-Augmented Generation (RAG), or context engineering, feeds external, live data to LLMs to produce more accurate and timely answers.

#3about 3 minutes

Navigating the complexities of modern web scraping

Modern websites use dynamic JavaScript rendering and anti-bot measures, requiring headless browsers, proxies, and CAPTCHA solvers to access data.

#4about 2 minutes

Cleaning messy HTML and scaling data extraction

To avoid the 'garbage in, garbage out' problem, you must clean HTML by removing cookie banners and ads, and manage complexities like sitemaps and robots.txt.

#5about 3 minutes

Demo of scraping a website with Apify Actors

A demonstration shows how to use the Apify Website Content Crawler to perform a deep crawl of a website and extract its content into markdown.

#6about 2 minutes

Building a RAG chatbot with scraped data and Pinecone

The scraped website data is uploaded to a Pinecone vector database, enabling a chatbot to answer questions using the site's specific content.

#7about 1 minute

Using the Model Context Protocol for AI agent integration

The Model Context Protocol (MCP) provides a fluid, dynamic interface for AI agents to communicate with and discover tools, unlike static traditional APIs.

#8about 3 minutes

Demo of dynamic tool discovery using MCP

An AI agent uses MCP to dynamically search the Apify store for a Twitter scraper, add it to its context, and then use it to fetch live data.

Related jobs
Jobs that call for the skills explored in this talk.

job ad

Saby Company
Delebio, Italy

Intermediate

d

Saby Company
Delebio, Italy

Junior

Matching moments

Powering real-time AI with retrieval augmented generation
08:26 MIN

Powering real-time AI with retrieval augmented generation

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Presenting live web scraping demos at a developer conference
17:41 MIN

Presenting live web scraping demos at a developer conference

Tech with Tim at WeAreDevelopers World Congress 2024

The symbiotic relationship between AI and web scraping
25:34 MIN

The symbiotic relationship between AI and web scraping

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Demonstration of an AI copilot for automated scraping
22:52 MIN

Demonstration of an AI copilot for automated scraping

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Training AI models with custom scraped data
04:22 MIN

Training AI models with custom scraped data

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Automating browser workflows with AI-powered tools
38:12 MIN

Automating browser workflows with AI-powered tools

WeAreDevelopers LIVE: Scammer Payback with Python, Grok Goes Unhinged, The Future of Chromium and mo

Understanding the power of autonomous AI agents
14:39 MIN

Understanding the power of autonomous AI agents

HR ROBO SAPIENS: Decoding AI Agents and Workflow Automation for Modern Recruitment

The new AI engineer role and the RAG pipeline
02:38 MIN

The new AI engineer role and the RAG pipeline

Chatbots are going to destroy infrastructures and your cloud bills

Featured Partners

Related Videos

See all videos
Scrape, Train, Predict: The Lifecycle of Data for AI Applications26:15

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Vidas Bacevičius

MCP Mashups: How AI Agents are Reviving the Programmable Web23:17

MCP Mashups: How AI Agents are Reviving the Programmable Web

Angie Jones

Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP25:32

Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP

Viktoria Semaan

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation21:17

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation

Carl Lapierre

How AI Models Get Smarter35:16

How AI Models Get Smarter

Ankit Patel

Build RAG from Scratch28:04

Build RAG from Scratch

Phil Nash

From A2A to MCP: How AI’s “Brains” are Connecting to “Arms and Legs”24:12

From A2A to MCP: How AI’s “Brains” are Connecting to “Arms and Legs”

Brad Axen

Building Real-Time AI/ML Agents with Distributed Data using Apache Cassandra and Astra DB28:20

Building Real-Time AI/ML Agents with Distributed Data using Apache Cassandra and Astra DB

Dieter Flick

Related Articles

View all articles
EM
Eli McGarvie
13 AI Tools You Have to Try
First, it was NFTs, then it was Web3, and now it’s generative AI… it’s probably time to stop collecting pictures of monkeys and kitties. Chatbots and generative AI are the next big thing. This time we’ve jumped on a trend that has real-world applicat...
13 AI Tools You Have to Try
CH
Chris Heilmann
Dev Digest 116 - WWWAI?
This time, learn how to un-AI Google's search results, what's new on the web, avoid a new security hole and go back to BASICS with us. News and ArticlesWhat a week. Google, Microsoft, OpenAI and many others had their big flagship events announcing th...
Dev Digest 116 - WWWAI?
CH
Chris Heilmann
Dev Digest 137 - AI'm not sure about this
Hello fellow developer, this is the 1st "out of the can" edition of 3 as I am on vacation in Greece going "whee are you cute" at donkeys. So, fewer news, but lots of great resources. Enjoy! News and ArticlesOpenAI has been the big topic winning in th...
Dev Digest 137 - AI'm not sure about this

From learning to earning

Jobs that call for the skills explored in this talk.