🕷️ Lesson 5.3: Building a Web Scraper Agent
🎯 Lesson Objective
By the end of this lesson, you’ll be able to:
-
Build an AI agent that can extract data from websites
-
Integrate a scraper with LangChain or your agent framework
-
Use scraped data in conversations, summaries, analysis, or automation
-
Apply ethical and legal best practices for scraping
🧠 Why Build a Web Scraper Agent?
Sometimes the data your AI needs:
-
❌ Isn’t available via API
-
✅ Is visible on public web pages (blogs, competitor sites, directories)
A web scraper agent can:
-
Monitor product prices or reviews
-
Extract company info or contact details
-
Summarize articles
-
Collect FAQs for training
-
Perform competitive analysis
🔧 Tools You’ll Use
| Tool | Purpose |
|---|---|
BeautifulSoup4 or playwright |
Extract content from webpages (static/dynamic) |
requests |
Make HTTP requests |
LangChain Tool |
Connect scraping logic to the agent |
Ollama/OpenAI |
Language model for response generation |
Trafilatura (optional) |
Clean article-like web content |
🛠️ Step-by-Step: Build a Web Scraper Agent
✅ Step 1: Install Required Libraries
pip install requests beautifulsoup4 langchain trafilatura
✅ Step 2: Basic Web Scraper Function
import requests
from bs4 import BeautifulSoup
def scrape_website(url):
try:
headers = {"User-Agent": "Mozilla/5.0"}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, "html.parser")
# Extract all text from paragraphs
paragraphs = [p.get_text() for p in soup.find_all("p")]
return "nn".join(paragraphs[:10]) # Limit to top 10
except Exception as e:
return f"Error scraping: {str(e)}"
✅ Step 3: Create a LangChain Tool for the Agent
from langchain.tools import Tool
tools = [
Tool(
name="ScrapeWebsite",
func=scrape_website,
description="Scrapes and returns article content from a public webpage URL"
)
]
✅ Step 4: Integrate into Agent
from langchain.agents import initialize_agent, AgentType
from langchain.llms import Ollama
llm = Ollama(model="mistral")
agent = initialize_agent(
tools=tools,
llm=llm,
agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
verbose=True
)
✅ Step 5: Example Use
agent.run("Scrape and summarize the main ideas from https://en.wikipedia.org/wiki/Solar_power")
The agent:
-
Calls
ScrapeWebsitetool -
Receives HTML-parsed text
-
Summarizes it using the language model
🧪 Use Case Ideas
| Task | Prompt |
|---|---|
| Blog summarizer | “Summarize this blog post: [link]” |
| Product data extraction | “Scrape price from this Amazon page” |
| Contact scraper | “Get all contact details from this company page” |
| Competitor intel | “Scrape and compare service features from [competitor.com]” |
🧼 Optional: Clean Full-Article Text
Use trafilatura to extract readable content from messy news/blog HTML:
import trafilatura
def smart_scrape(url):
downloaded = trafilatura.fetch_url(url)
result = trafilatura.extract(downloaded, include_comments=False, include_tables=False)
return result[:2000] if result else "No content found"
📦 Packaging Into a Chatbot (Optional UI)
import streamlit as st
st.title("🕷️ Web Scraper Agent")
url = st.text_input("Enter a URL to scrape:")
if st.button("Summarize"):
output = agent.run(f"Scrape and summarize the content at {url}")
st.markdown("### 🤖 Response")
st.write(output)
⚠️ Ethical & Legal Scraping Tips
| Guideline | Why |
|---|---|
Check robots.txt |
It may disallow scraping |
| Avoid spamming servers | Use headers and cooldowns |
| Don’t scrape personal data | Especially if not public |
| Don’t republish scraped content | Use only for insights or summaries |
| Prefer APIs | If available, always choose official APIs |
✅ Summary
| Step | Outcome |
|---|---|
| 🛠 Built a web scraper | Using requests and BeautifulSoup |
| 🤖 Wrapped it in a LangChain tool | So agents can use it like a skill |
| 📄 Extracted live data from websites | Pages, blogs, company info, etc. |
| 🧠 Enabled AI reasoning on top of it | Summarize, compare, extract insights |
80
