Web Scraper
Web Scraper visits the web pages you give it and returns their content as tidy text, with headings and sections, ready for a model to read. It needs no credential.
When to use it
- Summarising an article or a competitor’s page.
- Letting an agent read a page the user gives it.
- Storing your website content in a vector database for an assistant.
Configuration
| Field | What it does |
|---|---|
| URL (required) | The page address. For several, separate them with commas. |
Output
The Data output holds each page’s content. A page that cannot be read comes as an item with its url and an error field explaining why. In tool mode, an agent can read pages by itself; see Tool Mode.
Example
An agency wants to summarise three competitors’ pricing pages every week.
- Add Web Scraper with the three addresses separated by commas in URL.
- Pass Data to a model that compares the prices.
Good practices
- Some websites block automatic visits or load their content in a way that cannot be read; test with the specific page first.
Related components
- Web Search (SearXng): searches the internet for pages.