Web Extract Agent (Legacy)
Web Extract Agent (Legacy) pulls information out of web pages for you. You give it an address and an instruction in your own words (“get every product with its name, price and availability from https://example.com/shop”) and the agent plans what it needs to collect, goes through the pages required, uses a real browser when the site loads its content dynamically, saves what it finds while removing duplicates and returns the data organised.
It is an older component kept for compatibility with existing flows. For new work with web pages, see Web Agent.
When to use it
- Maintaining existing flows built with this agent.
- Extracting an online shop’s catalogue with prices and availability.
- Collecting the plans and prices a competitor publishes.
- Summarising the content of one page or several pages of the same site.
Configuration
| Connection | What to connect |
|---|---|
| Model (required) | The AI model that plans the work and decides which pages to visit. It must support tool calling. |
| Fallback Models | Backup models, in order. If the main model fails several times in a row with temporary errors, the agent retries and, if it keeps failing, carries on with the next backup model. |
| Tools | Extra tools the agent can use on top of its own, for example an internet search tool. |
| Field | What it does |
|---|---|
| User Prompt | The instruction on what to extract and from where. It must include the website address. Default: {input}, which is replaced with the contents of Input. Each variable in curly braces becomes a field with that name that you can fill in or connect. |
| Input | The text the agent receives, usually from Chat Input or when another agent uses it as a tool. If it has content, it replaces User Prompt. |
| System Prompt | Your own instructions to replace the agent’s built-in ones. Leave it empty to use the defaults, which already tell it to plan, save as it goes, avoid repeating work and stop when it is done. |
Advanced options
| Field | What it does |
|---|---|
| Max Iterations | The maximum number of agent steps. It stops earlier if it completes all its objectives or stops making progress. Default: 80. Raise it for large sites with many categories or pages. |
| Output Schema | The exact fields you want in each extracted item, for example {"name": "string", "price": "number", "url": "string"}. If you fill it in, the result only contains those fields. If you leave it empty, the agent decides the structure based on what it finds. |
| Enable Playwright | Allows a real browser to be used for pages that load their content dynamically (infinite scroll, “load more” buttons). If you turn it off, the agent is faster but may miss that content. Default: on. |
| Enable Python Sandbox | Allows the agent to run code in an isolated environment to clean, transform or group the data already downloaded. Default: on. |
| Enable SQLite | Allows the agent to save data in a workspace of its own for each run, which removes duplicates automatically. Default: on. |
| Enable Scratch Curation | When it finishes, reviews the data collected automatically and keeps what belongs to each objective. It has no effect if Enable SQLite is off. Default: on. |
Output
- Data: the extracted data, grouped into collections by type (for example
productsorplans). If Output Schema is set, each item only has those fields. It also includes a_curationsection summarising the final review. If nothing was extracted, it returns a text with the agent’s answer and the job statistics (pages visited, items found, errors).
Example
The consultancy Mercado Claro compares a competitor’s prices every week:
- Connect an AI model to Model.
- In User Prompt, write: “Extract the name, price and availability of every product at https://competitor.example.com/catalogue”.
- In Output Schema, define
{"name": "string", "price": "number", "available": "string"}. - Connect Data to a component that saves the data to a spreadsheet.
Each run returns the list of products with those three fields, without duplicates, ready to compare.
Good practices
- Be specific in the instruction: give the address and the fields you want. The clearer it is, the closer the result is to what you expect.
- Define Output Schema if later steps of the flow always need the same fields.
- Respect the terms of use of the websites you query.