URL Advanced
URL Advanced reads web pages and splits their content into title, text, links and images. You can tell it which specific parts of the page you want, for example just the price and description of a product, and it can read the addresses that arrive in a list of data, one per item.
If you only need the text of a few pages, URL is simpler.
When to use it
- Extract the name, price and description from a list of product pages.
- Collect all the links on an index page to read them one by one later.
- Get the images of a product sheet to review or store them.
- Enrich a list of companies with the text of each one’s website.
Configuration
| Field | What it does |
|---|---|
| URLs | The addresses of the pages you want to read. Click + to add more than one, or write them separated by commas. |
| Data | Optional. A list of data in which each item holds an address. When connected, the component reads those addresses instead of the ones in URLs. |
| URL Data Key | The name of the field in Data that holds the address. Default: url. |
| text selectors | Optional. Which part of the page is taken as text, given as a web page selector, for example .description or article p. |
| image selectors | Optional. Which images are collected, given as a selector. Leave it empty to collect them all. |
| link selectors | Optional. Which links are collected, given as a selector. Leave it empty to collect them all. |
Advanced options
| Field | What it does |
|---|---|
| Base URL | An address placed in front of the ones that arrive in Data. Useful when the list only carries the last part of each address, such as /product/123. |
Output
The Data output returns one item per page:
- When reading the addresses in URLs, each item carries the title (
title), the text (text), the links (links), the images (images) and the source address (source). - When reading the addresses in Data, each item keeps its original data and adds the text, links, images and source address. In this case the text starts with the page title and, if you entered image or link selectors, also includes those images and links.
Connect it to an agent, a model or Parse Data. The component can also be used as a tool by an agent.
Example
A shop wants to compare its prices with a distributor’s:
- Prepare a list of data with the address of each of the distributor’s product pages in the
urlfield. - Connect it to Data on URL Advanced and enter in text selectors the selector of the price and description block, for example
.product-info. - Pass the result to an agent that extracts each product’s price and compares it with yours.
Good practices
- To find the selector of a part of the page, ask whoever maintains the website or use your browser’s inspect element option.
- When reading many addresses from Data, try a few first.