How to record and scrap content on click-through

For example there is the category page
furniturewebsite.com/livingroom-furniture.html

This page would have products listed and I would need to click on each product in this category listing page, and scrape the product details page content. then go back up to the category page and "click" on the next product and scrape the contents.

Then ultimately go to pagination go to page 2, and "click" on products and scrape details for each of those products.

Can this tool do this?

Posted by Gheorghe Pasat 7 months ago


@Gheorghe You'd need to create 2 scrapers and connect them to follow the link extracted by agent 1.

  1. Scraper 1 : This will extract the product url.
  2. Scraper 2 : This will extract all the further details from each product page following the url extracted in Scraper 1

For example I created a sample agent and gathers the URLs only from stackoverflow question page . By running, this extracted 15 question links from the list page.

scraper connnector

And, on this 2nd agent setup I configured my agent to read URL from the agent #1 to extract additional details.

scraping agent setup by input url from source scraper

Execute the agent and it will read all the URL form source agent and extract the fields as you've configured.

click through extraction

Posted by anonymous 7 months ago

Topic Closed! This question is closed and don't accept posts now.

Close me