Scheduling a web scraper online

The hosted scraping app comes with powerful and flexible scheduling capabilities for your agents to run them automatically on minutes, hourly, daily, weekly basis or any particular day(s) you want to scrape the data. In order to schedule an agent, go the scraping agent page and click on the "Schedule" button on top right corner. It will display the below dialog box.

scheduling the scraper online
 

Here you can go to any tab per your scheduling requirement to select your choice of frequency and then select the days, month and time to setup the schedule of a particular agent. For example : I have selected "Weekly" scheduler tab and checked on "Monday, Thursday and Sunday" with start time as 08:10 am in early morning will run my agent on these particular days at the mentioned time UTC.

Note : All the time in scheduler are in UTC - Coordinated Universal Time

Once you are done with the scheduling configuration, close out the dialog box and you will see the scheduling description, next-run-time on the agent page.

scheduled scraping jobs

Note : A particular agent can have only one schedule and each time you click on the "Save" button, the existing schedule will be overwritten if any

If you want to delete the schedule? Clicking on the "Reset" red color button on same scheduler dialog box will remove the schedule permanently.

The scheduling feature is most interesting to automate the data collection service running on server and using the scheduling feature you can configure a scraping agent to run anytime automatically to refresh your data. Here are some examples use cases :

  • Extract stock price from a website every 5 minutes and send an email alert when price drop/increases.
  • Extract price from competitor websites every hour and import output to SQL using webhook.
  • Scrape new jobs from jobs portal on Monday, Wednesday and Friday at 10:00am and send to a Google spreadsheet in your account.
  • And lots more..

Close me