How to deal with error 301 in scraping?

How to deal with error 301 when the response location is a only a part of the url needed?

I.E: I request a website

I get a response location "/users/page?=1"

How do you deal with that?

You can enable auto redirect and set the maximum redirect value to let DSS know to follow the 301 redirect.

In order to do this, follow the steps below

  1. Edit the scraping agent
  2. Go to advance settings
  3. Click on "Throttling" tab
  4. Check "Enable auto redirect" and set the values.

auto redirect in scraping

Your Answer

Close me