How to export scraped data to files or post to server
Data Scraping Studio supports multiple output formats to export scraped data to files or to send data to server via HTTP Post(webhook) with both manual and automatic methods. In this tutorial we will cover how to write scraped data automatically to a file in local drive with
Overwrite mode or posting data to server.
Local output file format
Post to Server
Data Scraping Studio has 3 output options (Default, Save to Local Drive and POST to server) described in details below.
As name describes, this is the default option in Data Scraping Studio and scraped output will be displayed in output grid and will not be stored anywhere. But you can manually save/post to server after extraction completes or anytime you want.
Save/Save as : To save data manually, right click on grid anywhere and click on the save option will open the save dialog and you may provide location, file name and file type and then click on the save button.
Post to server : To post data on server manually, right click on grid anywhere and click the "POST to server" option will send a HTTP post request with JSON data to webhook URL provided on scraping agent setup.
Save to Local Drive :
The "Save to local drive" option will write the scraped data to a file in your local directory writing in parallel as extractor is scraping website.
Save to local drive option will start writing output to local file as extractor extract any records instead waiting for extraction completes and then writing the output to local file.
- Create/Edit a scraping agent > Go to "OUTPUT" tab
- Give a name to your output file with extension. For example (Output.csv will write a CSV file and Ouput.json will write a JSON file on selected location)
Write Mode : By default, overwrite mode. Means, if the same file exists on given location will be replaced with new data. To keep the old historical data you may change the write mode as "Append" will append the new data after last row of existing file.
Include Headings : Whether you want the output column headings should be in written in local file or not.
Note : Write mode and include heading option works only for CSV, TSV, TXT file types.
Post to server
The "Post to server" option is most used technique to automate the data collection process. It allows you to configure your webhook URL and let Data Scraping Studio post the scraped data to your server using
HTTP POST method when the extraction job completes.