Learning to screen scrape data using Google Chrome and curl

Pricescope.com Results

Screen scraping can be effective at getting free data very quickly. When attempting to screen scrape large amounts of data, I often use Google Chrome’s “Developer Tools” to obtain the steps necessary to recreate a web request. Here is an example process I used to screen scrape data from pricescope.com which contains a database about diamonds for sale online. I will use this data in an upcoming post on how to build statistical models.

Read More