Knowledge Base
npm install linux with packages for scraping
joe p screen scrape
2019, updated 4/2021
install:
cd /home/eturock
mkdir joep && cd joep
npm init –yes
apt get install npm
npm install request-promise-native cheerio jsonfile –save
Request is the library we’re using to make HTTP requests, cheerio will parse our document body and allow us to access it with Jquery-like navigation. We’re using request-promise-native here to make life easier a little bit further on.
Jsonfile combines Node’s native fs library with JSON parsing and conversion. This one is not essential, but I think it keeps it cleaner to not have to read file, parse JSON, do stuff, convert to JSON, and write to file again. Our first pass won’t really use this, but we’ll get to it later.