Puppeteer / headless chromium if you need to deal with complicated front end applications. Or node and linkedom if you can get away with parsing server rendered markup.
But also, if you're lucky, you may just find if you look through network requests that there's a sensible XHR/JSON api waiting for you.