Syntax - Tasty Web Development Treats
The Undocumented Web: scraping, private APIs, proxies and “alternative solutions”
This episode isn’t summarized yet
Create a free account and summarize it. In a few minutes you’ll have the key takeaways and a section-by-section summary.
Show notes
What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices.
Kyle Prinsloo’s Freelancing & Beyond — Sponsor Kyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at https://studywebdevelopment.com/freelancing and use the coupon code “syntax” at checkout to get 25% off.
Freshbooks - Sponsor Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section.
Show Notes 3:05 - What Is the undocumented web?
When traditional methods don’t give you what you want
Wild West
6:10 - Undocumented APIS
Instagram’s private API
Use dev tools’ XHR Request
React Dev Tools will also have the data
Ep 008 - Wes Bos Origin Story
Realtor.ca - MLS Mapping
Simon Willison’s Zeit Day Talk about XML files
Charles Proxy App - also good for debugging requests
17:34 - Faking
Faking User Agents Sending headers from requests
Free Blackberry wifi on plane
23:52 - Scraping
What is craping?
Cheerio - jQuery like node package
27:30 - Automation
Cron Jobs
Filling out forms instantly
Headless browsers Puppeteer
PhantomJS
33:30 - Proxies
Need a fresh IP address
Need a different country
Side note: Canadians buying stuff in USA M4S1P1 === 41100
API throttling
39:00 - Tips and Tricks
Multiple API keys
diskDB - Text files as databases
Search Github for API keys
Inspect Element and Delete Overlays
Delete Cookiesokay
44:45 - Stories
Photo Voting
MLS Listing
Top Entrepreneur voting cheating
Vanity Plate Checker
My Hockey Team Sucks
Vehicle auction bidding app
youtube-dl
××× SIIIIICK ××× PIIIICKS ××× Scott: Octopath Traveler & OST
Wes 1: AvE Youtube Channel
Wes 2: Mustie1 Youtube Channel
Shameless Plugs Scott’s Level Up Tuts Pro
Wes’ Courses
Tweet us your tasty treats! Scott’s Instagram
LevelUpTutorials Instagram
Wes’ Instagram
Wes’ Twitter
Wes’ Facebook
Scott’s Twitter
Make sure to include @SyntaxFM in your tweets