Scrape the web for data

Share & showcase
1

For a long time I wanted to have some kind of automation for scraping different kinds of web-databases and placing that data in an Obsidian-note and using Dataview to get an overview of music, movies, books etc.

I finally created a Keyboard Maestro macro for scraping Allmusic.com for album information. If we take an example like: Ghost Song - Cécile McLorin Salvant | Songs, Reviews, Credits | AllMusic, the Macro will then scrape the site for “album title”, “year”, “artist”, “genre”, “styles”, “duration” and finally “cover” (and this is where I need some guidance). Unfortunately, I can’t seem to manage to grab images and add as a variable. Any help would be appreciated!

Here is a link for the Macro - and please come with tips or ideas for improvement! I also want to do this for imdb.com.

Macro: Scraping webpages like Allmusic for data - Macro Library - Keyboard Maestro Discourse

2

And if you know how to do something similar with something else like Shortcuts, script or who knows from inside Obsidian, please share as well

3

Hey, I stumbled upon here while searching for a way to scrape GoodReads. What did you settle on finally?

I think for movies you could use tmdb along with the QuickAdd plugin.
Here is something I have worked on in the past. Shameless tweak of another brilliant script which gave me this idea.