Twitter favorites 🐦 as an obsidian brain with graphs and canvases

[zerkshop]

I’ve always bookmarked stuff on twitter with the thought I would review or explore them later, but try scrolling back in time further than a few weeks and see how fast you lose interest.

Enter gallery-dl

OK, so how about I download all my favorites, and then dump them into an obsidian vault with some python code to link stuff up?

WIP results below

I’ve never had much use for the graph in my own vaults, but with this dataset it is actually very interesting. I added groups for: red = tags, yellow = tweets, grey = users. The middle node is a linear in time bookmarks.md containing everything (as embeds)

Using tags like #tag aren’t as good as [[tag]]. If I link all the tags up as notes, you can see which tags are related to each other

I can filter to show only the relationships of the tags. I can easily identify “tag islands” and use a mix of spreadsheets/python/manual work to curate things from here.

Here’s the note for tags/gamedev.md for #gamedev in source mode

in reading mode

the local graph may be useful too

I’m auto-generating canvases for everything. Here’s the canvas for the above tag. Videos are on autoplay for max visual stimulation.

I also output all the bookmarks in the linear-time order into a few dozen canvases of some given rows x columns. Here’s a python snippet to demo what writing a canvas is like:

import secrets

def create_node(path, x, y, width, height):
    return {"id": secrets.token_hex(8), "x": x, "y": y, "width": width, "height": height, "type": "file", "file": path}

def tile_nodes(paths, width, height, columns, x_space, y_space):
    nodes = []
    for i, path in enumerate(paths):
        column = i % columns
        row = i // columns
        x = column * (width + x_space)
        y = row * (height + y_space)
        node = create_node(path, x, y, width, height)
        nodes.append(node)
    return nodes

rows = 4
cols = 8
width = 400
height = 600
x_space = 10
y_space = 40

canvas_paths_all = ['tweets/' + os.path.split(m)[1] for m in written_files]

canvas_paths_chunked = [[]]
for p in canvas_paths_all:
    if len(canvas_paths_chunked[-1]) < rows * cols:
        canvas_paths_chunked[-1].append(p)
    else:
        canvas_paths_chunked.append([p])

for i, paths in enumerate(canvas_paths_chunked):
    nodes = tile_nodes(paths, width, height, cols, x_space, y_space)
    canvas_data = {"nodes" : nodes, "edges" : []}

    canvas_file = os.path.join(vault_dir, f'Bookmarks canvas {i:02}.canvas')
    with open(canvas_file, "w") as f:
        json.dump(canvas_data, f)

The individual bookmarks are threaded, within certain constraints

• only include tweets that include the one I bookmarked, and/or the consecutive tweets by the original author of the conversation (it’s a little complicated!)
• discard non-author tweets (if I didn’t specifically bookmark it)
• multiple bookmarks from one conversation are ok and will be shown in the top section, followed by the rest of the conversation

Here’s one bookmark showing the tweet (includes a quoted tweet), followed by a Conversation section containing the rest of the consecutive replies by the OP, followed by a section for my Notes. You may notice in the other screenshot the embed of this tweet did not show the conversation. It was embedding with the ^000011 block reference (clean with no titles etc. or css hacks)

Here’s one where I bookmarked 3 items out of the conversation. The bookmarks are shown at the top of the note, followed by the full 15 tweets in the thread (“15 bad habits holding you back from your potential…”) (don’t judge my productivity porn!)

Working with gallery-dl is challening. I have a super messy patch to get the info on my favorites in a reasonable way, while only expanding threads/replies in a constrained way to not pull in irrelevant replies and advertisements. Also rate-limiting.

Could web embeds work? It would be much more limited. Side by side in a canvas:

As an ADHD person, I need these 2D visual views to see everything at once, and to have categorization. The flat list of favorites in twitter is terrible and I have no motivation to interact with it.

This applies to literally every social media site too (instagram, youtube, …). Zero or limited ability to organize followers, favorites, etc.

I will be experimenting more with canvas to try organizing more things like these, and other notes in general. Including somehow browser tabs… which I have 640 open in 12 windows and 27 tab groups (vivaldi vertical tabs). And it’s not working haha. Maybe I can canvas-ize them somehow to see them all at once, or under different tags/intentions. Currently I have a script to export them to markdown, but the 1-D text file is not very inspiring.

Thanks for viewing my WIP and/or rant! I hope to share the source after fixing bugs and making it a bit less fragile.

[vincer]

Impressive! I assume youre doing all that in a vault solely for that? Imagining that many entries and tags dumping over an already filled vault could be a bit disorienting…

I was thinking on doing something somewhat similar but manually(im no coder)- so not all my favorites or such but some curated tweets or threads.

Im wondering, could there be some version of what youve made to create/populate the canvases by tag more open ended- like running the code/script just changing a target tag and such? I also have ADHD and one of my main struggles with Obsidian was the flatness of it all - i had even gave up on Obsidian and resorted to Hepta for awhile, i just came back thanks to the canvas…

heck, ive pretty much ditched page views and write directly on canvas, i barely use tabs now- ive been making canvases called desks, as temporary, and if any becomes cohesive i rename then, its awesome… And while the canvas is great for plotting things, thinking, manually making relations… its all manual

I still struggle with obsidian having no way of seeing notes in some sort of overview- but unlike the graph showing the contents. Some grid/thumbnail view filterable…

Some way of auto-generating canvases from tags could be a interesting workaround…

[zerkshop]

Hmm, I haven’t heard of Hepta before. It looks interesting. It looks like it is taking some notes from Obsidian as far as the style/theme and offline access. I’d be curious to hear what worked/didn’t work for you with it.

could there be some version of what youve made to create/populate the canvases by tag more open ended- like running the code/script just changing a target tag and such?

Yeah for sure. Having played a bit more with the Obsidian plugin API I think this would also be an easy plugin to make. One of my existing ideas is to make a sort of external interface to query and combine tags and stuff, and that could then generate and open the canvas or other outputs (a “query” note with a bunch of embeds?) it generates in Obsidian. I’ll let you know when/if I pick this up again…

heck, ive pretty much ditched page views and write directly on canvas, i barely use tabs now- ive been making canvases called desks, as temporary, and if any becomes cohesive i rename then, its awesome… And while the canvas is great for plotting things, thinking, manually making relations… its all manual

That’s interesting to hear and makes me want to try writing directly into canvases again.
My recent tricks are using a bunch of tab rows and using this rofi tab switcher from my desktop (although needed less since switching to mulitple tab rows).

Also do you think the novelty of your new method will wear off and need to pivot to a different method in some time? That’s a definite ADHD thing, and also worth expecting and accommodating that pivot. I’m sure it will keep happening for me, and I’m trying to work with it.

Whenever I’ve tried writing directly on canvas, it hasn’t stuck for me. I kind of end up falling back to just pen & paper to have more freedom. But I’ve also thought about if there’s a workflow where I can use a printer & my scansnap scanner to sketch & mark stuff up, and get it in and out of obsidian quickly. I got to playing around with pdfs in canvas, and found it looked and felt better with the pdf exploded into individual pages shown side by side in a grid as images (get rid of the annoying pdf viewer widget, and page scrolling, also the pages rendered crisper this way as images at different zoom levels).

Here’s an example of dumping a stack of scanned pages as images side by side, with little boxes for additional notes below:

Other issues preventing me from spending more time living in the canvas:

• It can’t remember the scroll position of long notes. So I feel like I need to re-do everything as super atomic notes with no scrollbars (which is also something I’m thinking of… like just make every note a “tweet” right?)
• No render of nested canvases. It feels like it would be much more powerful if we could view and interact with nested canvases, similar to embedded notes… Maybe I will spin this comment into it’s own feature request

I still struggle with obsidian having no way of seeing notes in some sort of overview- but unlike the graph showing the contents. Some grid/thumbnail view filterable…

Yeah I’m with you on wanting some way of “seeing everything at once”, or some kind of summary. I also like the idea of injecting randomness into the process like having a slideshow of specific notes/tags always on, so something might pop up and catch your eye. Actually one of the ways I’ve done something similar with all that twitter data from gallery-dl is to have a spare screen showing a shuffled video slideshow, playing loops of all the .mp4 files in a grid. That has been helpful/inspiring. I can see taking it further to making a sort of slideshow of the full tweets (that could done in a few ways, using the extracted markdown, or using screenshots of tweets, or… hmm feeding a “playlist” of tweet URLs to a chromium driver that will periodically go to a new random URL (e.g. with selenium or debug mode))

[vincer]

Hmm, I haven’t heard of Hepta before. It looks interesting. It looks like it is taking some notes from Obsidian as far as the style/theme and offline access. I’d be curious to hear what worked/didn’t work for you with it.

I was in love with Hepta (kinda still am, a bit torn between the two) but also increasingly more worried about what route it might take (it started local only, based on markdown, then markdown only on export, now increasingly focused on cloud).
In short Hepta is what if Obsidian started and centered around canvases (called boards there), and unlike Obsidian`s canvas theres no cards from boards- every card/block creaed on a board is a individual card globally and can be in any number of boards - wich have its pros and cons.
Boards can be embeded into others (so entering one shows in the breadcrumbs the previous ones) and linked elsewhere, any cards can be folded and embeded boards renders some of the content- but instead of seeing the whole board in miniature it becomes a list/grid of all cards inside showing only their first line/titles.

Besides that its very very close… theres a graph, tabs (albeit vertical, on the left), and no custom panels- instead a universal right side pane that can have any number of notes stacked on top of each other, foldable (pretty much like Logseqs right pane)… and instead of a file tree on the left notes can be browsed as 1 big list on the main view, but also as grid-thumbnails (the one thing i miss the most in Obsidian).
…and it have recently started rolling out a native database function, that unlike notion is always global, but have multiple views (ie multiple ‘databases’) - any tag can be used to make its own individual table, properties, so on.

Theres a lot of quality of life regarding boards that Obsidian could definetly learn a thing or two- like listing on every note what boards theyre on. Im also hoping Obsidian eventually bring grid-thumbnail view, or just go all way bringing ‘views’ (amongst then tables/database-like)

Only thing i missed from Obsidian was the amount of flexibility and plugins (none there)- but i also missed on pictures. As an alpha tester i still have access to hepta without paying (currently i think the acess is still gated for subscribers), something i cant afford right now (low income country, unemployed, my freelancing doesnt leave much room) - and both pictures and pdf were gated for subscribers only. Given my interests (design, gamedesign, also my ttrpg table) pics is a must for me…
PS: Hepta just rolled out local pictures, apparently for local use only too. I dont think its a coincidence it happened just after Obsidian released canvas, that alone is what made jump right back into Obsidian.

I’ll let you know when/if I pick this up again…

That’s interesting to hear and makes me want to try writing directly into canvases again.
My recent tricks are using a bunch of tab rows and using this rofi tab switcher from my desktop (although needed less since switching to mulitple tab rows).

Oh i will definetly try the multiple rows - one of the reasons ive been favoring tabs less was how many(little) could fit at the top. But after limiting tabs and focusing on canvas i dont know if i would switch so soon- swaping tabs much like alt-tabing kinda breaks the flow for me and get me lost. Maybe something like rofi could help (but then im on windows, so no Rofi). Much like alt-tabbing swapping tabs works better with less tabs, or swapping between both - at wich point a panel or second window would be as effective. 4-5 tabs or more i get lost quickly (swapping on the fly i mean, with the intention to go back to the previous tab)

What ive been loving as alternative to tabs (beside writing on canvas) is the Hover Editor Plugin (ITS A LIFESAVER, i dont know if i could have Obsidian without it now)… basically hovering over any links or files shows the contents in a window, but said window can also be edited, dragged, stay on screen, folded… i use recent files(another plugin) more then the filetree, i just hover over what would be tabs, drag then around, fold then, and swap right there…

…its almost as making a ‘temporary canvas’(minus arrows) on top of the entire app, one where every note can be out of the way (folded). Its awesome. I even made a canvas called ‘clipboard’, starred it, and always make it hover folded- so i can just copy/paste, link or whatever between notes and canvases (oh btw theres also that- it also works with canvases, entirely navegable and foldable on the screen on top of anything)

Also do you think the novelty of your new method will wear off and need to pivot to a different method in some time? That’s a definite ADHD thing, and also worth expecting and accommodating that pivot. I’m sure it will keep happening for me, and I’m trying to work with it…

Oh definetly. Almost nothing sticks for me ever, save rare things across the years (its almost like natural selection, except things can go back and forth based on mood) - but in this particular case im not sure it will go away, just because it fits with a lot of my habits over the years- like ‘whiteboarding’/mind-maping on paper, wich never gets old for me. In every app i could ive always written in at least 2 columns (if not 3 or more)- reason why i stuck with onenote for years (basically all pages are canvases)… on notion i always turned every page in 3 columns at least…

Its probably an habit from school/notebooks- i also left margins or almost half a page for ‘sidenotes’… and that was also one of my main struggles with obsidian (ive tried it once in the past and gave up)- sure we can have panels, and stacked tabs (andys mode, wich i also love)- but those are different notes. I always feel the need for some separation between ‘main note/body’ and whatever observations, some checklists or task list, etc.

Btw i figured a ‘hack’ using hover editor just to get ‘sidenotes’- on some notes i make a header on the end, called sidenotes, and make a link on the top of the note to itself, directly to the sidenotes header… now when i open a note that have said sidenotes link i hover it and the plugin allows me to have a pop-up window already targeted there… so im writing on the main view and plugin on the same note- on file theyre linear, i just make 1 section becomes ‘on the side’ - btw that also works just opening the same note on a sidepanel, but hover editor is quicker and i can fold/unfold

…

Whenever I’ve tried writing directly on canvas, it hasn’t stuck for me…

I feel you, theres some pain involved - cards/notes dont resize as we type, longer notes doesnt work that well… and also the view - sure i zoom in when writing on a longer note, but it isnt as nice as full page view.
But those pain points so far have been outweigthed by the pros for me, at least for now.

I do use some workarounds- some shifting in how things are done

1. Ever since i started ive leaned more and more as treating writing on canvas as drafting and now even the start of making notes- and instead of one or few longer texts i separate as cards. What would probably be different paragraphs can be cards… clusters and so on
2. then later i decide if/how to slice a cluster- 1 single note? All cards go into one(cut-paste) then i just right click ‘turn into note’… some get rewritten or shortened (often because i followed a tought, as i tought, that could be much quicker summarized). Sometimes a cluster become 2-4 different notes
3. Also have worked with previously longer notes. Theres a neat feature on right clicking on a note and choosing ‘narrow to heading’. The same note can be multiple cards, each narrowed to show a particular heading.
4. case by case a long note entirely on canvas have also been useful to me, like longer notes im migrating, with mostly ‘finished’ text so to speak - i just stretch the note(really long)- and start making cards on parts im comenting or wondering what to change… once you get used to it it works really well- later i just make changes based on the cards already on the side. The rest is scrolling on the canvas itself- with the note fully extended (instead of rolling inside a smaller card) it becomes a main ‘path’ with localized smaller cards on the side
5. If i ever feel too much pain writing too much on 1 card i can always open it in new tab, window, pane… or just right click ‘open in hover editor’ (wich can also be edited full screen, resized as a size pane… with the benefit of folding/unfolding)
6. The canvas mindmap plugin can also make things easier. Its a bit buggy for me on some of its features, theres room for improvement, but the ctrl+arrows shortcut always work: writing on a card, one quick shortcut to make a new card next to it (up, dow, left or right by choice) already in editing mode… its really neat

I also felt the current pdf widget on canvas to be lacking (one pro to Hepta on that) and im not sure yet how i will fare with it on canvas (i do want to analyze pdfs in a canvas, but those would be big books- too much to explode in individual pages)

But in regards to sketch/drawing on paper- have you ever tried it digitally? I still like paper, sometimes i use it, but ever since i went and tried to stick with digital the benefits were too big to ignore. I hook up a drawing tablet on the pc (for me its a Huion kamvas). I got it mostly for actual drawing on illustration apps and vector design, but on a daily basis i end up using it more to quick sketch notes like i did on paper- only it saves all the old trouble of scanning or taking photos or missing papers, or then becoming scratched or damaged in any way… ive been doing that in different apps for awhile (like Mischief, abandoned but it still working, infinite canvas and zoom wich is insane), but now that i came back to Obsidian theres the Excali Draw plugin…
And it embeds on canvas beautifully. Just double clicking on it to pull up the sketch for editing, and all there showing changes on the canvas.

But overall im of the same opnions- i also hope for embeded canvas being rendered, better card handling and so on… at least im hopeful that sort of thing might come sooner rather then later, after all they just added canvases.

Actually one of the ways I’ve done something similar with all that twitter data from gallery-dl is to have a spare screen showing a shuffled video slideshow, playing loops of all the .mp4 files in a grid. That has been helpful/inspiring. I can see taking it further to making a sort of slideshow of the full tweets (that could done in a few ways, using the extracted markdown, or using screenshots of tweets, or… hmm feeding a “playlist” of tweet URLs to a chromium driver that will periodically go to a new random URL (e.g. with selenium or debug mode))

Wow, now thats interesting! ive taken notes with moodboards on the side, but mostly static pictures. Ive even done it with some moving ones (i use Eagle.cool, i still make moodboards but much less now since its so easy to filter by tags or colors there, and it animates gifs on the grid), but i never tought of videos on loop(on a grid) - that would probably be an overkill for me, but it got me curious and could sometimes be interesting… what you use to plot the videos looping in a grid?

What youve mentioned of slides tough caught my eye even more- sure we have hundreds of apps that can auto-slide through media, but imagining a slide like that inside of obsidian… a panel that could be cycling (in order or random) ANY notes/files, probably filtered by tag… now THAT would be incredibly useful for me- it wouldnt just mean random references but also resurfacing our own knowledge and ideas… wich in turn would help a lot of linking- i always feel like im missing possible links or ‘a-ha’ moments that could happen if i remenbered all my notes or wondered if A and B could go together. We have plugins like random notes, the dice roller plugin can also be great for that, but those are ‘clicking to pull up something random’… a slideshow would be AMAZING

[zerkshop]

Thanks I love the info dump here. I’ve skimmed through some and I think there’s a bunch of stuff I’ll comment on when I can!

[zerkshop]

I printed your last reply in newspaper columns to read it and highlight some stuff which was fun . Some notes from me:

For no rofi on windows: it might be possible with WSL2, or alternatives https://alternativeto.net/software/rofi/?platform=windows

On hover editor: i was experiencing some glitchyness when I tried it recently, but the way you describe it makes me want to look again.

Recent files plugin, different than builtin sort by date modified?

I’d be interested to see how you use canvas/hover editor and everything. It would make for an informative showcase post or video!

I like the points you made about how to edit in canvas, making stuff that might be paragraphs in a single doc into cards. And similarly making cards alongside a longer doc, possibly as a way to start breaking it up. I think I want to experiment with some of that.

For pdfs and stuff, have you looked at this one? I have yet to try it…

GitHub - elias-sundqvist/obsidian-annotator: A plugin for reading and...

• It allows you to open and annotate PDF and EPUB files. The plugin is based on https://web.hypothes.is/, but modified to store the annotations in a local markdown file instead of on the internet. Also supports youtube annotations

Funny you mention Mischief, I still have it and saved the installer. I also saved this link if I want to do something with the data at some point like convert it to svg maybe GitHub - m1el/mischief-re: Reverse-engineering a vector format

Excalidraw, I also poked around with, but somehow it didn’t vibe for me at the time. Maybe another one for me to try again. I’ve also figured out using Weylus GitHub - H-M-H/Weylus: Use your tablet as graphic tablet/touch screen on your computer. which might solve me wanting to use my desktop PC for everything, but another tablet for drawing, but not needing to open a 2nd vault on a 2nd computer.

For looping videos, I’m using mpv and e.g. opening 4 instances, setting --geometry= for each, and giving each instance a playlist and so on. And then that’s further scriptable if you want to vary the tiling, 1x2, 2x2, 3x6, etc. And --panscan=1 to crop black bars in the tiles.

For slideshow potential in obsidian, here’s a low effort idea:

• have a note call slide.md
• have a script re-write the note every N seconds
• the note content could be either the fulltext of a random note, or an embed link to one
• and then the script could be customized to pick which subset of notes or tags to show
• obsidian should auto-reload the note on changes, so if you have it in a permanent “slideshow pane”, there’s your flash card section

probably an easy obsidian plugin as well, but seems like a pretty simple script to run on the side. I just asked ChatGPT to write this powershell script if you want to try:

while($true) {
  $files = Get-ChildItem -Path "C:\path\to\markdown\directory" -Filter *.md | Where-Object { $_.Name -ne "slideshow.md" }
  $file = $files | Get-Random
  Copy-Item -Path $file.FullName -Destination "C:\path\to\markdown\directory\slideshow.md" -Force
  Start-Sleep -Seconds 10
}

[zerkshop]

GitHub - digitalsignalperson/gallery-dl_twitter-obsidian: gallery-dl with...

Dumped the code here, but sorry I haven’t used it in about a year. For sure it would need rebased, but if you are willing to hack on things it may be of use.

[sundayscoop]

Wow, I was looking for something similar to this but with the capability to download all the tweets from a twitter profile with attachments and make them notes to be able to study and search through them later. This is the closest thing I have found to what I am looking for.
I am willing to pay if you could build something like this. Thx