My first note on 2022-03-23 was Meta.md and in summary I wrote this:
Objectives of this vault:
- Storage of ideas: Prevent link rod, improve retrievability
- Originality: Foster new connections between ideas
- Mind Mapping: For easier restoration of the original
- Review: Implement spaced repetition and journalling
- Productivity: Inspire new projects and developments
Principles:
- Economics: Prioritize - time is finite
- Automation: Let machines handle repetitive tasks
- Freedom: Express yourself without fear and constraints
- Portability: Stay independent of specific tools, plugins or formats
- Flexibility: Remain open to change
- Enjoyment: Keep it fun, not work
- Content First: Structure serves the content, not vice versa
Yesterday I listened to Dwarkesh’s interview with gwern and this passage stuck with me:
Dwarkesh Patel: Now that you know that AGI is a thing that’s coming, what’s your thinking around how you see your role in this timeline? How are you thinking about how to spend these next few years?
Gwern […] I’m doing things now because I want to do them, regardless of whether it will be possible for an AI to do them in like 3 years. I do something because I want to. Because I like it, I find it funny or whatever. Or I think carefully about doing just the human part of it, like laying out a proposal for something.
[…] otherwise, I’m trying to write more about what is not recorded. Things like preferences and desires and evaluations and judgments. Things that an AI could not replace even in principle.
The way I like to put it is that “the AI cannot eat ice cream for you”. It cannot decide for you which kind of ice cream you like. Only you can do that. And if anything else did, it would be worthless, because it’s not your particular preference.
When I look back at what my intentions were when I started using Obsidian and how I ended up using it for past 2.5 years, I notice I have had very little motivation to do things I felt could be “patched up in post” with AI like linking notes, formatting, grammar. I think I maybe linked 20 notes in total and opened the graph view once. I catch myself regularly being envious of people sharing their neatly structured vaults e.g. on youtube.
But that interview with gwern has now given me a clearer view and confidence for how I can use my vault to create lasting value for me out of the little time I have outside work to use it. I’ll sketch out some ideas for how to implement them below as well as some dead ends and pain points along the way.
First up, I basically have only two folders. One for journalling which I imported from Bear at some point. The other is my version of a Zettelkasten system for notes.
Folder structure
- /Journal: periodic notes. don’t polish, just write.
- /Notes: Everything else goes here
- /Templates
- /Inbox.md: Anything that’s not yet sorted lands here
- /Index.md: Entry-point to the vault.
- /Meta.md: notes about this vault
Journalling
My ideal workflow for journalling would feel similar to calling my partner, a close friend or therapist and sharing with them what happened today and how I feel. And just as people who know you well occasionally point out patterns in your behavior you might not be aware of, my journal would also include regular reviews of past experiences and track lifestyle patterns. Unfortunately, I feel quite far from that vision.
My journal notes are structured [YYYY]/[MM]/[YYYY-MM-DD].md and go back to 2012. There are gaps, sometimes months, but I feel like a lot of those wholes could be reconstructed from personal messages and emails I wrote to friends or conversations with ChatGPT.
My first paint point here was that pulling in content from external sources is not easy and can bloat the daily note. For example, emails, chats with friends, my browser history, Steam games played, screen time, location data and various health metrics and activities could be used to create an automated lifestream of everything I did digitally. Every year I export a takeout of every online service I use.
At some point it felt natural that Obsidian could be the hub where all of that data flows together. I tried various tools but eventually gave up on that because none were frictionless or could be easily automated. I blame it on the incompatibility of Obsidian’s mobile experience paired with Apple’s ecosystem (I use iCloud for storage).
It is also very tempting to just use Apple’s Journalling, Screen Time and Voice Notes apps. My partner used the Journalling app on her phone during our vacation and it was fantastic to see how the app can use all the location data and photos to lay out entries for you. I even tried Rewind.ai for a while but they all just create new silos and need another expensive workflow to liberate the data again. A year ago it really seemed like Apple was missing the boat on AI and I ended up building my own whisper API Shortcut to transcribe voice notes and import them into Obsidian. Today I’m bit less worried about where Apple is headed in terms of UX but I want to own my data. That’s where I saw Obsidian had in principle a unique advantage and I spent considerable effort on making it work the way I wanted.
Today I think I better use Obsidian a bit more narrowly for traditional journalling for when I feel like typing out my words on my laptop and then use it as one (heavily weighted) source of input to combine with other apps and data to eventually create my lifestream elsewhere in the future.
In terms of reviews, I have these:
- Annually:
[YYYY]/Review.md - Monthly:
[YYYY]/[MM]/Review.md - Weekly: [
YYYY]/[MM]/[W].md
Here too I was hoping they could be automated based on my daily notes alone. My motivation to write down how many times I went to the gym this week is limited when that data exists on my Apple Watch already for example. But even if I just limited myself to writing about how I felt this week, knowing how many times I went to the gym this week would actually be really helpful in drawing conclusions about it.
I once did a massive annual review where I pulled in data from about every source I had. It took me a week of full time work during my christmas holidays. In the end it was maybe one of the best things I ever did but it was spent 95% doing laborious tasks that should be automated instead.
I’m not at the level where I would trust an AI to give an accurate summary of my life even if it’s just the past week. But something like a sentiment analysis like “this week you wrote less enthusiastically than last week. Is everything ok?” would be helpful.
There already is a “source” and “reading” mode in Obsidian, I’m thinking the reading mode could be much more than just rendering markdown. It could also do summaries, add punctuation/grammar, generated frontmatter, show related notes or be chat interface. But it would never touch the “source” which only you can edit. Which brings me to my next point
Maintaining notes
I’m fine with notes just being one folder full of unorganised text files. An LLM is better suited at arranging them in a latent space than me having to decide how I want to tag my movie reviews.
That being said, even if I leave the “gardening” entirely up to the future robots, I still would like some paths and vantage points for myself, to use a metaphor, to track the progress. For example, I still would like a way to differentiate between what notes are original ideas about future projects and what are reviews or highlights from existing things like books or articles. Or whether a note is a rough draft I want to work over time vs a finished piece. When I’m writing down my ice cream preferences in note, I still would like there to be some metadata that says “this was written in 2018” or “this is a food preference”. Even if all that can be inferred eventually from context, I would like to keep track of what things I have already written about to easily spot areas I have not written about yet.
I prefer to have phases where I’m doing quick spontaneous writing that’s not slowed down by thinking about how this fits into anything and then phases where I tend to the “gardening” of annotating and polishing notes. This fits naturally with my need for creation and curation.
Unfortunately I haven’t found good ways for Obsidian to achieve that balance. Once a note is closed it easily disappears among thousands of others.
Products like Superhuman have shown that “gardening” can be gamified. I would like a view in Obsidian that lists notes by some arbitrary query like “haven’t edited in a year, no tags or frontmatter”
and single letter keyboard shortcuts for “delete”, “stash for later processing”, “apply tag/property X” (inferred by the content of course). This should also work on mobile via an intuitive “swipe” UI that allows quick triaging of notes.
Haven’t found anything like that yet.
For a while I used the spaced repetition plugin to resurface some older notes but ultimately also found that it was too narrow for my use case.
Instead I use tags:
Tags are abstract, meta-level categories typically not mentioned within the note itself but useful for grouping similar notes. They can be nested to indicate a taxonomy structure on the highest level:
- status: the state of the note like draft, finished or needs review
- library: reviews of something that somebody else produced like books, movies, food at restaurants
- type: notes can represent anything but some are articles or highlights
- bio: something I have a personal relationship with like a person I have or places been to
Keep first-level tags to a minimum. Some tags are enforced by plugins like, flashcards, review or task.
Prefer setting tags in frontmatter and using properties instead.
Untagged: usually a good indicator that the note is stale and needs review
For each of these tag taxonomies I first used MAKE and then settled on DB Folder to create a database e.g. for all movies I reviewed or all places I’ve traveled to. That plugin is slow, buggy and no longer maintained so I’m instead peeling my eyes on alternatives like datacore or vault explorer. It feels like a whole that would be better filled by Obsidian itself but there is little that makes me hopeful in that regard.
For instance, I’m transitioning some tags to properties but what’s stopping me from going all the way to properties is that renaming tags is much easier with Tag Wrangler than what I found with properties UI. For example, if I had #task I could rename that to #todo within seconds.
Not to mention nested tags: with properties I would have to write something like type: highlight \n highlight-source: readwise and the only bulk editing Obsidian gives me is renaming the property key not the value. There is also no nice way to see how many “highlight-source: readwise” notes there are, just notes with that property key. I didn’t find any community plugins for that so the whole property aspect feels neglected.
Obsidian’s search is amazingly powerful but unfortunately its UI feels only adequate for a quick search widget, not for a full-on database-like view to interact with your vault with bulk editing, columns, shortcuts etc. Datacore is promising since it is being built with editing in mind as a first-class citizen. I have a lot of hopes in that regard and wish there was an SQL-like syntax not just for querying notes but also bulk editing frontmatter. I did some research for existing tools that essentially convert the YAML frontmatter of a folder full of text files into a database-like abstraction but that went nowhere.
Let’s say I decide today that importing my reading highlights was not the right move. They are all easily findable via their tags/properties in frontmatter. But there is no way for me to do a “select all delete” command over all matched files.
Active reading (importing highlights)
As part of my active reading habit, I import highlights from Omnivore, Readwise, Kindle, Zotero and Apple Books so that they can either form the basis for new ideas or reflect things I’m currently interested in. At various times I considered ditching all of these services and instead tried simply importing all online articles, ebooks and PDFs directly into Obsidian. However, the reading experience is just awful in comparison (especially on mobile), there is no native highlighting (I have a plugin under review that tries to fix that) and with no centralized server, fetching content across devices is painful. Web clipper is a step in the right direction but feels isolated. I can’t use it when I’m looking at a page through a web view on iPad for instance. I wish it was integrated with the mobile app and share sheet. Again, a limitation of OS-level integration / ecosystems.
I also want to be careful with how I “contaminate” my vault with other people’s ideas. I read way more than I write and I tend to read things that are new and challenging to my beliefs so if everything I read ended up in my Vault it could dilute my own writing and ideas. I rather have these explicitly marked or tagged so that they can be more easily identified.
After the shutdown of Omnivore I moved my reading sources to Readwise Reader and wish I had done it sooner. Now I consider using Readwise instead for my central hub of online reading highlights. The daily digest of highlights is exactly what I wanted to resurface ideas. They launched their Ghostreader tool relatively early and it’s quite powerful. That gives me confidence that I rather store my reading highlights there than in my own vault for now.
Workspaces
Each of these modes of interacting with Obsidian that I mentioned so far - journalling, reviewing, gardening, writing, active reading - require a different set of Obsidian tools, plugins and views. I once created dedicated workspaces for each but found that they are too static for my fluid way of working. I wish workspaces were deeper and more dynamically integrated into the context of what I’m working on so that it could automatically detect what views are needed now.
For example, I want the outline and backlinks view to be there when I’m browsing notes but not necessarily while writing a new note, where I prefer more of a barebones “focused” UI. What happens instead is that I start in one workspace, change my work mode, add/remove the views I need and now the workspace no longer looks like what it was supposed to do. Also workspaces have to be saved manually and force me to think ahead of what I might need rather than adapting and anticipating in real time to what I’m doing. It would help if it was possible to open multiple windows of Obsidian on my Mac each with a different workspace. Alas, that’s not possible.
So I ultimately moved back to just having one default workspace that clutters every view and tool I could ever need and that I can only use in a maximized window on my external 27" display and not on my laptop. I already abandoned the iPad/iPhone workflow due to the loading times and now sitting with my laptop display on the couch or on the train is another situation where I’m less includes to open Obsidian. The UI just never feels right for the task that I’m trying to achieve and either wastes screen space or hides things that would be useful to me.
That would be less of an issue if I only used Obsidian for one specific use case like journaling or note taking. But over time I have tried a lot of things at the same time in the same vault:
Plugins
As per my goal of being flexible and not locking myself into any particular format, I reduce plugins to a minimum and prefer those that work with the content I already have rather than forcing a new way of working with or structuring my vault upon me.
That being said, I’ve probably tried all of the top 100 Obsidian plugins at one point during a phase where I tried to make Obsidian “my operating system” for virtually everything I did as a freelancer (project management, calculating expenses/estimates, keeping coding snippets etc) and academic reading (Zotero, PDF highlights etc). I read the Obsidian Newsletter every week and installed the newest plugins via BRAT.
That phase also ended when I hit the limits of cross device syncing, performance degraded to the point of being no longer fun and I started working with clients that provide their own infrastructure for projects. It felt better to keep work away from my personal Vault.
I’m at the point where I prefer a dedicated paid tool or service over relying on an open source Obsidian plugin maintained by an idealistic volunteer for a year before it’s being abandoned.
When I look at the (community) plugins I use now, I mostly see little writing helpers and UI tools. Nothing exciting.
attachment-management
better-fn
calendar
darlal-switcher-plus
dataview
dbfolder
double-colon-conceal
editing-toolbar
graph-analysis
ibook
lazy-plugins
link-favicon
nldates-obsidian
note-refactor-obsidian
nuke-orphans
obsidian-auto-link-title
obsidian-contextual-typography
obsidian-footnotes
obsidian-git
obsidian-image-toolkit
obsidian-linter
obsidian-local-images
obsidian-media-db-plugin
obsidian-notes-from-template
obsidian-tasks-plugin
obsidian-tweaks
obsidian-version-history-diff
pane-relief
periodic-notes
quickadd
readwise-official
recent-files-obsidian
tag-wrangler
typing-assistant
vault-explorer
zotlit
Syncing
I already mentioned issues with iCloud syncing and I was really hoping things would improve with the latest iOS versions that keeps files downloaded. Unfortunately I still had several situations where I had an idea on the go, excitedly opened Obsidian mobile and after starring at the loading splash screen for 15 seconds and waiting for another 5 seconds for the UI to become responsive, the spark was gone. I now just reach for Apple Notes instead.
I consider trying Obsidian’s paid sync service to see if that improves things. But after I downscaled many of my more ambitious use cases, I ask myself, do I really want Obsidian on my mobile devices anymore? Or is it enough to open it on the weekends, run a couple importer scripts from
AI
The final point on this list and a main reason that prompted me to write this all today in the first place. I tried various of the popular AI plugins last year but none of them stuck with me or felt like they would achieve a benefit greater than rewriting an existing note or being more than a wrapper around the OpenAI API. If the entire note can be autocompleted by an AI today, then why even write it?
I use VSCode with Github Copilot and the Claude Sonnet model for work. So today I loaded my entire Vault into VSCode and asked copilot “who is X?” where X was the name of my father. And it replied “X is your father. You have written about it him at various times in your journal for example…” followed by actual clickable links to actual notes where I had written about my father. It also supports local indexing and search as well as bulk editing files according to my prompt.
I was deeply impressed that a tool that I already use like VSCode could just casually provide a better AI experience than anything I have witnessed with Obsidian plugins. It makes me reconsider what I use Obsidian for in the first place. I recently installed Sharp, a markdown editor/preview extension for VSCode. It has many flaws but it’s not too difficult to imagine a world in which Obsidian would be a plugin in my IDE rather than “the IDE for note taking”. And to be clear, it’s a strength of Obsidian that it ultimately is just “text files in a folder” that gives me the option to use another app that also just reads text files.
Final thoughts
After two and half years it feels like I have come full circle. I’m reminded of the advice I read early on, back when I started my vault, about “your vault is for your ideas not other people’s” and the distinction between collector and creator. Let alone the danger of becoming nerdsniped by tools and managing instead of writing. But it’s not that these wise words were true. It’s also a bit of resignation with the Obsidian ecosystem in the run up to AGI.
Moving forward, I want to focus my intentions with Obsidian more on what writing about those things that are not recorded, as gwern put it. I may not need 20 plugins or a database for that. I think I will spend more time on journaling instead and drafting ideas for future projects.
I will likely not import my reading highlights into Obsidian anymore but keep them in Readwise and think about a way to use their API or export tool to draw conclusions from them. If something inspires me, I can always just create a new note in Obsidian.