Understanding Obsidian and how it works

Meta
1

How Obsidian works—and why it’s so powerful and fast

With a few tips on tapping into its Roam-like outlining capabilities

A recent reply by @pjeby (as PJ) to this blog post was the single best explanation of how Obsidian works and why it’s so powerful and fast. It made me even more convinced that I’ve chosen the right notes and PKM application—and even more grateful to silver and licat and to all the plugin developers.

It also includes some useful tips on making Obsidian work as an outlining app like Roam.

With permission, I’m reposting it here:

I think the public perception of Obsidian by Roamans is based on a combination of outdated information (some of the block indexing I’m describing has only existed for about 9 months) and the lack of highly-visible UI affordances for some (well, most) of the functionality.

For example, Obsidian has always had swap line up/down commands, and indent/outdent commands, not to mention fold-toggling commands that would, in combination, allow you to use it as an outliner.

But the swap up/down commands don’t have a default hotkey assignment, so you have to assign them yourself and know they exist in the first place, and you’d also have to know to enable the “fold indent” setting that lets lists expand and contract. (There’s also an outliner plugin that gives you even better ability to move things around without first selecting all the lines you want to move, but it’s also a relatively recent development.)

Similarly, though Obsidian has more-or-less always had the ability to link and embed blocks and track them in back references, there are many small creature comforts like showing reference counts inline or easily creating a block ID from the block (as opposed to linking to the block from elsewhere) that require a plugin to get a nice UI affordance.

So, if you wanted to capture all the same functionality at the UI level, you would probably need to install the Outliner, Copy Block Link, and perhaps the Zoom plugin as well, that lets you focus on a small part of a note.

(That being said, I would be surprised if adding any of those plugins substantially affected any of your benchmarks aside from startup time, as they mainly provide additional commands, menu items, and hotkeys in the editor.)

Given that the block and outline functions of the core app are basically hidden unless you know which settings to turn on and what commands to add hotkeys for, and that the only way by default to create a block link or embed is the autocompletion when you type (and you have to know to type [[note name#^^ to trigger it), it’s 100% understandable someone could walk away with the impression Obsidian lacks the same outlining and block reference capabilities as Roam. And in truth, to get something close to the “full” experience, you do need the plugins I mentioned. So it is in some sense fair to say “it’s not the same” because of this additional setup/learning curve.

However, the technical argument that “Obsidian’s not the same because it’s not a database” is just flat wrong. Obsidian does have a database, it’s in IndexedDB, and it indexes every paragraph and every line of every list along with block IDs and the hierarchies thereof, and it’s almost-instantly updated whenever the notes change, even if you edit them in another program! Which gives Obsidian a level of ability to integrate with other tools that’s unmatched by the competition.

There are probably other misconceptions as well, such as it not being immediately obvious that Obsidian updates links when you rename files, or that you can have it update links to sections as well by using the “rename this heading” command. There are tons of tools and commands like that, that you’ll be unlikely to discover in the first hour or two of use, and/or were added in the last six months. (If you aren’t involved in the community or follow the releases, you won’t find out a lot about this stuff.)

Anyway, to answer your question about whether there are “real differences” between how Obsidian and Roam work, well absolutely there are. Roam doesn’t have note files stored as markdown, after all. Roam is DB-centric, Obsidian is text-centric.

That is, Obsidian creates its database as a background-updated eventually-consistent index of the text. This is what makes it faster: since the text is primary, it is available and editable right away. While you’re using it, it updates its index in the background, so you are never waiting on the database to do something, it just takes a second or two before secondary views (like tags, backlinks, outline, etc.) will be caught up with whatever you typed.

(And you generally won’t notice that because if you’re typing you’re not looking at things that show you what you changed, you’ll be looking at those things later when you’ve moved to a different note!)

Thus, you could say that Obsidian gets its performance by cheating: it’s cleverly designed so that nearly all the hard work takes place at times and in places where you can’t benchmark it doing so. :wink:

Finally, the inner workings I’ve been describing are actually documented on Github, at obsidian-api/obsidian.d.ts at master · obsidianmd/obsidian-api · GitHub – the MetadataCache and its CachedMetadata records describe precisely what information is indexed and how, so that plugins can use it.

27 Likes
2

I’d add the Drag and Drop Blocks plugin to this list.

As part of this, it’s helpful to understand how to use headers and the value of the core Outline plugin with its recent feature update (for managing file structure rather than being related to outlining).

I suspect only a minority of users use the full power of a markdown file being focused instead on short notes in separate files.

4 Likes
3

I’d add the Drag and Drop Blocks plugin to this list.

I can’t find that title in the community plug-ins. Am I doing something wrong, or is it not yet available to the community?

Thanks

Angel

5

It’s probably still in development or is still waiting to be approved.

3 Likes
6

It’s installable through BRAT; still in beta.
Another one that doesn’t work in Live Preview yet. But is being worked on.

2 Likes
7

I think the main difference between Roam and Obsidian when it comes to backlinks and blocks, is that Roam retains the full context when working with backlinks. In Roam, you can work entirely from the ‘backlinks’ section of a note, seeing all relevant information without needing to actually navigate to the other note. When people talk about ‘block referencing’ in Roam I think this block-level structure is what they are really getting at - the outliner structure is preserved and provides context wherever that block is referenced. Also I think that this post overstates the level of similarity between Obsidian and Roam when it comes to outlining - even with all of these plugins and settings set up, I don’t think the experience in Obsidian will be as fluid when it comes to block referencing and seeing all referenced blocks inline.

5 Likes
8

Thanks, @Synchronicity and @Dor

Angel

9

I use a lot of atomic notes. What is the full power of a longer Mardown file?

2 Likes
10

One thing that intrigues me about Roam is the ability to have a block live in two places at once. Add a block to a daily note and have the same block also exist inside a note within a particular project area, for example. I have not found a satisfactory way to do that in Obsidian. On the other hand, Obsidian has, from my perspective, so many benefits, and Roam has so many drawbacks (again from my perspective) that I just live with it.

This is an extremely valuable topic for me. Thank you for it.

4 Likes
11

This is an important point, and one I failed to raise in my original references to Roam. I have been an Obsidian user for a year or so now, and quite frankly I’ve forgotten what I used to like about Roam :smile:.

So if I understand correctly, there is an aspect of outlining that Obsidian doesn’t necessarily replicate (yet?), which is parentage, i.e. if a block is indented beneath another, it is a child of the other block, and in references you can use that hierarchy to traverse the note structure in a useful way. I do think Obsidian has hierarchy in the sense of header levels, but outliner hierarchy is potentially different.

I haven’t tried the above recommended plugins and I’m not enough of a fan of outlining to bother, so perhaps this hierarchy and/or the navigation of it (from e.g. references) is part of what they add (or maybe it already works differently than I think it does). If not, however, then I do think this may be one aspect of the performance picture not taken into account in the discussion on the original blog post.

In any case I may well (still) be wrong. I really appreciate @pjeby 's original explanations there regardless!

Wouldn’t this just be transcluding? Like this: Link to blocks - Obsidian Help

12

Yes, transcluding is a partial solution — but I think it’s just partial. I can’t edit the transcluded block inside the note containing the transclusion; I have to go back to the original note for that.

Or at least that was the behavior last time I checked, which was some months ago.

4 Likes
13

I believe that’s still the case. However I suspect that with ongoing improvements to Live View, eventually you will probably be able to edit them directly in-place.

3 Likes
14

Also, with LP it’s now a lot easier to both 1) see the transcluded block and 2) jump to the original note (with either a click or a keystroke).

But it’d be nice if the jump actually went directly to the block instead of just highlighting it. That’s fine if you’re mousing, but not so good for keyboarding.

1 Like
15

If you imagine every block as a separate note, that’s what Roam means when it says the block is the core element.

In Obsidian, if I have a paragraph in Note A, I can transclude that paragraph into Note B. If I delete Note A, I also delete the paragraph. In Note B, the translcusion is broken. In Roam world, Note A isn’t one thing. It’s an outline of blocks. I can delete any of the blocks without affecting the other blocks.

You can get this same effect in Obsidian if your notes were super atomic: every heading or paragraph or list item was a separate note and larger “notes” were created by transcluding a bunch of other notes together.

Most people don’t need Roam’s super-atomicity. For those that do, Obsidian’s transclusion is too clunky, not fine-grained enough.

(Begs the question, what kind of workflow really benefits from that level of atomicity?)

3 Likes
16

So if I understand correctly, there is an aspect of outlining that Obsidian doesn’t necessarily replicate (yet?), which is parentage, i.e. if a block is indented beneath another, it is a child of the other block, and in references you can use that hierarchy to traverse the note structure in a useful way. I do think Obsidian has hierarchy in the sense of header levels, but outliner hierarchy is potentially different.

Here is example, showing what I mean about seeing the context and editing block references inline:

roam-example
roam-example837×815 58.6 KB

You can see that I can see all references to the block underneath the referenced block itself, and also which header they are under. I can also choose to unfold any blocks nested underneath, and edit them in place without navigating to any other note. The same goes for page references. What this means is that I can potentially see and edit all relevant information for a topic, without navigating away from that single note. In other words, a block or page reference is not just like a little tag leading me back to another relevant page - instead it fully includes that information wherever I need it, so that I can basically see a kind of cross-section of my notes from wherever I am. In Roam you could spend an entire session working only from the backlinks section or from the block references underneath a block, never editing or looking at the actual content of the page you are on.

Of course there are benefits also to the way Obsidian works, and I think in some ways having a page-level granularity actually makes us break down ideas more carefully. However, I think it is misleading for the OP to say that Obsidian has “all the same functionality at the UI level” that Roam has.

5 Likes
17

This is a very helpful discussion for those who have been using Roam and need to understand the overlapping concepts, alternate concepts, and very different approaches Obsidian takes. Thank you for your explanation.

1 Like
18

saw a really great idea on reddit/obsidianMD: treat zettels as holders for individual “blocks” (however defined)—if you create every paragraph (for example) as a separate zettle, they can all be linked to individually – and seen as if within – any number of documents, so that a change to one such block is done only once, and immediately reflected everywhere else that block is included/transcluded (whether by reference or tags)

the plugin keystrokes needed for this can be automated: immediately upon creating a new paragraph/block, a new zettel is created with a transclusion at the top from the immediately preceding paragraph/block/zettel into that new one, so – at any point in the process of creating or reviewing the whole set – every single individual zettel will always appear to be unified into a single document, just like Roam shows

1 Like