Help unscrambling a vault once thought hopeless

[I-d-as]

What I’m trying to do

I am trying to sort out a vault that has proven impossible to tame as of yet. I created it as I learned Obsidian and it contains lots of notes organized in many different ways.

Things I have tried

If anyone has any suggestions of how to get a handle on a vault like this I would appreciate it.

Thanks

[Pixionus]

I feel like this is more a call or working session sort of thing, but I’ll see what I can offer in conversation. Let me start by asking you, is there a given method that you preferred most out of the many you have tried? If so, what prevented you from using it as your primary?

[I-d-as]

I preferred trying to atomize things and it worked for certain types of Notes. I like the fact that it was easy to see what I had by looking in a folder. I also liked that it forced me to come up with short names to describe it. However, that was also the downside because of the limitation of file name length.

I was unable to stick with this system because it did not allow me to relate the ideas very well. I also came to the realization that many of the ideas I thought were atomic were actually ideas that related things together. So I went down the rabbit hole trying to work with properties and tags, etc. I used index Notes or maps of content, but was unable to consistently continue using them as I often would want to reframe things in a different way. So while doing that I never got rid of the old index notes that were framing them originally. I never kept clearing out what was left from the old systems.

I Began Relying heavily on voice recordings that I would transcribe. These would work well because I could relate many ideas together and often in the process of describing them I would come across an interesting twist, and it would be recorded. I would manually transcribe these recordings and sometimes break those out into outlines. But rather than pull ideas or sentences into notes where they were related, I opted to often keep them in place in the vicinity of the semi related ideas that were described during that recording session.

I used template scripts to extract content into other notes, leaving behind the original content as an embed from the note where it gets extracted. But also had it so that in the note where it was extracted to, there is a link back to the block where it was extracted from. Extract to specified heading with Note Composer - #13 by I-d-as

I also experimented with embedding visual notes using Excalidraw Systems like those that Zsolt Viczian shows in his visual thinking workshop. I also used all types of dataview systems, including some wild ones designed by people here on the forum Dataviewjs query that separates 2nd level of nested tags into headers - #5 by holroy

I could go on and on about the many different ways I’ve had organized my notes, and in fact, I believe that this is actually a very helpful process. The hardest part is unraveling all that you’ve created after the fact. It can be very disheartening

[Pixionus]

What is the most important thing for you in your notes? Relationships and patters? (that’s what it sounded like you were focussed on with your attempts)

To me it sounds like your methods are all input sources with high WiP cost. Would you say that’s the case? It almost seems like the thing you were searching for was how best to input into your db and then reaching around to build your vault’s system around the input method.

[I-d-as]

Thanks for the thoughtful and helpful question! Relationships and patterns used to be the main purpose of the vault. Considering the purpose of the vault was as staging ground for ideas to be assembled into parts that could be used in pieces of images within a film, it made sense until it didn’t. At a certain point it feels like the goal should’ve changed from an anything goes approach into a let the best parts rise to the top approach. Constantly hybridizing everything with everything else results in a very connected, but very tangled and at certain places rigid system.

I would agree with what you said about the high cost of effort to keep things current as they are works in progress with my approach. But, since the system wasn’t managed properly, it was impossible to keep things current so I would just try my best until I got to a point where it was easier to just create a note and re-explain many things that are explained elsewhere in the vault just so that I could describe a new idea that would tie them together or extend from them. By repeating this cycle and not correcting the bad habits, it was inevitable to end up with a vault that doesn’t even know itself, at least in most of the places.

In many ways, I wasn’t trying to figure out how to best input into the vault, but instead, I was trying to figure out how to process everything that already existed. If you look at the initial post for this thread, you will see that it was edited and if you click on that and see what it was edited from you will realize the point I am at with this vault. It is filled with gems, but it is also a complete mess. There were times in the past where I tried to bundle everything together and make sense of it, but then the effort failed so I re-added some of the pieces that had already been bundled. There are issues with almost anything you could imagine. I guess my question is how do you start over with what you have when it’s a mess without actually starting over?

[Pixionus]

Not a problem. It’s what I look for in responses, so it’s what I offer. Sorry still composing my reply. Sometimes, I compose a draft and let it ferment before I share it. I think that is sometimes the cost of a thoughtful response, and perhaps why you don’t see many in the quick results age.

[gino_m]

I’ve seen some of your FR’s and posts in the past and now some new plugin that does what you needed (Quadro) but I myself am reluctant to switch to using any of what it offers, yet.

[I-d-as]

Thank you so much! This is very promising! I am going to download and try out the preconfigured example vault and see how it works. I really appreciate that you really considered what I am trying to do and provided a thoughtful suggestion that addresses my problems. Being part of this forum has been invaluable and inspiring. I’ll return with my findings, hopefully good, on this plugin. I feel hopeful.

[Pixionus]

Oh wow! This Quadro is really interesting! I think I’m probably going to end up working this or something like it into my workflows, so thank you for sharing!

Sorry for the delay. Life got in the way. Hopefully, Quadro turns out to be your solution .
So as I said before, I feel it’s hard to grok the situation and your preferences in text so that I can provide useful feedback, but I was trying. In the end, I decided to just give my use-case as an example, though some of it may or may not work for you, and so I went back and forth on if it was constructive.

Just in case it’s helpful here’s that long update I only just finished editing to post:

Personally, I subscribe to a let the best parts rise to the top approach, so if we’re thinking the same thing, I endorse that strategy.
I don’t worry too much about the input method as long as things end in text.

Here are some strategies that I use that I found helpful reprocessing my own vault:

• My early vault notes exist in a mess of directories and formats so they are relatively undecipherable so I created a new directory named 1.0 for the mess, and one named 2.0 for the newer greener pasture. 2.0 is quickly becoming my actual vault.
• I’ll compose new notes in 2.0 and if I link back to or revisit a note in 1.0, I’ll add a maturity label and some tags and move it to 2.0. I’ll even sometimes just spend some time tending my vault (I think of it as weeding). Adding the emoji tags for maturity and seeing real progress as the new tag and file systems develop, just kind of gamifies it enough that I enjoy ‘weeding’ from time to time.
• I label notes with one of 3 grades of maturity. The idea is that a note progresses from a seedling () → to budding () → to evergreen (). I don’t recall where I got this system, but it really fits with the evolution of a note for me and comes in handy, so it’s become the heart of how I have reprocessed my old mess into a cohesive and reference-able, more Zettelkasten, “second brain”.
  Maturity Definitions

  1. Seedling / - A rough note with no formatting and maybe missing links and the like to tie it to ideas that are adjacent or overlapping… These can be audio or video recordings that I still need to process into a note. I make a point to tag Seedlings when I actually make them but otherwise don’t worry too much about the content. Basically, a seedling that with some attention can grow into something greater.
  2. Budding / - A note that has enough polish as to be useable by me. It generally has headings and I’ve started tying it into other notes. I usually re-label to Budding the second time I visit a Seedling as a result of a search or to add information. At this point I’ll spend a minute or two of attention on it, adding headings and links, but not caring if they are neat / easy on the eyes or something I’m actually completely happy with.
  3. Evergreen / - A note I’d actually feel is of a quality that I might post online. If I’ve ended up linking back to a note repeatedly, revisited and found useful, or for one reason or another just thought it was really worth cleaning up, I’ll spend about 5-10 min getting the note to the point that it’s got some polish. Notes for projects that I am actively working on tend to become Evergreen the fastest because they have the traffic, and the neatness contributes to the work I’m doing. Evergreen notes are always evolving. Few notes are ever “done”, and if that ever happens, maybe I’ll come up with a fourth emoji to signify something that isn’t growing anymore, though I almost feel like at that point a might fit
• I use one of the following Dataview metadata at the top of a note to label its maturity. The syntax means I just have an emoji as the first line. This clearly and pleasantly marks the notes maturity for when I next visit without all the metadata syntax and still makes it more actively reference-able in Dataview

  (Note Maturity :: #🌱)
  (Note Maturity :: #🌿)
  (Note Maturity :: #🌲)
  

• When working on a new note, If something feels familiar, I’ll search the vault with line:() for words that are pertinent (this finds tags as well) until I find where it’s come up before, then tag the note , move it to 2.0, and tie the notes together. Even a very loose tie of a [[wikilink]] link between the two notes is enough for a .
• I also try to follow an “atomic” strategy of note-taking when reprocessing notes to a higher maturity. I don’t stress too much about having this perfect and if I feel I want a different phrasing in the new note, I’ll just write it in how I like it but leave a wikilink to the atomic section. It’s not 1 or 0 for me but some softer mix as is appropriate. It’s becoming a large part of achieving status, but a lot of what I record is technical and YYMV.
  How I use 'Atomic' Notes without extracting content

  I like that splitting notes into their component parts (where each note is a specific idea or input, with descression) means that if I need to pull in an explanation from another note or a section I can use the ![[fileName#section]] syntax to pull in that section or bite-size note directly into another note. I feel this makes use of a good inherited source of truth strategy in my notes, so they are easier to tend to and more compact.

• The next step for me, now that the most important notes from 1.0 have risen to the top, is to set up something that will randomly bring up Seedlings when I open Obsidian (there’s a plugin for this, but there are a number of ways you can do it. I think I’ll just use Dataview on my daily notes to list notes with and randomly pick one from that list when I feel I have time).

Since I first started writing this note, my vault got to a point of maturity/stability where I just felt 2.0 was redundant because the system really worked. I decided I could just move all the directories out of 2.0 into my vault’s root directory and delete the now empty and superfluous 2.0. There are still a bunch of notes in 1.0, but that directory is dwindling every day. Some week soon, it’ll be gone, and all my vault will feel in a much less “scrambled” state.

There are other things I have been doing, but I feel this strategy with normal tag connections, dataview, and wikilinks creates a lot of the relational data map without index notes or properties, etc. Keeping it simple has been key to this process’s success. I think it can be applied to your desperate note types. It also breaks the whole process down to basically:

• Rebuilding the file structure off of what you’ve learned, but without the clutter of the old one because it’s hidden away
• Adding tags at the top of a page (I have a line under the H1 and will just turn important words into tags in line).
• Following a maturity grade. It gamifies things a bit and makes the vault less scrambled.

If this is helpful at all, please let me know. I am working on a basically fool proof backup strategy that I’d be happy to share as well (since you mentioned that in your original edit), but I feel your current 1 note with a printout strategy would be untenable in the long run as the vault grows. For me, my vault has become enormous and I need to remain agile with it and every day I revisit ideas and improve things, as is the Zettelkasten way.

[Pixionus]

If you like, I’d be happy to have a zoom call or w/e, and we can discuss things directly; Just DM me if you think that’d be constructive

[I-d-as]

This is incredibly generous. Thank you for sharing all of this. It’s amazing how when a technique hasn’t worked for me in the past for whatever reason, I continue to shy away from using it without clearly thinking it through. It’s almost as if I have multiple individuals in my mind that confer and one or two of them are able to tangle the whole process up leading me to pull back from what I know to be a perfectly reasonable and helpful approach. I have this problem with the maturity tags which I over complicated in the past and have yet to reimplement due to disgust with the time I wasted, thus my continued aversion to trying again.

It is very overwhelming to feel the need to categorize everything. But it is less overwhelming when you realize that the work you do will be helpful in the interim. The notes and directories within the 2.0 or eventually the root are isolated from 1.0 and the helpful seeds, budding, and evergreen can be easily accessed via a query. This is something easy to overlook when you’re looking at a mountain of work. Thanks for reminding me of this truth. I can’t say it’s the first or even the 2nd or 3rd time I’ve re-realized this, but hopefully it will be the last.

I love the idea of weeding and even the emojis to match the theme. The problem I’ve had is that I enjoy coming up with these complex allegorical setups so much that it takes on a life of its own. It’s so easy to get carried away. I will definitely be enforcing some simple maturity grade like you have. My next step will likely involve some sort of preprocessing of the mess. But before that I will definitely be checking out Quadro. While in the past I opted for actually extracting segments of the text into other locations, I am finally accepting the reality that I will never be able to turn a massive knot into a comfortable woven sweater, and I should not try, so embedding a few threads here and there can be an efficient alternative.

So, if I can slowly but surely call out ideas by embedding them elsewhere in organized notes, I might begin to get a better grasp of what I have. The issue I have becomes difficult because single notes or ideas have the power to overturn the logic of my entire plan for the end result. Since it is art and I have been composing ideas within the same realm for years now, they all relate. But some of them take the whole thing and add another layer and perspective to parts of it which in turn requires many other adjustments. It is because of this reality and my decision not to let myself frame the whole thing first before fitting the pieces within, that I continue to allow reframing and understandably am skeptical about fully adopting a technique that works with the current plan. If I ever did this process again, I would have a working plan that would force me to never create ideas in isolation unless I was OK with them being left behind at the end.

Anyways, I have some thinking to do and a plugin to check out then I will surely check back. Your kindness means a lot.

[gino_m]

I am also somewhat overcomplicating my life with Obsidian but I have managed to stay true to a few principles:

• Use A-Z folders so many thousands of files and their attachments won’t be dumped into a single folder. On my Digital Garden published online, these A-Z folders are nicely nested in my sidebar, making it easy for anyone to navigate the content. No other folders are used (apart from meta folders housing templates, queries, scripts, and others for canvases, Excalidraw drawings, etc.). What people usually use folders and subfolders for, in my vault they are actual notes with headings 2, 3, etc. – Which I can make more of as I revisit my notes. Tags and other properties I use for Dataview querying purposes and are not fundamentally important. They are more like crutches for an able walker.
• I use backlinks extensively as one would do on Wikipedia. (Lately I have been giving myself extra jobs to find the relevant paragraph, add a block ID to it with the Carry Forward plugin (Quadro seems to be an upgrade on this) and edit the formerly plain backlink so I or any of my prospective readers will be taken straight to the relevant block/section.)
• I name notes as I’d do for a Wikipedia entry – short but for more complex ideas some descriptive name is in order. If the name was not exactly bang-on, I can rename it within Obsidian or add 3-4 aliases with some proper rephrasing. I write bilingually, so I can add English and native aliases (more work). If during writing I’d want to refer to any topic, with the Various Complements plugin, I get pop-ups for these notes and can include the links easily in my sentences. One caveat with this plugin is that matching happens in the order of words so Greek principles of life will be found of you type Greek pr but won’t be offered if you type principles of life in Gr, for instance. A quick hopping out to the QuickSwitcher can help find the proper wording or with QuickSwitcher plus (or ++?) one can filter for headings if the topic was put under a heading 2 somewhere.
• I use Heading 1 titling matching my Title property. This is handy for finding any notes in the editor with the core Obsidian [[## method (if e.g. Various Complements no longer works fast for a larger vault or on a less capable hand-held device or I want to cut down on third party plugins), without having to use the Quick Switchers (core or any third-party upgrade).
  So one can find any heading level which again should be properly phrased.
  One can even put this Templater scriptlet on the mobile toolbar or assign a hot key to it:

<%*
const editor = app.workspace.activeLeaf.view.editor

// The text selected before running the snippet
const selection = editor.getSelection() || ''

// Prepend the selected text with [[##
const link = `[[##${selection}`

// Replace the selection
editor.replaceSelection(link)
_%>

• Adding these aliases is important so one would avoid creating new notes with much the same content in the vault. But repetition is not entirely bad because that’s how you teach yourself. The problem comes if the newly created note contains some faulty argument or lack of references that the older one had and you don’t find or recall the existence of the older note.
• The naming convention should follow your own logic and arguments. Science likes to name things based on its own (political) agenda. But if you want address the public with your writing, you need to use the scientific jargon otherwise there is no common set of dialectics. This happens on an individual level as well: as you grow, your skill set evolves and you no longer find certain things important or you use terms foreign to you before more easily. I am/was lucky because I read the important books first and had a good fundament. Also my native language is more logical (in Hungarian we focus on big first and narrow down to the details, so we don’t start with forenames, house numbers or days of the week; the stars come first and the mud you’re bogged down in you will snap out of – today, life wants you to be boxed up and be bogged down, wallow in materialism and not to mention other things). These conflicts or bipolared-ness can actually help with or be the agent or driving force behind the sifting process as you try to alienate from you the alien (there’s plenty of it going around today) and keep the familar close. For example, a pyramid per se is not something you see as a tourist (they want you to be a stupid consumer/spectator always), not a burying ground (we hear this all the time), not a giant transformator (new-age esoterism), but a symbol of many things: a star mapped down on Earth, with its four sides corresponding to the four seasons, the tip representating the higher power or its seat in the Pole Star/Summer solstice, etc. One needs to focus on real things, and it’s not easy today at all.

There are some plugins I have known or recently noticed that can help refactoring or with janitor jobs:

• File Cooker
• Multi Properties
• Linter (see YAML section and docs on how to use it)
• Quadro can be interesting but I’d need to rewrite some parts of it to make it work for me and my scripting skills are close to zero.
  • By janitor jobs I mean you assign new (temporary) tags or properties or move notes to temporary folders in another kind of sifting process.

I don’t hard-code emojis but use the Supercharged links plugin with the Style settings companion.
Also for embedding blockquoted text, I need to remove the double lines on the left and my CSS skills are not perfect.
So with embedding galore, come new problems.

[I-d-as]

Wow! This is beautiful. I’d love to check out your online digital garden. So many of these techniques and explanations speak to me. Fewer notes is exactly the solution I needed, and I believe your method of heavily using headings the way others use folders will get me there. I guess you save a lot of time not dealing with properties for the headings either. If a specific heading requires links or backlinks, you simply create them. I guess what has stopped me from adopting this method in the past was the inability to sort backlinks by the heading they come from. This seems like a minor inconvenience in hindsight.

Thanks for all the suggestions! I can’t wait to really dig in and use these recommendations to help me get a handle on the lost ideas that keep me up at night. I am so glad I created this topic. These answers have brought be back. I was at a point where something lazy and drastic felt imminent. It would have been a shame to not do it right after all this time. I really appreciate the unique thoughts expressed here. Writing really is an art and Obsidian and plugins provide so many inspirational tools.

I’m sure I will return and further comment on everything here once I have a chance to fully comprehend and experiment with yet another life changing response. It’s such a gift to be part of this community.