Hi, I did a very basic and rudimentary test to see the number of notes Obsidian can handle in a Vault. I created 2,80,986 text notes in a folder and then tried to open it in Obsidian to check if it could handle it. I found that its takes several minutes for Obsidian to process the folders and list them, and then several more minutes to process the list of files in a folder and open the drop-down list. Then I tried to open one text file which did not show any matter, although I had inserted 2000 words of dummy text to start with.
I wonder if anyone else has done a test like this or know how many files Obsidian can comfortably handle in one Vault. Please do share.
Thanks for your comments. You could be right. My curiosity led me to test this. Evernote has a limit of 1,00,000 notes for individual pro users. I don’t know about Notion. As for computer specs, mine is a Windows 10 Laptop, i5, 10 Gen which is only a few months old so I think that’s quite good for normal use. Logically, given that the files are saved locally, the number of files should not really matter, but maybe Obsidian has a capacity to process and present the files. Saving files in separate Vaults may not be an idea because the linking may not work. Eager to know if anyone has ideas, thoughts and information about this.
We stress tested against 20k Markdown files (found here: https://github.com/Zettelkasten-Method/10000-markdown-files), and it works okay on a relatively fast desktop computer. We’ve done extensive optimization on file explorer and search, but the initial loading and processing could take some time for sure.
Haven’t seen real-world cases that would require more, unless you want to for example display Wikipedia inside Obsidian.
Would it be possible to display a downloaded copy of Wikipedia using obsidian? I have some of Wikipedia downloaded on kiwix on my phone, but it’d be fun to play around with inside obsidian on my pc.
Someone was playing around with this idea on Discord yesterday. To get started I think you need to at least convert the MediaWiki HTML pages to Markdown files. There seems to be some projects on GitHub for this purpose.
Just some perspective, if you create 20 notes per day (pretty good for note taking) it will take you 14049 days to get to 280986 notes, or 18.25 years non-stop.
When that happens I’m sure there are ways to get this working faster. Like moving things around to separate vaults for totally different topics.
That is indeed an interesting perspective. Thanks for sharing it.
Just to add to the context, my process of note-taking differs slightly as I prefer to take shorter notes (like in a note-card model) which are a mix of self-written notes, book notes & quotes, clippings from the web articles etc all of which can actually add up to 150 to 200 notes per day.
Calculating say at 200 notes per day I could touch 2,80,986 in 1404 days that is in less than 4 years or earlier. (Just to add: I am also looking to transfer my old notes to Obsidian) For the past few years, I have been using Evernote for this purpose, but realize that transferring notes from Evernote when it gets clogged up is a difficult and manual task. Now, I was experimenting with Notion, that’s when Obsidian came around.
What I liked about Obsidian is the it (1) allows me to keep the files locally in text format, (2) has back-linking & graph view along with Tags etc which is definitely an icing on the cake, (3) allows me to view and work with multiple panes which enables me to see different notes at the same time, (4) feels much lighter.
My fear is that if I load it too much I should not mess it up. If there is a certain limit on the number of files that Obsidian can comfortably handle then, if it is known, I might as well create separate vaults to begin with, because back-links and tags won’t work once I try to use my text reader to access these notes.
Interesting! Thanks for sharing. Well, there are file system issues even if I think 20k should be OK for most modern file systems, will be lesser caching support but one could live without this and just burn laptop batteries.
But the bigger issue is that most UI widgets for file viewing such as listing file names were never designed for 20k amount of file names. This as it’s not common amongst normal users to have such an amount of files in a directory, such files are mostly spread around sub-directories.
So that’s where the optimization has to get involved with. Maybe just ignore certain amount of files, or have pre-filtering of file names so only some show up. Search also has to be optimized.
Another option, later, is to make sub-directories, let’s say two letter names for the start of the file name and move files into those plus fix any internal links (aa, ab, ac… up to zz), that should spread out the files. Need to then also map the directory listing in a way so these internal files are not directly shown…
I am intrigued by your use case, 200 notes per day seems unsustainable over any significant period to me (20 notes/hr=1 note every 3 minutes, 10 hrs per day, every day). What kind of info is on these notes?
However, I do see the need for wanting to know where Obsidian maxes out. If you intend to import a lot of pre-existing notes, or notes collections from other people, then totals like 280k notes seem very realistic and the limits need to be understood before you head down this path.
One idea: Maybe you could refactor your note structure, given that Obsidian now allows section-level links. So what used to be 10 notes is now 1 note with 10 sections, each is still directly linkable from elsewhere.
All that said, at levels like 280k, this seems to go beyond personal knowledge management by today’s standards, and a different level of sophistication of method and tooling may be merited.
What you should do is to try to get an estimate of the formula for space/time use! That is, see the space/time use when there 0, 1K, 2K, 4K, 8K etc. files and do curve fitting. You can also try different average size notes. If you can package this up in a test script, that will help the developers as well!
One other thing to note is that if you are writing 300K notes, when are you going to have time to read/review them and make connections? Also note that with shorter notes you will need many more clicks to visit them. At least until the time Obsidian allows editing embedded files!
When the app starts up, are you trying to process all files in the root of the vault(or even the entire vault recursively) at once? The important thing is start up. I might suggest chunking this work, maybe first loading any open notes, then enough to fill the file tree, then process chunks of a reasonable size (1000?) at a time. The important thing is that you don’t block the normal functioning of the app. Maybe you could shell out to another process (or multiple) for this processing if you find more than 1000 file descriptors in the vault?
Generally, laziness solves most of these problems with startup time.
@ksandvik Thank you for sharing your thoughts and observations. If I am getting your point right, I completely agree with you, that normal work laptop will not support so many files in a single folder/directory. Even if I am able to open a file in a folder containing 280k files today, it may not be a case a few months later when my laptop is order. I completely agree that having 280k files a single folder is not normal at all. As I mentioned, I was just trying a very rudimentary experiment to see how Obsidian handles so many files. Obsidian don’t allow linkages between multiple Vaults (wish it did, then my concern would be sorted). I also agree with you point about optimization and have been thinking about it myself. The question is how. I was thinking about creating separate Vaults based on subjects, but that limits my ability to create inter-subject linkages using Obsidian.
Would love if you could please elaborate your thoughts about optimization. For instance, how can I have Obsidian ignore certain files or pre-filter files based on file names. I admit that I haven’t experimented with creating sub-folders/directories for the dummy text files. I posted all the 280k files in a single folder. I will try an experiment with the sub-folder model that you allude to and see how it works. If that works then the letter based sub-directory model which you suggest should also work perfectly allowing me to create linkages between multiple notes.
@randulfr It does sound intriguing when you calculate the number of notes by the minute. But honestly, it is more simple than that. Over the years, I have collected notes in multiple formats - both on paper and digital (including Pocket highlights, Weava highlighter, Liner, Instapaper, Evernote, OneNote, Notion, Zettlr and others). Apart from notes, these also include a variety of highlights, clips, comments, etc. In fact, I ended up using so many applications because I was always on the look-out for the one which is best suited to be a go-to Dashboard for notes. As mentioned in one of my earlier replies to @ksandvik there are four reasons why I liked Obsidian and wanted to look at it as this over-arching application for my notes.
As you mentioned, I wanted to find out the level at which Obsidian will max out, hence did this kind of a “stress” test. I must admit that it was possible only because, Obsidian uses local storage (for instance, uploading so many to Evernote would simply not be possible).
Thank you very much for your tip to refactor my note structure and merging my notes. It is one of the options that I have been thinking about but was feeling lazy that I will have to go back to each of my notes. Moreover, I did not know that Obsidian allows section level links. So it is something that I will definitely try.
I also agree that this may not be the perfect tool for a very large number of notes. Wonder if there are some software/applications like this Forum which can be used locally for note-taking. Need to search that.
@scmwiz Thank you for your comments. Although I did not quite get what you mean by “curve fitting” and “packing a test script”, I think I get your tip that I must make an estimate of of the time, space and size at various note counts (1k, 2k etc.)
As for your second point, as mentioned in my replies to @ksandvik and @randulfr I have a lot of notes from the past saved in various places. Although linkages are new for me, I have been tagging my notes (like in Pocket or Evernote) and saving into various folders/notebooks like in Instapaper or OneNote. As you may be aware, some allow tags, some folders and some both. Obsidian got me excited because it allows me to save all my notes in my local drive and also use Tags, Folders and Linkages (couldn’t ask for more!).
I agree shorter notes take more clicks, but on the other hand is easy to grasp and with Obsidian’s feature to split the screen into multiple note-panes it feels all the more better. Anyway, as mentioned in my earlier replies, finding a system to merge notes in something that I will have to think about going forward.
Thanks for the suggestion. As for me, I had all the dummy files in the root. As mentioned, I haven’t tried the sub-folder structure, which I will be testing. Will keep your tip in mind when I do that.
If you plot points on a 2D graph of (x-axis size, y-axis time or space), curve fitting is finding a smooth mathematical function that closely fits those points. See https://en.wikipedia.org/wiki/Curve_fitting
The idea is then you can estimate space/time use for larger numbers without actually testing them.
One more point about many small files is that (as I think @ksandvik pointed out) the file explorer won’t work well. You’d have to use a search criteria to reduce the number of files shown on the graph or file explorer (sort of how search works on MacOS finder). The current search window shows some content of files but when you have too many files that may not work well either.