Search speed improvement ideas?

[maartin]

Searching thousands of notes is slow in Obsidian at this moment. Let’s help the developers by ideas how to speed it up.

[maartin]

I just migrated all my notes to Obsidian and have now 6000 notes. But not of these notes are equal. There are a couple of hundred important ones and 80% of my search is within this set of notes.

Here’s a rundown of how my search priorities are:

• by date: Newer notes rank higher and most of the notes I search are from within the last month
• by folder: I could move important notes into a special folder whenever I find some
• by tags: I have some topics that I work on every day, they are in a limited set of tags

So if limiting the search scope would speed it up, I would use it. Also for progressive results – show scoped results first, slowly searching the rest after.

[alexmikhalev]

In terms of algorithms for fast search is look for Levenstein automata and finite-state transducers, the current state of the art described here in great details. I would love Obsidian ship with embedded FST create (it’s possible to add rust into Electron via neon).
All use case above can be covered by automata and finite-state transducers.

The number of options is to choose is quite large, I will point out to few:

1. crates sonic-server, state of the art search backend (can be deployed on server/forum) as part of the premium sync feature
2. Search github ripgrep - local search (super fast)
3. Embeddable into websites https://endler.dev/2019/tinysearch/, I think it will handle electron app.

Avoid TFIDF type search engines until the need to search long documents content/descriptions.

Apologies, I had to remove links.
I know quite a bit about search, let’s roll discussion.

[tomsb]

In slow-performance-with-large-vaults/16633/9, it’s claimed that search should be fixed in 0.12, but I don’t see any mention of this in the changelog for 0.12 through 0.12.3, and, after updating to 0.12.3, I don’t see an improvement in search speed.

For comparison, if I use GNU find with this little pair of Bash scripts, I can find about 1300 hits on the word “the” in about eight seconds, in a Dropbox with 513446 files total and 2081 .md files. That’s with 38 lines of code on my part. I guess this wouldn’t be portable to Windows (Mac too? I don’t really know what is and isn’t included in posix-y shell), but it shows what should be possible.

Obsidian takes maybe 5 seconds to start returning results, and I don’t really expect it to continue to show 1300 results in the search sidebar. But, for a simple tag search with a couple hits, it might take a minute to return anything, and five to admit that the search is complete.

Wikilink-completion is right out.

[oleggera]

Hi @tomsb,

The issue I’ve posted in slow-performance-with-large-vaults/16633 was about linking to other documents from the text area, and it is fixed in 0.12.3 for me. There is a small lag, but it’s less than 1s and barely noticable.

I’ve just tried searching for the in the sidebar search, and it was almost instant. Maybe it’s cached somewhow. Searching for "data structure" was a lot slower and took 4-5 seconds.

[oleggera]

Actually, I’ve just downloaded Obsidian from the website (for Electron 12), and it seems to be a lot faster.

[jalur]

I am using Obsidian in Windows 10. My main vault is store in iCloud Drive for Windows. I notice that when I perform search in Obsidian, it is not instant and does take around 3-5 seconds. While the progress bar is still moving with some search result shown and I click on the result, it does not show me the file, as though, it is not able to open the file. Once the progress bar completes, then it works.

I am not sure if this is due to Obsidian or due to ‘cloud base drive’ such as iCloud. Is there anyone here who also use iCloud for Windows who can comment?

Update: I moved my vault to another local drive and perform my search. INSTANT!
It looks like putting the vault in iCloud is not a good idea. And yet, I need to have it in iCloud as I use Drafts as my entry point to capture text and export it to iCloud.

Ideas?

[chrsle]

Are you using Drafts on mobile? If so, then you could also directly export from Drafts mobile to Obsidian mobile and let Obsidian Sync take care of syncing.