Search very slow (>1 minute)

Bug graveyard
, ,
1

Searching my vault of 12k notes can take more than 1 minute. Searching the same notes with Visual Studio Code takes no more than 2 seconds.

Steps to reproduce

  1. Do a search that combines a tag (tag:...) with a word (hello) or quoted string (such as "hello()").
  2. If this specific search isn’t done earlier, it can take a minute before the first result shows.

Example where search took 1:40 minutes: video recording.

Expected result

Search should be much faster, definitely not more than 10 seconds.

Actual result

Search easily can take a minute.

Environment

  • Operating system: Windows 10 64-bit.
  • Computer: Intel i5-10400 CPU (6 cores, 12 threads) @ 2.9Ghz with 24GB RAM and a 500GB SSD drive.
  • Obsidian version: 0.12.3

Additional information

  • There are no errors in the Developer Tools’ console.
  • I don’t use community plugins or themes.
  • I couldn’t find Obsidian log files on my computer.
  • I’ve been using this vault on this computer every day for about 3 weeks. So I believe the search index should be cached by now.

(I searched the forum but didn’t find a related report. If I overlooked something, let me know.)

1 Like
2

How long does it take to search just for the tag?

3

Thanks. About 1-2 seconds, even for tags I haven’t searched yet (so un-cached if it works like that).

4

what if search just "text.align_left"

5

I did a quick test with VS code. The speed is comparable.
However, what happens is that both Obsidian and VScode are slow the first time you search (while the files are loaded in memory) and then they are fast.

6

Thanks for looking into it. Is there something I can do to fix it on my end?

Or is there a way for to provide more information, in case the issue is caused by something that wasn’t in your test vault?

7

I don’t think thare is an issue here. Both obsidian and vscode are slow the first time you search in your vault.
After the first seach, the files are in memory and the search is faster.

8

Will be fixed on 0.12.5

9

Sorry to revive a bug labeled dead here (and, @WhiteNoise , I know we were talking about some of the same things on my other killed tickets). I’m at v0.13.23 here (v0.12.19 installer), and search still takes about a minute to return a first result for a single-word query that I know is present in four MD files.

That’s on a second attempt; I’m not sure how fast slow the uncached attempt is, since I left the same query running all day yesterday.

And, as I said on that other ticket, I know there are four hits for that word among my 2,526 markdown files because my own cobbled together find/xargs/grep script finds them in just over a second on this SSD ext4 machine. So, speed is definitely possible.

While writing this for the past five minutes or so, the search has been running in Obsidian, and it still has only returned the first of the four results. This doesn’t improve any if I open the four files in question via ctrl+o. (That is, the files are in the folders nominally indexed by Obsidian, and the search apparently does not give any special preference to opened files.)

10

The first search a day is for me still slow, around 2 minutes to get the first result. But after that, the rest of the day all searches are within 2 seconds.

So what I do is close Obsidian every day with a query in the search field. Then when I launch Obsidian the next day, it automatically does that search.

But as I usually start my day with a different app, Obsidian can in the background do its first slow search of the day.

Would this approach work for you Tomsb? Or all your searches after the daily first also slow?

Good luck!

11

I ended up writing an unpublished plugin of my own that farms the search out to GNU find. Only tested on Linux; it might work out-of-the-box on Mac, and might also work on windows with some Cygwin magic.

However, for my own markdown search/parsing project, I’ve found that I can get very fast results on a slightly different task (watching for changes to any files in a 500k-file tree, then filtering out any non-markdown changes) by using the Python package watchdog, which, though cross-platform, uses win32file.ReadDirectoryChangesW under the hood on Wind I ended up writing an unpublished plugin of my own that farms the search out to GNU find. Only tested on Linux; it might work out-of-the-box on Mac, and might also work on windows with some Cygwin magic.

However, for my own markdown search/parsing project, I’ve found that I can get very fast results on a slightly different task (watching for changes to any files in a 500k-file tree, then filtering out any non-markdown changes) by using the Python package watchdog, which, though cross-platform, uses win32file.ReadDirectoryChangesW under the hood on Windows. I assume there’s an equivalent interface in Node.js; I’m not really a JS programmer. Building a full-text search index off of this would be relatively easy: you have some sort of callback that is triggered by ReadDirectoryChangesW which updates an index, and then query that index upon demand.

Of course, this Obsidian also needs to work on its initial load time. It really shouldn’t take more than about a minute for it to scan that 500k directory so it at least knows where the markdown files are. The main thing is to use the tools the OS gives you, not reconstruct them from scratch yourself. Doing so might fragment the code across OSs (unless you can rely on nice packages like watchdog to abstract this away for you), but it’s absolutely worthwhile.ows. I assume there’s an equivalent interface in Node.js; I’m not really a JS programmer. Building a full-text search index off of this would be relatively easy: you have some sort of callback that is triggered by ReadDirectoryChangesW