Indexing Large Number of Files

Bug graveyard
1

Issue

  • The app fails to complete the indexing when there are a large number of files

Background

  • I have three vaults.

  • Vault a has < 1K

  • Vault b has ~ 3.5k files

  • Vault c currently has >113K.

  • Everything was working great with all three until I started adding more files to the third vault.

  • I added the files slowly to the third vault (about 1K) at a time

  • The two vaults with a small number of files still work. However, the vault with the large number fails to complete the indexing and locks up / freezes

  • The mobile app does the same thing

  • The desktop app gets to about 97K then freezes

  • The mobile app gets to about 36K then freezes

Steps to reproduce

  • I have uninstalled / reinstalled the desktop app several times with no success
  • In addition to the un / re-installation, I have deleted the obsidian folder from the .config folder many times

Expected result

  • I need the obsidian app to handle >500K files

Actual result

  • See the background info above

Environment

  • OS: Linux 6.0.7-arch1-1
  • Obsidian version: v1.0.3

Debug info

  • Debug info: It’s not possible to get any sort of debug info, because the app locks up

Additional information

2

I am sorry but I don’t consider this a “bug”.

You may be beyond what your device can handle memory-wise/system-wise with Obsidian. We are aware that there’s currently an issue in electron on linux that caps the memory utilization at 4gb.

You can try to disable third party plugins and see if it helps.

The work on pushing Obsidian’s limit is an ongoing effort.

3

Despite the large number of files I need, currently they total 803M which is much less than 4G

4

Are all these files markdown? What is the composition of the vault?

What does it mean it freezes? Post a screen recording, maybe you can can collect some info from the console (ctrl-shift-i) and by lunching obsidian from the terminal

5

Screenshot_2022-11-10_11-07-28
Screenshot_2022-11-10_11-07-281020×798 92.8 KB

Here’s a screenshot of what it looks like when it freezes.

Before this run, I disabled graphview. Obviously, that plugin is not the problem.

I will try dataview next

BTW, all the new files are md and all are about 1K size

6

Looks like dataview may have been the problem.

I disabled it, and the indexing finished.