Indexing Large Number of Files - Still Not Working

Bug graveyard
1

Continuing the discussion from Indexing Large Number of Files:

The BUG is not resolved!

  • Obsidian still does not finish the indexing. It either hangs or closes itself without any input
  • Obsidian indexes about 80K to 128K of the 251K files then hangs or closes itself (the number indexed varies each time)
  • When it hangs, nothing on the window responds to mouse input

History

  • Everything was indexed and syncing well until I added over 247K small md files

Troubleshooting

  • I have since disabled ALL of the plugins, both core and community
  • I am currently using the default theme
  • I tried deleting the obsidian directory in .config and the .obsidian directory in the vault to force obsidian to start fresh
  • Attempted to launch obsidian via the cli. Received the following:

2022-11-11 22:00:11 Loading main app package /usr/lib/obsidian/obsidian.asar
xdg-settings: default-url-scheme-handler not implemented for xfce
2022-11-11 22:00:11 Checking for update using Github
[12277:1111/160011.175472:ERROR:sandbox_linux.cc(377)] InitializeSandbox() called with multiple threads in process gpu-process.
[12277:1111/160011.178921:ERROR:gbm_wrapper.cc(275)] Failed to export buffer to dma_buf: No such file or directory (2)

A whole lot more of those error messages, then the following:

[12277:1111/160011.180156:ERROR:gbm_wrapper.cc(275)] Failed to export buffer to dma_buf: No such file or directory (2)
2022-11-11 22:00:11 Success.
2022-11-11 22:00:11 Latest version is 1.0.3
2022-11-11 22:00:11 App is up to date.
Trace/breakpoint trap (core dumped)

Vault Info

  • Total items (including directories) - 251,602
  • Total size - 921.1 MB
  • File types & quantity (other than md):
    – ods = 15
    – odt = 28
    – pdf = 27
    – jpg = 287

System Info

  • Obsidian version v1.0.3
  • Electron version 20.0.3-3
  • OS: Arch Linux
  • Kernel: x86_64 Linux 6.0.8-arch1-1
  • Uptime: 6h 37m
  • Packages: 795
  • Shell: l
  • Resolution: 3849x2160
  • DE: Xfce4
  • WM: Xfwm4
  • WM Theme: Galaxy
  • GTK Theme: Adwaita [GTK2]
  • Icon Theme: Adwaita
  • Font: Roboto 12
  • Disk: 1.3T / 4.6T (29%)
  • CPU: AMD Ryzen 9 5950X 16-Core @ 32x 3.4GHz
  • GPU: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn XT GL [FirePro W7000]
  • RAM: 5696MiB / 128730MiB
2

As i told you in the other message. We do NOT support this many files at the moment. I do not consider this a bug.

Efforts on expanding Obsidian limits are ongoing and are not bugs.

3

Also we don’t make the arch package and we don’t support it. Next time you file a bug report make sure to try the appimage or snap.

4

You never said obsidian does not support this number of files. you said, “We are aware that there’s currently an issue in electron on linux that caps the memory utilization at 4gb”

I have seen lots of notes asking about a file limit but so far, I haven’t seen any answer to those questions.

5

I said that you may have hit the limit of what obsidian can currently handle. Which is the same thing in my mind. Maybe i was more explicit this time.

Maybe there will be some sprint in the future to push the current limits, but I don’t consider this under the category of bugs.

6

You may not consider it a bug, but the dictionary does

“A software bug is an error, flaw or fault in the design, development, or operation of computer software that causes it to produce an incorrect or unexpected result, or to behave in unintended ways.”

Unless you did in fact intend to limit the number of files. If that was the case, you should have stated so in the documentation

7

BTW, the appimage from your website doesn’t work either

8

It’s not a bug, we do not currently expect Obsidian to work with that volume of data. I am sorry.

9

What’s your use-case, if I may ask? It’s none of my business, I’m not a dev. I’m just asking out of curiosity.

10

Sorry for the very late reply. I didn’t receive a notice about your comment.

I am building a cross-linked Bible using the BSB. There about 500K Hebrew / Aramaic / Greek words in the BSB translation. The people at the Berean Study Bible provide a free spreadsheet with each of the word and additional details

I wanted to link each of those words to the English translation on a verse by verse basis and link them to a summary file.

I can provide more details if you are interested.

1 Like