Unpractical vault load time for large vaults on mobile (indexeddb transactions are not flushed to disk)

mnaoumov · September 16, 2024, 1:19am

Steps to reproduce

Prepare a vault with over 50k notes + 40k attachments.
Open the vault in the Android app

Did you follow the troubleshooting guide? [Y/N]

Y

Expected result

Fast vault load, especially after initial indexing
No unnecessary reindexing

Actual result

Vault load time is ~3 min
Vault reindexing is happening on every vault load
Vault reindexing time is ~27 min

Environment

SYSTEM INFO:
 Operating system: android 12 (OnePlus HD1913)
 Obsidian version: 1.7.1 (162)
 API version: v1.7.1
 Login status: not logged in
 Language: en
 Live preview: on
 Base theme: adapt to system
 Community theme: none
 Snippets enabled: 0
 Restricted mode: on

RECOMMENDATIONS:
 none

WhiteNoise · September 16, 2024, 3:23pm

1.7.1 introduced a set of timers for mobile startup. Settings->Advanced what is the output of that?

If you hook up a desktop debug session to the mobile app, do you see errors in console?

There will be another round of mobile
improvements in v1.7.2.

mnaoumov · September 16, 2024, 5:57pm

Obsidian start-up time breakdown

Operating system: android 12 (OnePlus HD1913)
Obsidian version: 1.7.1 (162)
API version: v1.7.1

- Total startup time: 314,929ms
- Initialization: 384ms
- Vault (86,685 files): 311,762ms
- Workspace: 1,048ms
- Core plugins: 1,735ms

It shows load time as 314 seconds = 5 minutes, however it doesn’t show that the vault indexing takes another 30-40 minutes . I don’t see why the index is not being saved.

BTW, the phone battery is heavily drained. The phone is connected to the computer, but the phone consumes more energy than gets from the computer.

Regarding errors console, the only I have:

app.js:1 Capacitor plugin "Keyboard" already registered. Cannot register plugins twice.

app.js:1 M: Directory does not exist
    at returnResult (http://localhost/:915:32)
    at win.androidBridge.onmessage (http://localhost/:890:21)

I also noticed but I am not sure if it means anything

app.vault.adapter.dir is null

According to my reverse engineering of your obfuscated code, it supposed to be a string, rather than null.

WhiteNoise · September 16, 2024, 7:07pm

If you wait with the console connected while it is indexing, does anything else pop up in console?

mnaoumov · September 16, 2024, 7:17pm

Nothing

mnaoumov · October 18, 2024, 5:47pm

Retested

Obsidian start-up time breakdown

Operating system: android 12 (OnePlus HD1913)
Obsidian version: 1.7.4 (167)
API version: v1.7.4

Total startup time: 317,166ms
Initialization: 451ms
Vault (98,029 files): 315,962ms
Workspace (8 tabs, 7 deferred): 555ms
Core plugins: 198ms

Indexing

Indexing vault content took ~40 mins.

Nothing is stored to IndexedDB, files and metadata tables are empty

There are no errors in the console

UPD: When I test on smaller vault, the IndexedDB is being written properly

mnaoumov · October 20, 2024, 8:42pm

I’ve done more research for different sizes of vault. I just created thousands of empty md files to see when the cache is stopped being stored

On 25k files I found that if you close the Obsidian app soon after Indexing complete message disappears and then reopen the app, it starts reindexing from scratch.

If you wait long enough before closing the Obsidian app, the index is saved to the file and reindexing not happening.

Sometimes I get the intermediate result but I don’t know how to get it consistently. Sometimes after reopening the app, only part of the files are being reindexed.

It seems that writing to the IndexedDB is happening asynchronously and UI has no visible indicator of this process, so closing the app before it’s done, loses the index (fully or partially).

In my opinion, you need to show Indexing completed message only after IndexedDB is written down.

I am trying to find the exact amount of files where the indexing is stuck (or maybe takes more time than my patience allows)

mnaoumov · October 20, 2024, 11:41pm

More from my research

To make it more or less reproducible, I test in the following condition

Force Stop the Obsidian app and clear its storage.
Open the vault with N empty md files.
Wait until Indexing complete message is shown.
Start stopwatch.
In exactly 1 min, close Obsidian app
Reopen Obsidian app
Check if the vault reindexing is started.

Based on my bisect approach, I found that if I have 9845 notes, reindexing is not starting, but for 9846 notes, reindexing is starting.

The number will probably be different on other phones. For reference, the reindexing of 1000 md files takes ~7 seconds.

Attaching a sample vault with 10k records for your testing
test.zip (940.4 KB)

WhiteNoise · October 21, 2024, 3:04am

I did some experimenting tonight. We believe that the problem lies in the fact that that the indexeddb transactions are not actually flushed to disk when they are complete (even when using strict mode).
Unfortunately, we have no way on knowing when the process is truly done.

The good news is that the process is incremental, so you will make progress every time you use obsidian.
There are two tables that are created the first time you add vault.

files (populated during load workspace) and
metadata constructed during indexing.

You are observing “indexing” restarting from zero because the preceding file table isn’t finished yet. Once file is complete, loading workspace becomes fast, and you will start see progress being made (saved) on metadata during indexing.

mnaoumov · October 21, 2024, 3:41am

Have you considered using readwriteflush mode for IDBTransaction? This seems like exactly what we are missing here.

Alternatively, you can check

const filesRowCount = await new Promise<number>((resolve, reject) => {
  const request = indexedDB.open(`${app.appId}-cache`, 19);
  request.addEventListener('success', (event) => {
    const db = event.target.result;
    const transaction = db.transaction('file', 'readonly');
    const objectStore = transaction.objectStore('file');
    const countRequest = objectStore.count();
    countRequest.addEventListener('success', () => {
      resolve(countRequest.result);
    });
    countRequest.addEventListener('error', () => {
      reject(new Error('Error counting rows'));
    });
  });
  request.addEventListener('error', (event) => {
    reject(new Error(`Error opening database: ${event.target.errorCode}`));
  });
});

I checked that on normal vaults, it returns the number. On a vault that still not flushed, the promise is waiting forever

Licat · October 22, 2024, 10:08pm

No browsers support readwriteflush. See IDBDatabase: transaction() method - Web APIs | MDN

If it hangs forever what would we be even checking for?

I think from the amount of debugging you provided we can conclude this is an indexeddb issue (possibly just the chromium/android web view implementation), and there doesn’t seem like there is any API that we can leverage to force it to behave the way we want.

mnaoumov · October 22, 2024, 10:38pm

I understand we can’t force it to save, but having a visual indicator that the saving is still taking place and potentially showing the warning if you try to exit before it’s done, would improve the UX

mnaoumov · October 22, 2024, 10:39pm

Maybe it doesn’t take forever, maybe it’s just a matter of hours that I didn’t wait

Licat · October 22, 2024, 10:56pm

To be honest I don’t think it’s worth attempting to address given the situation is more of an edge case and the rest of the app is pretty much non-functionally slow anyway, that it wouldn’t practically achieve anything. Maybe by the time devices gets fast enough to handle vaults of this size the chromium team would have already improved the indexeddb perf, or introduced APIs to handle this better. For now I think it’s just unpractical, like your title say.

mnaoumov · October 22, 2024, 11:09pm

I guess I’ll try to implement it as a plugin then. Dunno if it has a chance to be approved, though