Basic memory management to avoid OOM errors

[clehene]

Obsidian

• limited to 4GB of usable memory due to Electron/V8 memory cage.
• uses memory-intensive background operations - e.g. indexing
• it has no control or limits over these operations

This can cause Obsidian to frequently crash when

• it’s indexing large files
• it’s indexing several files (e.g. when performing a file / heading rename which causes changes in multiple files at a time)

In fact, with multiple files open, it will index files on startup actually getting into a startup crash loop.
This has been the case on my iPhone for a long time, but also happened on my desktop and I had to manually delete the open files from the state JSON files as it was crashing on every start.

There are plenty of ways to reduce memory use, however given that there’s an upper memory limit that will cause crashes, there needs to be a limit on what can run when in the background based on the memory utilization.
Electron/V8 comes with APIs that allow tracking of memory use.
When starting background processes there needs to be a controller that prevents crashes.

This can be done by estimating the memory use for a task either statically (e.g. based on measurement based on the file size, or dynamically by actively measuring and keeping statistics). Alternatively, a user configurable setting for the max number of background operations (or if indexing is the only large one, the max number of indexing operations).

[CawlinTeffid]

Closely related, possibly should be merged:

Obsidian build without V8 pointer compression

The default Electron/V8 comes with pointer compression this limits the per process memory footprint to 4GB. Pointer compression (aka memory cage) compresses pointers to 32 bit. As a result the total addressable space is 4GB. This makes a lot of sense if you can fit in 4GB heaps, but the current indexing implementation doesn’t. Related Don't reindex whole file every few keystrokes My vault constantly crashes with with OOM errors during indexing. This can happen if I type a character in cert…

[clehene]

@CawlinTeffid thanks - they are related (I wrote both) - they are both workarounds for memory related crashes, but not the same, but complementary.

While a build without pointer compression would be interesting, until it gets directly supported by Electron it would be at most a testing thing (experimental).

Memory management, on the other hand, would be needed and helpful regardless.

A third one, also related and also complementary, is the incremental indexing which would tackle on of the main culprits for both memory use and latency.