Windows: Sometimes changes made outside of Obsidian are not reflected in the Files pane

mnaoumov · December 21, 2023, 9:04pm

Steps to reproduce

You have a big vault. Here is the sample.
You copy a big amount of files at once in this vault while you have Obsidian open on that vault.

Did you follow the troubleshooting guide? [Y/N]

Y

Expected result

All file system updates are reflected in the Files pane in Obsidian.

Actual result

Some of the changes are missing.

See video: https://youtu.be/C-uKULzPNX4

Environment

SYSTEM INFO:
Obsidian version: v1.5.2
Installer version: v1.4.13
Operating system: Windows 10 Pro 10.0.22631
Login status: logged in
Catalyst license: insider
Insider build toggle: on
Live preview: on
Base theme: adapt to system
Community theme: none
Snippets enabled: 0
Restricted mode: on

RECOMMENDATIONS:
none

Additional information

The usual workaround for this problem is to close and reopen Obsidian or invoke Reload app without saving command, but for big vaults such workarounds adds undesired waiting time.

I’ve built a plugin to work this issue around more efficiently: GitHub - mnaoumov/obsidian-file-explorer-reload: Obsidian Plugin that reloads the file explorer files list

WhiteNoise · December 21, 2023, 9:54pm

Where is that vault stored exactly?
Like, is it a locally attached nvme with ntfs filesystem or something else?

Does this happen only if you copy a few files?

mnaoumov · December 21, 2023, 10:11pm

Yes, this is located on NVMe SSD with NTFS file system.

It might happen with only a few files as well but it is significantly more difficult to reproduce. I have a feeling that it is happening when the background Metadata Cache Worker is busy indexing some files, it can miss new files updates.

Fred_V · January 18, 2025, 4:32am

Hi,

Contrary to what is claimed in the readme, the File Explorer Reload plugin is not listed in the community plugins repo.
Check Plugins - Obsidian

mnaoumov · January 18, 2025, 4:45am

@Fred_V README clearly says

File Explorer Reload is not available on the official Community Plugins repository yet.

Where is it lying?

Team explicitly decided NOT to add the plugin: Add plugin: File Explorer Reload by mnaoumov · Pull Request #2783 · obsidianmd/obsidian-releases · GitHub

Fred_V · January 18, 2025, 5:14am

I’m so sorry. I missed the yet. I must be tired.

mnaoumov · March 18, 2025, 2:42am

I found the root cause of the problem.

Obsidian uses fs.watch() which is known to have issues, especially on Windows.

I tried alternatives and even found reliable watcher, which I used in my other plugin

github.com/mnaoumov/obsidian-external-rename-handler

src/ExternalRenameHandlerPlugin.ts

1d4f19835


      
          private registerWatcher(): void {
            const watcher = new Watcher(this.app.vault.adapter.basePath, {
              ignoreInitial: true,
              native: false,
              recursive: true,
              renameDetection: true
            });
          
            watcher.on('error', this.handleWatcherError.bind(this));
            watcher.on('all', this.handleWatcherEvent.bind(this));
          
            this.register(() => watcher.close());
          }

native: false seems to be important, without it, the issue persists.

A year ago my other plugin was rejected because the team wanted to have a proper fix, not the hacky solution I suggested.

Here is the proper solution for the team: replace fs.watch with something more reliable like watcher

mnaoumov · March 19, 2025, 12:14am

After some more testing, I found this is still not 100% reliable.

watcher still uses fs.watch() inside, which on Windows relies on ReadDirectoryChangesW which can miss some events under certain circumstances.

Testing chokidar instead

mnaoumov · March 19, 2025, 3:08am

Managed to get chokidar working more reliably

mnaoumov · March 25, 2025, 11:21pm

After investigation, I came to conclusion: all file system watchers in Node.js are using fs.watch(), which uses ReadDirectoryChangesW Windows API, which is not 100% reliable and can miss some events, especially if many of them happens at once.

The only way to ensure the changes are not missed, is to do a regular polling. Doing it too often, will degrade the performance, especially for the large vaults.

chokidar has convenient additional polling mechanism.

Obsidian team could consider doing something similar.

Use my plugin for the reference.

luckman212 · March 28, 2025, 11:58am

Damn @mnaoumov that is good research and kind of unfortunate nasty bug in Windows. Hard to believe they left this bug to persist for so long. I don’t use Windows much, maybe only 10% of the time so I haven’t been bitten by this bug. I hope at least, if implemented, the fix would target Windows only to avoid unnecessary polling on other OS’es.