FR : Maintain a registry of all the files which has been modified when Obsidian was closed

Developers: Plugin & API
1

Problem Statement

Here is a problem I want to solve…

One of the biggest power of Obsidian is that is Open Access. That means, any external application or service can modify the content from the Vault the way they want and at anytime. Especially, when Obsidian is not active (on desktop or mobile), there is no way for plugins to know which files had been updated between the time when Obsidian was closed and re-opened again.

For example, yesterday I closed Obsidian at 10pm and today opened it again at 11am. Between this time, I used another application to edit some of the files from the vault. Or there was an external synching service which updated some of the files.

Now, as a plugin, if I want to find out which all files had been modified between this time. The only approach I have is to iterate over all the files from the vault => Get their last modified time => And check whether this time falls between this window (from 10pm to 11 am). I havent implemented this yet, so not sure how much resource consuming process this would be. But surely sounds like its going to take some time if there are thousands of files in the vault. (Also, as per my google searching, I dont think there are any APIs provided by the file system itself to fetch all the files by sorting them in a modified-time criteria and then getting the top files which were modified after a particular time).

Possible Solution

What I am requesting here from the Obsidian is…

If I am correct Obsidian is running a indexing mechanism when it boots up. Now, I feel its running it over all the files within the vault. Because bigger the vault, its takes longer to index.

If thats true, can Obsidian provide an API, where a plugin can call this API and will return all the files which were modified after the Obsidian application was closed last time.

The next step would be to pass the last modified time and get all the files modified after that. But this is really optional and the basic functionality which will really going to help is to be able to get all the files which were modified after the application was closed last time. As a plugin doing this indexing whenever the plugin loads seems really unnecessary as Obsidian has already done that.

I am all ears to learn more about whatever the community has to share, whether my assumption on Obsidian indexing is correct and is there a better approach to achieve this.

2

There was an already easy solution available to find all the modified files using the Obsidian’s app.vault.getFiles() API. The solution was provided by Fevol in Discord, as shown below :

			const modifiedCreatedFiles = this.app.vault
				.getFiles()
				.filter((file) => {
					filesScannedCount++;
					return (
						file.stat.mtime > LAST_UPDATED_TIME ||
						file.stat.ctime > LAST_UPDATED_TIME
					);
				});

The only thing will need to store is the LAST_UPDATED_TIME, which can be stored at the time when the application is closing. Or when the last change was made by user from the Obsidian.

Through the above method we can easily are able to find either the modified or newly added files to the vault. But it has few limitations :

  • There is no simple and faster way to find the deleted files. The obvious solution will be to store all the fileNames in a cache, which should be done by the plugin. And then, at the startup time, filter out the files which are no longer found in the new list provided by the API.
  • Similarly, the reverse calculation will be required to be done, to find out the files which has been renamed. The older names files will get in the above calucation as deleted. But to find out which files has been renamed to what is not possible. We have to do another filtering operation vice-versa to the first one to get the new files. And please note that, the approach mentioned in the code will not give us these renamed files, since their .mtime or .ctime value dont change. It remains as the older value.
  • A third issue, which can be also a rare issue is, when a lot of changes has been done or a lot of new files has been added to the vault. Obsidian runs an indexing mechanism at the start. And for a plugin there is no way to find out, when this indexing process completes. So, in a very rare case, we might not get all the modified files when will use the above approach.

Feature Request to Obsidian

So, through this forum topic, I would like to request either one the below :

  • To provide a callBack function, just like the onLayoutReady(), which will provide us with the files which has been modified, the files which has been deleted and the files which has been renamed (possible with the older fileName).
  • The first request seems a little too much, so a easier way would be to provide a callBack function, which will run when the indexing process has been finished. This will help a lot to know when to run our functions to start with our plugin’s calculations.