iCloud on Windows creates duplicate files

Help
1

I’ve been an Obsidian user on Windows desktop for a while and I’ve recently moved my vault to my iCloud drive on Windows, so that I can access my vault on my iOS device as well. However, it seems that the iCloud drive on Windows doesn’t work well with Obsidian. iCloud creates lots of duplicate files. This even caused me to lose some data.

noteOne.md
noteOne(01).md
noteTwo.md
noteTwo(01).md
...

I’ve searched the forum for some guidance. It seems that the issue is known and that it might be related to Obsidian saving notes/files very frequently.

Is this a general issue for iCloud on Windows still, or is it only happening in particular circumstances? I don’t see these issues with OneDrive (and, no, I’m not using both at the same time), but then again, I cannot access the vault on OneDrive from my iOS device.

3 Likes
2

Here’s the big thread: iCloud sync issues

1 Like
3

This happens all the time - every time autosave is triggered on a file being edited this happens. It never happens when editing with an external editor with autosave switched off. If autosave could be switched off in Obsidian this problem wouldn’t exist.

1 Like
4

I’ve been playing with blocking iCloudDrive connections with the windows firewall, see

1 Like
6

Thanks so much everyone for your quick replies. To see if could replicate the problem, I wrote a quick batch script which writes to a file on my iCloud drive ~5 times per second. I did not succeed. iCloud handled and synced it flawlessly. So, I guess there’s more to it and that my test setup didn’t replicate Obsidian’s file handling specifically.

I kept investigating and tinkering and I was able to find a solution to decouple my local vault from the vault on the iCloud drive, and to keep them in sync. So far it has worked well for me. I’ll keep on testing it to see if it is reliable in the long run, but this is what I did.

Solution

  1. I set up two separate directories: One main directory on my local drive to keep the vault for the Obsidian desktop app and which isn’t synced to the iCloud drive. And one “remote” directory on my iCloud drive to keep a copy for my iOS apps:
    C:\Users\bobbieeray\Documents\04 Research\obsidian\Notebook
    C:\Users\bobbieeray\iCloudDrive\iCloud~md~obsidian\Notebook

  2. I installed Unison File Synchronizer for Windows. Unison is a file-synchronization tool for POSIX-compliant systems, macOS and Windows. I extracted it to a local directory and I also added the ../unison/bin/ directory to the PATH variable to make it more easily accessible from the command line (the latter is optional though).

  3. I added a new task to the Windows Task Scheduler, calling Unison every 2 minutes as a SYSTEM user so that it runs in the background. To create the task, I opened my PowerShell as an administrator and ran
SCHTASKS /Create /RU "SYSTEM" /SC MINUTE /MO 2 /TN "ObsidianNotebookICloudSync" /TR "C:\SDK\Unison\bin\unison.exe -batch -confirmbigdel=false 'C:\Users\bobbieeray\Documents\04 Research\obsidian\Notebook' 'C:\Users\bobbieeray\iCloudDrive\iCloud~md~obsidian\Notebook'"

Result

Unison allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

So, running Unison every 2 minutes ensures that both directories are synchronized both ways, without running into the sync issues and duplicate files, when storing the vault for the desktop app on the iCloud drive directory directly.

This will only work for a single user setup though. It can handle changes that happen either on the local machine, or in iOS. It isn’t capable to handle multiple users, working on the same note, simultaneously, i.e. it offers no (automatic) merging and/or conflict management capabilities.

3 Likes
7

that is fantastic!!! Thanks for researching it and coming up with a solution.
So it seems that when a file gets saved sometimes something weird goes on. If you ignore it iCloud drive syncs just fine. But somehow Obsidian thinks there is a conflict and creates those (1).md files. Did you have the note in Obsidian open while running those batch scripts?

8

To be honest I really don’t know why these issues occur when storing your vault on the iCloud drive directly. I tried to replicate the behaviour, but I didn’t succeed. I also tried to store the vault on MS OneDrive and that worked just fined.

I keep my notes open all of the time and I haven’t had any issues so far. If you set it up this way, you won’t even notice that the batch scripts are running in the background every 2 minutes.

The only issue you could encounter is, if you updated a note on your iOS device and immediately opened it on your PC, before the script had a chance to copy the iOS version to your local vault.

1 Like
9

The issue occurs when Obsidian tries to save the file at the same time as iCloud tries to sync it - seems both are trying to lock the file. If you use an external editor with autosave switched off the problem doesn’t occur. This makes it very difficult to understand why Obsidian must have an autosave function.

10

I’ve setup unison on my new Windows 11 machine. The documentation isn’t very user friendly but the windows GUI program allows for setup and manual testing.
Basically:

  • I copied my entire obsidian folder from the iCloudDrive to /Documents

  • opened obsidian from that new vault

  • to test it change a note there an/or on mobile obsidian then click “rescan” in the unison GUI and it shows the changes, then click go to sync those changes

    image
    image1137×663 32.8 KB

  • if you make changes in the same file on the local vault and the mobile vault it will show as a merge conflict. But it can’t merge it without setting up additional software. But I don’t make changes on both at the same time …

  • I used task scheduler to automatically run unison.exe with the arguments “obsidian -batch” as a background task (with option “run whether user login in or not”) every 5 minutes

  • I also added ignore = Name .obsidian to the .unison/obsidian.prf setup file to not have the obsidian config synced. That way I can have different settings and plugins on desktop and mobile

A side benefit of having a the obsidian vault (folder) inside the /Documents directory is that it’s super easy to automatically back it up with dropbox or most other backup programs.

1 Like
11

Wow, complicated changes. To much for me. I’m getting bunches of duplicates, every time, I switch from using O on m desktop (W11) to my laptop (W11). I’m also getting a “hold, downloading notice” file 1/12 (numbers change) and I can’t use O for sometime. Last night after O on desktop has been off for hours I opened my laptop to continue. It took 20 minutes to wait, then about 25 duplicates. I closed the file and it opened again with the duplicates. I closed the file and returned to my desktop and those particular duplicates weren’t there but another file was duplicated 5 times. Both are accessing the O file on iCloud.

12

If I buy the O sync service will this stop?

13

That’s two separate things.

If you stop using iCloud in Windows, that will stop. You still might have to clean up the duplicate files.

Then, if you choose to buy Obsidian Sync, it doesn’t have this problem. But there are other syncing options too.

If you buy Obsidian Sync, but keep using iCloud, the problem will not stop.

Whatever you decide to do, you should stop using iCloud plus Windows to sync your Obsidian files.

14

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.