[Bug] File Loss iCloud/iPad/Mac Sync

Platform

[x ] iOS
[ ] Android

Obsidian Mobile version: v1.0.4


File loss on sync

I thought I had seen this before but the file was trivial and I dismissed it. I just now lost a large file into which I was putting many notes.

Chances are that the file was open on mobile and on desktop at the same time.

I opened the file (A) on desktop and cut a block of text from it.

I then created a new file (B) and pasted the cut block of text.

At this point the file list refreshed as it does during a sync and file (A) was gone.

I looked in the trash on the Mac and no file was there.

I tried undoing several steps but my file never returned so it was not manually removed.

There is no other file service being used (other than iCloud) and no automation on the folder.

1 Like

Do you have iCloud syncing activated for the same folder that is synced with Obsidian Sync? There is a strong suggestion to not have iCloud / any other sync service running for the same files because it will cause problems.

I only had iCloud at the time. I purchased obsidian sync has a result of this file loss. And removed the iCloud sync stuff at that time. My files are now hosted locally and obsidian synch is handling it.

At the time of this file loss only iCloud was being used.

Thanks for the report. It’s a strange scenario. You should be able to keep the same file open on multiple devices and edit them roughly simultaneously on either iCloud Drive sync or Obsidian Sync.

Hard to diagnose from the info we have, unfortunately. If you think of anything else that might have played a role, please add it here!

Of course, I knew that as I wrote that. It might be a bug in iCloud and now that I’ve purchased Obsidian Sync and moved my files off iCloud I doubt it will be a problem.

1 Like

Honestly I’ve run into this 3 of the last 3 days. New file. Modify. Poof. There’s some race here.

I’ve been trying to narrow down the repro but I’m nearly done trying with iCloud. I can repro self-conflicting files in Terminal as well. I have a big repro script that I’ve been trying to get more limited.

In my case I’ve had plenty of times where I’ll trivially modify a file and it just disappears into the system(or iCloud) trash, and many conflicted files show up in the obsidian trash. None of it makes sense.

I would not trust iCloud for anything after this experience.

Designing for Documents in iCloud makes me think Terminal isn’t expected to “work” and that Obsidian may be using the APIs wrong. I find it hard to believe iCloud is so broken. Seriously 3 times in the last 3 days I’ve seen files disappear.

Sorry to hear about the trouble. Odd though, I have been using iCloud Drive sync for months with no trouble.

Are you using any other apps or services with your vault files?

In my case I was most certainly not touching my vault outside of Obsidian.

I am only using 1 client actively. All I do to repro this is open a new file, wait a second, modify it, then the file gets deleted. Though I’ve only done it in Obsidian by accident so far. Sounds crazy right? I am going to brain dump here as there are nearly 0 results for any of this on various search engines, and maybe someone else will have ideas.

This script trivially creates conflicted files if ran from ~/Library/Mobile Documents/com~apple~CloudDocs on OSX. It can take a few minutes. For my real use cases I wasn’t going crazy though trying to create races I was typically just doing things “normal”.

#! /bin/sh

DATA="$(mktemp)"
DST="synctest"
# Just get some data. The contents are irrelevant.
ps uaxwwww > "${DATA}"
while [ ! -e "${DST} 2" ]; do
        rm -f ${DST}*
        sleep 3

        touch "${DST}"
        sleep 1
        cat "${DATA}" >> "${DST}"
        sleep 1
        cat "${DATA}" >> "${DST}"
        sleep 3
done
# Exits when "synctest 2" is created.

An odd thing is that the “synctest 2” file is actually a renamed “synctest” file since the main file was redownloaded. This matches what I see in my Obsidian .trash folder. That the “2” files are the newer ones I expect. This is seen with a more verbose version that shows inodes.

Of course that’s not Obsidian, and it uses the wrong APIs. But it’s an example and matches the steps I’ve taken that resulted in deleted files. Specifically for Obsidian when this sort of conflict occurs the file is straight up deleted into the trash, as well as the new conflicted download. In my example neither file is deleted when the script exits.

brctl is useful for monitoring this.
while :; do brctl status com.apple.CloudDocs; sleep 1; done
brctl monitor -i com.apple.CloudDocs
For Obsidian:
brctl monitor -i iCloud.md.obsidian

At the point where the problem happens brctl monitor will claim that the file was modified on the server, “server edit to download”, and needs to be redownloaded. At another point I saw “generation counter mismatch”.

I am still looking into this and trying to come up with a concise bug report for Apple, assuming they care about the POSIX API I am using. There are some other oddities involved that I don’t fully understand yet. I am not any kind of Apple developer. I do BSD development for a living though. Right now I am wishing I was working at Apple and could truly debug this. There are maybe 2 hits on any search engine relating to any of the technical details that I can find.

TLDR: There is clearly some kind of race with modifying new files too quickly where the server side thinks there is an edit and causes the iCloud client to redownload into a conflicted state, and rolls back to an older version. Obsidian seems to delete the file in this case which is specific to Obsidian. I do suspect that Obsidian is not properly using the cloud APIs, or some plugin is doing something wrong.

more verbose version

#! /bin/sh

DATA="$(mktemp)"
DST="conflicted"
# Just get some data. The contents are irrelevant.
ps uaxwwww > "${DATA}"
i=0

showmeta() {
        DST="$1"
        echo "# stat -x ${DST}"
        stat -x "${DST}"
        echo "# xattr -l ${DST}"
        xattr -l "${DST}"
        echo
}
while [ ! -e "${DST} 2" ]; do
        i=$((i + 1))
        rm -f ${DST}*
        sleep 3

        touch "${DST}"
        sleep 1
        {
                echo "$i" # add version into the file
                cat "${DATA}"
        } >> "${DST}"
        showmeta "${DST}"
        sleep 1
        i=$((i + 1))
        {
                echo "$i" # add version into the file
                cat "${DATA}"
        } >> "${DST}"
        showmeta "${DST}"
        sleep 3
done
echo "Conflicted '${DST} 2' detected." >&2
echo "i= $i" >&2
showmeta "${DST}"
showmeta "${DST} 2"

And some output near the conflict:

# stat -x conflicted
  File: "conflicted"
  Size: 291726       FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (1087209050/  [REDACTED USERNAME])  Gid: (720748206/(720748206))
Device: 1,4   Inode: 4334562453    Links: 1
Access: Wed Nov 10 20:37:32 2021
Modify: Wed Nov 10 20:37:32 2021
Change: Wed Nov 10 20:37:32 2021
# xattr -l conflicted

# stat -x conflicted
  File: "conflicted"
  Size: 145863       FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (1087209050/  [REDACTED USERNAME])  Gid: (720748206/(720748206))
Device: 1,4   Inode: 4334562541    Links: 1
Access: Wed Nov 10 20:37:39 2021
Modify: Wed Nov 10 20:37:40 2021
Change: Wed Nov 10 20:37:40 2021
# xattr -l conflicted

# stat -x conflicted
  File: "conflicted"
  Size: 291726       FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (1087209050/  [REDACTED USERNAME])  Gid: (720748206/(720748206))
Device: 1,4   Inode: 4334562541    Links: 1
Access: Wed Nov 10 20:37:40 2021
Modify: Wed Nov 10 20:37:42 2021
Change: Wed Nov 10 20:37:42 2021
# xattr -l conflicted

Conflicted 'conflicted 2' detected.
# stat -x conflicted
  File: "conflicted"
  Size: 291726       FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (1087209050/  [REDACTED USERNAME])  Gid: (720748206/(720748206))
Device: 1,4   Inode: 4334562603    Links: 1
Access: Wed Nov 10 20:37:45 2021
Modify: Wed Nov 10 20:37:32 2021
Change: Wed Nov 10 20:37:45 2021
# xattr -l conflicted
com.apple.lastuseddate#PS:
00000000  0C 9E 8C 61 00 00 00 00 00 00 00 00 00 00 00 00  |...a............|
00000010
com.apple.metadata:_kMDItemUserTags:
00000000  62 70 6C 69 73 74 30 30 A0 08 00 00 00 00 00 00  |bplist00........|
00000010  01 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00 00 09                    |..........|
0000002a

# stat -x conflicted 2
  File: "conflicted 2"
  Size: 291726       FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (1087209050/  [REDACTED USERNAME])  Gid: (720748206/(720748206))
Device: 1,4   Inode: 4334562541    Links: 1
Access: Wed Nov 10 20:37:44 2021
Modify: Wed Nov 10 20:37:42 2021
Change: Wed Nov 10 20:37:44 2021
# xattr -l conflicted 2
com.apple.metadata:_kMDItemUserTags:
00000000  62 70 6C 69 73 74 30 30 A0 08 00 00 00 00 00 00  |bplist00........|
00000010  01 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00 00 09                    |..........|
0000002a

The xattrs are probably irrelevant but there is a pattern I noticed that I can’t articulate yet.

% head -n 1 conflicted*
==> conflicted <==
13

==> conflicted 2 <==
15

So conflicted 2 is the newest version here. And version “14” is just GONE. Rolled back to 13.

1 Like

Curious. When does this happen in practice? Like, what would you have to do to see this in Obsidian?

In practice I believe it has been a fairly uncommon workflow of testing Templates.

  1. Create note titled FOO
  2. Edit FOO
  3. Delete FOO
  4. Create FOO again

This is at least from what I recall.

Looking over the source more I see this uses capacitorjs which deep down appears to just use the POSIX API as well.

The workaround is to just wait 10 seconds before recreating a deleted note or renaming to one.

Just to add on: I’ve encountered the same issue. Not sure how to recreate it consistently, since I literally just started using Obsidian a few days ago.

  • iCloud Drive Sync
  • Only two files right now
  • Open on iPhone and iPad only

Three times now, I open my main file on Obsidian (where I’m first dumping notes right now), start typing a few letters, and then suddenly the interface updates to the “No File Selected”, the file is gone from the Obsidian File Explorer, and I (at least) see the file in “Recently Deleted” in the Files app. It’s always after coming back to Obsidian after doing other things and then typing a few letters.

I just restore it in these cases, but it’s been getting annoying to keep dragging it back from Trash into Obsidian’s vault in Files.

I’m not fully sure it’s the exact same issue (since I can find mine in Trash, and it’s also a much more basic use case vs. what Bryan outlined), but it is files disappearing and Google brought me here after searching for more info on the issue.

2 Likes

To re-reply for myself and follow up:

After searching around Google even more, I found one particular thread in the Obsidian forums that talked about how iCloud sync on Windows (through iCloud for Windows) was deleting files.

I have not been using iCloud Drive for Windows because it was buggy and I never got it working correctly, but it turns out that it was still technically checked on my Windows desktop. I unchecked it, and so far as I can tell, I have not experienced a single file loss/deletion since.

It was happening consistently enough previously that I’m pretty confident in saying it wasn’t a coincidence. Anyway, wanted to report back in case anyone else also comes across the thread.

1 Like

My issues were on OSX, but I too have a system elsewhere with iCloud Drive (from Windows Store) running. I am out of town for work for a few weeks. I don’t remember if Obsidian is open there either. Once I get back home I will retest after disconnecting iCloud Drive on the Windows system and post an update too.

1 Like

Well, the weird thing is Obsidian was NOT open on my Windows machine. I figured I’d set up Syncthing or something later, but hadn’t at that point. Nothing else seems to have this “file disappearing” issue except Obsidian–so maybe there’s something odd in terms of Windows iCloud Drive and however Obsidian uses the API or saves things.

There are recurrent reports of iCloud losing files on many platforms.
Obsidian requires all files in a vault to be local.
iCloud has a tendency to save memory on mobile OSs by losing local copies and serving from the cloud.
Google tries to switch users to streaming on Windows, which causes Obsidian problems when they’re not local.
Any chance iCloud is doing something similar here?
Most programs won’t have the same problem because they are either actively using a file or can wait for the cloud copy.

interesting. I’m having issues with notes disappearing on my iphone. I’ll make sure to put my PC to sleep when I’m working in the iphone to see if that prevents it from happening.