Obsidian Sync: updates from one device overwritten by another

rongyi · February 23, 2022, 8:31pm

This happens when using Obsidian Sync on multiple devices (Macs + iOS devices).

For a specific file (Daily Note for example), one device (Mac) starts with version 1 and sync across all devices using Sync. Then I add some update to the file, making it version 2. At this time the iOS device hasn’t sync this version 2 due to lack of background sync when the app is not opened. If I now open Obsidian on iOS with daily note at version 1, and immediately add some content, if it works correctly, the file should be version 2, and try to resolve merge with server-side version 2 to make it version 3. But no, Obsidian think this updated file on iOS (version 1 + update) is version 3, and overrides the file on the mac with current content on iOS, so essentially the previous updates on the Mac was erased.

I now have to keep an eye on the Sync status every time I open Obsidian on iOS, and have to wait for the sync to complete before I can edit the note or use QuickAdd plugin, otherwise it would intermittently trigger the above issue.

And moreover, there is no warning or merge conflict in this case, the file overwrite process is silent. If I notice the data loss, I have to use “View Sync version history” to restore the previous version and redo my current edit. But most of the time I wouldn’t even notice the file content was missing part of my previous update…

Steps to reproduce

I currently don’t have a clear minimal reproducible case / procedure. But it can be briefly illustrated as follows:

Assume two devices using Obsidian Sync: Mac and iPhone.

1. Start new file with content on Mac

Version 1 content:

# Heading
- This line was from Mac, initial version

2. Wait for it to sync to iPhone

Now both devices have file at Version 1.

3. Close iOS Obsidian app

4. Make Edit on Mac

Version 2 content:

 # Heading
 - This line was from Mac, initial version
+- This line was updated on Mac, line 2

5. Open Obsidian app, and immediately edit the file

Usually Sync was not yet completed (either not started or in progress). Now the file on iOS should look like:

 # Heading
 - This line was from Mac, initial version
+- This line was updated on iPhone, line 3

6. Sync override the Mac updates

Version 3 content:

 # Heading
 - This line was from Mac, initial version
+- This line was updated on iPhone, line 3

Line 2 update was lost.

Expected result

Either properly handle the merge, resulting in:

 # Heading
 - This line was from Mac, initial version
+- This line was updated on Mac, line 2
+- This line was updated on iPhone, line 3

Or handle it as conflict, concat both versions

 # Heading
 - This line was from Mac, initial version
 - This line was updated on Mac, line 2

+# Heading
+- This line was from Mac, initial version
+- This line was updated on iPhone, line 3

Actual result

 # Heading
 - This line was from Mac, initial version
+- This line was updated on iPhone, line 3

Environment

Operating system: macOS & iOS

Debug info:

SYSTEM INFO:
Obsidian version: v0.13.24
Installer version: v0.12.15
Operating system: Darwin Kernel Version 21.4.0: Tue Jan 18 13:02:08 PST 2022; root:xnu-8020.100.406.0.1~18/RELEASE_ARM64_T8101 21.4.0
Login status: logged in
Catalyst license: supporter
Insider build toggle: on
Live preview: on
Legacy editor: off
Base theme: light
Community theme: Minimal
Snippets enabled: 1
Safe mode: off
Plugins installed: 28
Plugins enabled: 24
1: Advanced Tables
2: Calendar
3: Quick Switcher++
4: Natural Language Dates
5: Linter
6: Custom window title
7: QuickAdd
8: Advanced URI
9: Metadata Extractor
10: Graph Analysis
11: Diagrams
12: Tasks
13: Annotator
14: Open vault in VSCode
15: Highlightr
16: Local images
17: Contextual Typography
18: Global Hotkeys
19: Templater
20: Minimal Theme Settings
21: Dataview
22: Admonition
23: Excalidraw
24: Style Settings

RECOMMENDATIONS:
Custom theme: for cosmetic issues, please first try updating your theme to latest. If still not fixed, please try to make the issue happen in the help vault or disable community theme and snippets.
Community plugins: for bugs, please first try updating all your plugins to latest. If still not fixed, please try to make the issue happen in the help vault or disable community plugins.

Additional information

File content is private, but notice the file size. I did not delete anything when edit on the iPhone (only appended one line). All the edits from Mac between the two iPhone edits was lost. The last iPhone edit was based on the 1.47KB file version.

I can provide the screenshot of Sync Version History screenshots with file content to the developers privately upon request.

WhiteNoise · February 26, 2022, 2:58am

The way the sync algo is designed, what you are describing can only happen if a new file is simultaneously created and not synced on two devices. Since the file was already created and synced, this should not happen.

Are you sure about the repro step you provided?

Licat · February 26, 2022, 3:21am

Looking at the version history, I’ve heard of one more case of this happening.

I feel like this is due to a race condition between the editor saving and sync downloading the file. If that’s the case, it would mean that you won’t be able to reproduce the issue consistently. In fact, given modern file systems are pretty fast, this should be rarely reproducible.

However, this is a serious problem and I will try to brainstorm ways to fix this.

More info I’d like to request, if you can give it a try - based on your repro steps, how reliably can you reproduce the issue? Is it like a 5/5 reproducible, or 1/5, or is it 0/5 (completely not reproducible, so it must be rare)? Knowing this would help me better predict the causes of failure.

Thanks!

rongyi · February 26, 2022, 5:45am

Hi @Licat , thanks for the explanation. I’d say it’s definitely not 100% consistently reproducible, but a maybe 3/5 when the iPhone was on Cellular data. I have encountered this issue multiple times (>3 in the past few weeks), with most of them happened when I go outside to the mailroom and the phone switches to 5G.

And about your theory of editor saving and syncing the same file, I doubt that’s the case. As you mentioned the filesystem write should be instantaneous, so this should be way less than 1/5 reproducible (probably even 1/10000), which is not the case for me. For me it feels more like a problem of determining the newer version between server-side 2pm file and client-side 1pm file plus some additional 3pm updates, but don’t quote me on that, as I have no idea how the Sync works.

And @WhiteNoise, no I’m not creating new files, this happens to the daily note, which is always created early in the morning, way before any of override happens.

Is there anything specific that I can do to help debug this issue? I can probably pull up the sync version history with file content for the last few cases of this. For daily notes, I only append lines to the end of the file, so negative file size update normally won’t happen, unless this issue is triggered.

Here’s another case, which I explicitly wrote about the data loss in the daily note, thus easy to find through search. There’re definitely more, but takes time to locate.

Licat · February 26, 2022, 6:00am

If you’re able to reproduce this issue on purpose, please do that and show me the sync log on the iPhone (which is the device that overwrites the file). I’m looking for everything in the sync log about that specific file.

The sync log can be accessed from Settings > Sync > Sync activity > View.

rongyi · February 26, 2022, 6:30am

Sure, but reproducing it on purpose might be difficult, as I’m not 100% sure of the cause / condition for this issue to happen. Meanwhile @Licat can I get a build of Sync that does not clear the sync log on exit (iOS return to home)? Most of the time when I realize the data loss, I’ve already open / quit the app multiple times.

Just tried to intentionally cutting off the internet with Airplane Mode, but Obsidian Sync seems to handle the merge just fine. And in this case there is a notice of two versions being merged: “modified externally, merging changes automatically.” I did not see that notice when the override happens.

Licat · February 26, 2022, 6:44am

The reason I ask is because while you’ve only observed the overwrite a couple of times, it is difficult to actually count how many possible occurrences that did merge properly and did not cause an overwrite issue, to properly calculate the ratio. Here, I want to get an idea of how often you can reproduce the issue specifically when attempting to reproduce on purpose.

Meanwhile @Licat can I get a build of Sync that does not clear the sync log on exit (iOS return to home)? Most of the time when I realize the data loss, I’ve already open / quit the app multiple times.

Unfortunately the sync log is only saved in memory and so when the app closes it is lost. Currently I don’t plan on persisting the sync log to disk for performance reasons and to avoid accumulating disk usage.

I will continue to investigate and attempt to add more safety checks. In the meantime if you happen to spot the issue happening in front of you, please go and make a copy of the sync log and send it here.

chriswaterguy · February 27, 2022, 12:48pm

This happened to me (again) just now.

Sync log shows 3 saves to my “Shopping” file today (at 22:42, 22:43 and 22:43) on my Windows device. Then 40 min later this was overwritten by my Android device with the previous version from yesterday.

I’ve saved the sync log to a .txt file and am happy to share it privately – how do I do that?

WhiteNoise · February 27, 2022, 12:51pm

DM them to licat on discord or here.

What exactly is you sequence of operation?

chriswaterguy · February 27, 2022, 1:03pm

Do I need to be on a particular server? Can’t find you.

Not quite sure what you mean, beyond what I posted, sorry.

WhiteNoise · February 27, 2022, 1:04pm

when was the file created? on which device was the file edited and when? when where the changes synced?

chriswaterguy · February 27, 2022, 1:08pm

The file was created many months ago and has been edited many, many times on at 5 devices (probably just 2 devices in the last few weeks)… the recent changes will show in the log file that I will send.

The problematic changes are those described above:

Sync log shows 3 saves to my “Shopping” file today (at 22:42, 22:43 and 22:43) on my Windows device. Then 40 min later this was overwritten by my Android device with the previous version from yesterday.

chriswaterguy · February 28, 2022, 2:23am

This is absolutely not my domain, but FWIW…

At this stage of development, good log files must be very valuable. Solving serious issues like this one is probably worth a very slight performance hit.
The app is already saving and updating text files constantly.
This would be one more text file.
Maybe they could be saved in weekly log files, and automatically deleted when they’re more than 2 weeks old. So they never become big.
There could be a switch to turn off the saving of log files, for people with older phones.

To do all of that might be a lot of work so I’m just putting it out there.

Also, including a diff check tool (or creating a one-click way to export sync versions to a diff check tool) would also help to identify sync bugs and fix any data loss. I would value this a lot. I’m not sure how much work that would take.

Btw, I can’t express how valuable Obsidian is to me . My suggestions are in the spirit of wanting it to be something I can safely rely on even more, and appreciating how awesome it already is.

WhiteNoise · February 28, 2022, 5:57pm

A change was introduced in 0.13.27 that may affect this.

rongyi · March 2, 2022, 7:28am

@Licat Happened again with Desktop 0.13.28 and Mobile 1.1.0(38). But this time I realized the data loss while writing on the iOS app and managed to save the Sync Log.

The file involved in the overwrite: log/daily/note/2022/03/2022-03-01.md

Sync Version History

The latest 1.69KB version from iPhone (containing one appended line) was not based on the previous version (1.86KB at 10:03PM), but based on an earlier one (1.64KB at 6:22PM). And Obsidian Sync did not realize the conflict, which essentially ignored the two updates from 8:58PM and 10:03PM.

Sync Log on iPhone:

Highlighted the log lines related to that file. No errors are reported in the log, and I didn’t get the “modified externally, merging changes automatically” notice either.

 2022-03-01 18:51 - Fully synced
 2022-03-01 22:59 - Connecting to server
 2022-03-01 22:59 - Connection successful. Detecting changes...
 2022-03-01 22:59 - Server pushed [LER0ever-MacMini-2.local] .obsidian/plugins/metadata-extractor/tags.json
 2022-03-01 22:59 - Server pushed [LER0ever-MacMini-2.local] .obsidian/plugins/metadata-extractor/allExceptMd.json
 2022-03-01 22:59 - Server pushed [LER0ever-MacMini-2.local] .obsidian/plugins/metadata-extractor/metadata.json
 2022-03-01 22:59 - Server pushed [LER0ever-MacMini-2.local] .obsidian/appearance.json
+2022-03-01 22:59 - Server pushed [LER0ever-MacMini-2.local] log/daily/note/2022/03/2022-03-01.md
 2022-03-01 22:59 - Downloading file .obsidian/plugins/metadata-extractor/tags.json
 2022-03-01 22:59 - Downloading complete .obsidian/plugins/metadata-extractor/tags.json
 2022-03-01 22:59 - Accepted .obsidian/plugins/metadata-extractor/tags.json
 2022-03-01 22:59 - Downloading file .obsidian/plugins/metadata-extractor/allExceptMd.json
 2022-03-01 22:59 - Downloading complete .obsidian/plugins/metadata-extractor/allExceptMd.json
 2022-03-01 22:59 - Accepted .obsidian/plugins/metadata-extractor/allExceptMd.json
 2022-03-01 22:59 - Downloading file .obsidian/plugins/metadata-extractor/metadata.json
 2022-03-01 22:59 - Downloading complete .obsidian/plugins/metadata-extractor/metadata.json
 2022-03-01 22:59 - Accepted .obsidian/plugins/metadata-extractor/metadata.json
+2022-03-01 22:59 - Downloading file log/daily/note/2022/03/2022-03-01.md
+2022-03-01 22:59 - Downloading complete log/daily/note/2022/03/2022-03-01.md
+2022-03-01 22:59 - Accepted log/daily/note/2022/03/2022-03-01.md
+2022-03-01 22:59 - Uploading file log/daily/note/2022/03/2022-03-01.md
+2022-03-01 22:59 - Upload complete log/daily/note/2022/03/2022-03-01.md
+2022-03-01 22:59 - Server pushed [LER0ever-iPhone] log/daily/note/2022/03/2022-03-01.md
 2022-03-01 22:59 - Fully synced

Sync Log on Mac:

Looks like it just blindly accepted the server side version without any warnings.

2022-03-01 21:59 - Fully synced
2022-03-01 22:03 - Uploading file log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:03 - Upload complete log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:03 - Server pushed [LER0ever-MacMini-2.local] log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:03 - Fully synced
2022-03-01 22:59 - Server pushed [LER0ever-iPhone] log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:59 - Downloading file log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:59 - Downloading complete log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:59 - Accepted log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:59 - Fully synced

Hope this helps with debugging the issue, and please let me know if you need any additional information. I still have no luck in reproducing it on purpose, probably because the condition required to trigger the issue is not what I thought before (version resolving / race condition). This time I had a perfect network connectivity, with both devices in door and connected to WiFi. The Obsidian (desktop) are updated to 0.13.28, so the attempted fix in 0.13.27 is in place, at least for the Mac.

WhiteNoise · March 2, 2022, 2:12pm

The fix wasn’t specific to this issue and it hasn’t been released for mobile yet.

It’s possible that some plugins or some other software on your devices is changing the file behind your back.

Run obsidian everywhere without plugins and let us know if it happens and think if you have any other software operating on these files (like another sync service).

Licat · March 2, 2022, 3:59pm

Our other report from @chriswaterguy mentioned he has the Calendar and Tasks plugin, both of which were installed in @rongyi 's original debug log as well.

Is it possible that one of them (I feel like it could be the Tasks plugin) that’s causing this behavior?

To explain a bit further, the log seen here is identical to the one posted by @chriswaterguy in behavior - Sync seems to download the file when it comes online, then very quickly upload a different version which is smaller.

I’ve gone back to audit the sync code and I don’t see any possibility of the code hanging on to an older version after it has downloaded the latest version. The only possibility here is if a plugin loaded the file in question before Sync downloaded the new version, and then wrote back the old version after Sync has completed the download, which causes the rollback.

rongyi · March 2, 2022, 9:29pm

Thanks for the update @WhiteNoise @Licat

It’s possible that some plugins or some other software on your devices is changing the file behind your back. Run obsidian everywhere without plugins and let us know if it happens and think if you have any other software operating on these files (like another sync service).

I can rule out third party software changing the file, since this happens on iOS. And I’m not using any other sync service besides Obsidian Sync.

I probably cannot experiment with no plugins at all, but I can try to disable some of them.

Is it possible that one of them (I feel like it could be the Tasks plugin) that’s causing this behavior?

I’ll start a plugin bisect by disabling some of the plugins that I don’t actively use / need including Tasks.

Annotator
File Tree Alternative
Graph Analysis
Highlightr
Linter
Tasks

I’ve disabled the above plugins, and will report back if the overwrite happens again.

rongyi · March 3, 2022, 8:06pm

Happened again today with the above plugins disabled… So it’s not caused by Tasks plugin. I believe I have some clue about what’s causing the issue, but have to double-check.

File: log/daily/note/2022/03/2022-03-03.md

Sync Version History

Sync Log on iPhone

2022-03-03 11:54 - Downloading file .obsidian/plugins/metadata-extractor/allExceptMd.json
2022-03-03 11:54 - Downloading complete .obsidian/plugins/metadata-extractor/allExceptMd.json
2022-03-03 11:54 - Accepted .obsidian/plugins/metadata-extractor/allExceptMd.json
2022-03-03 11:54 - Downloading file .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:54 - Downloading complete .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:54 - Accepted .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:54 - Downloading file log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:54 - Downloading complete log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:54 - Accepted log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:54 - Fully synced
2022-03-03 11:55 - Uploading file log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Upload complete log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Server pushed [LER0ever-iPhone] log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Fully synced
2022-03-03 11:55 - Connecting to server
2022-03-03 11:55 - Connection successful. Detecting changes...
2022-03-03 11:55 - Fully synced

Sync Log on Mac

2022-03-03 11:53 - Server pushed [LER0ever-MacMini-2.local] .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:53 - Merging conflicted file .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:53 - Downloading file .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:53 - Downloading complete .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:53 - Accepted .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:54 - Uploading file log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:54 - Upload complete log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:54 - Server pushed [LER0ever-MacBookAir] log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Server pushed [LER0ever-iPhone] log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Downloading file log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Downloading complete log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Accepted log/daily/note/2022/03/2022-03-03.md

Enabled Plugins

	Plugins installed: 28
	Plugins enabled: 19
		1: Advanced Tables
		2: Calendar
		3: Quick Switcher++
		4: Natural Language Dates
		5: Custom window title
		6: QuickAdd
		7: Admonition
		8: Excalidraw
		9: Minimal Theme Settings
		10: Advanced URI
		11: Metadata Extractor
		12: Dataview
		13: Diagrams
		14: Style Settings
		15: Open vault in VSCode
		16: Local images
		17: Contextual Typography
		18: Global Hotkeys
		19: Templater

@chriswaterguy do you have the enabled plugin list when this issue happens?

rongyi · March 3, 2022, 10:36pm

@Licat It is now 100% consistently reproducible in my case. Bug report filed to the corresponding plugin repository: [BUG] External file updates lost during Capture prompt · Issue #210 · chhoumann/quickadd · GitHub . Could really use some suggestions on how to fix that either on the Sync side or the plugin side.

In summary, it turns out a plugin (Quickadd) has a long running operation (prompt for user typing). It reads the file before that, and writes the updated file afterwards based on the version it read. The Obsidian Sync updates the same file during that prompt period.

this.app.vault.read(file)
const newFileContent: string = await this.formatter.formatContentWithFile(content, this.choice, fileContent, file);
    // the above function waits for user input. 
this.app.vault.modify(file, newFileContent);

I’m not sure what’s the best practice here, deferring the file read operation to after the user input finishes seems like a solution, but I don’t know much about Obsidian’s API / plugin dev.