#6

Sure, but reproducing it on purpose might be difficult, as I’m not 100% sure of the cause / condition for this issue to happen. Meanwhile @Licat can I get a build of Sync that does not clear the sync log on exit (iOS return to home)? Most of the time when I realize the data loss, I’ve already open / quit the app multiple times.

Just tried to intentionally cutting off the internet with Airplane Mode, but Obsidian Sync seems to handle the merge just fine. And in this case there is a notice of two versions being merged: “modified externally, merging changes automatically.” I did not see that notice when the override happens.

#7

The reason I ask is because while you’ve only observed the overwrite a couple of times, it is difficult to actually count how many possible occurrences that did merge properly and did not cause an overwrite issue, to properly calculate the ratio. Here, I want to get an idea of how often you can reproduce the issue specifically when attempting to reproduce on purpose.

Meanwhile @Licat can I get a build of Sync that does not clear the sync log on exit (iOS return to home)? Most of the time when I realize the data loss, I’ve already open / quit the app multiple times.

Unfortunately the sync log is only saved in memory and so when the app closes it is lost. Currently I don’t plan on persisting the sync log to disk for performance reasons and to avoid accumulating disk usage.

I will continue to investigate and attempt to add more safety checks. In the meantime if you happen to spot the issue happening in front of you, please go and make a copy of the sync log and send it here.

1 Like
#8

This happened to me (again) just now.

Sync log shows 3 saves to my “Shopping” file today (at 22:42, 22:43 and 22:43) on my Windows device. Then 40 min later this was overwritten by my Android device with the previous version from yesterday.

I’ve saved the sync log to a .txt file and am happy to share it privately – how do I do that?

#9

DM them to licat on discord or here.

What exactly is you sequence of operation?

#10

Do I need to be on a particular server? Can’t find you.

Not quite sure what you mean, beyond what I posted, sorry.

#11

when was the file created? on which device was the file edited and when? when where the changes synced?

1 Like
#12

The file was created many months ago and has been edited many, many times on at 5 devices (probably just 2 devices in the last few weeks)… the recent changes will show in the log file that I will send.

The problematic changes are those described above:

Sync log shows 3 saves to my “Shopping” file today (at 22:42, 22:43 and 22:43) on my Windows device. Then 40 min later this was overwritten by my Android device with the previous version from yesterday.

#13

This is absolutely not my domain, but FWIW…

  • At this stage of development, good log files must be very valuable. Solving serious issues like this one is probably worth a very slight performance hit.
  • The app is already saving and updating text files constantly.
  • This would be one more text file.
  • Maybe they could be saved in weekly log files, and automatically deleted when they’re more than 2 weeks old. So they never become big.
  • There could be a switch to turn off the saving of log files, for people with older phones.

To do all of that might be a lot of work :man_shrugging: so I’m just putting it out there.

Also, including a diff check tool (or creating a one-click way to export sync versions to a diff check tool) would also help to identify sync bugs and fix any data loss. I would value this a lot. I’m not sure how much work that would take.

Btw, I can’t express how valuable Obsidian is to me :heart:. My suggestions are in the spirit of wanting it to be something I can safely rely on even more, and appreciating how awesome it already is.

1 Like
#14

A change was introduced in 0.13.27 that may affect this.

1 Like
#15

@Licat Happened again with Desktop 0.13.28 and Mobile 1.1.0(38). But this time I realized the data loss while writing on the iOS app and managed to save the Sync Log.

The file involved in the overwrite: log/daily/note/2022/03/2022-03-01.md

Sync Version History

The latest 1.69KB version from iPhone (containing one appended line) was not based on the previous version (1.86KB at 10:03PM), but based on an earlier one (1.64KB at 6:22PM). And Obsidian Sync did not realize the conflict, which essentially ignored the two updates from 8:58PM and 10:03PM.

image
image920×1488 205 KB

Sync Log on iPhone:

Highlighted the log lines related to that file. No errors are reported in the log, and I didn’t get the “modified externally, merging changes automatically” notice either.

 2022-03-01 18:51 - Fully synced
 2022-03-01 22:59 - Connecting to server
 2022-03-01 22:59 - Connection successful. Detecting changes...
 2022-03-01 22:59 - Server pushed [LER0ever-MacMini-2.local] .obsidian/plugins/metadata-extractor/tags.json
 2022-03-01 22:59 - Server pushed [LER0ever-MacMini-2.local] .obsidian/plugins/metadata-extractor/allExceptMd.json
 2022-03-01 22:59 - Server pushed [LER0ever-MacMini-2.local] .obsidian/plugins/metadata-extractor/metadata.json
 2022-03-01 22:59 - Server pushed [LER0ever-MacMini-2.local] .obsidian/appearance.json
+2022-03-01 22:59 - Server pushed [LER0ever-MacMini-2.local] log/daily/note/2022/03/2022-03-01.md
 2022-03-01 22:59 - Downloading file .obsidian/plugins/metadata-extractor/tags.json
 2022-03-01 22:59 - Downloading complete .obsidian/plugins/metadata-extractor/tags.json
 2022-03-01 22:59 - Accepted .obsidian/plugins/metadata-extractor/tags.json
 2022-03-01 22:59 - Downloading file .obsidian/plugins/metadata-extractor/allExceptMd.json
 2022-03-01 22:59 - Downloading complete .obsidian/plugins/metadata-extractor/allExceptMd.json
 2022-03-01 22:59 - Accepted .obsidian/plugins/metadata-extractor/allExceptMd.json
 2022-03-01 22:59 - Downloading file .obsidian/plugins/metadata-extractor/metadata.json
 2022-03-01 22:59 - Downloading complete .obsidian/plugins/metadata-extractor/metadata.json
 2022-03-01 22:59 - Accepted .obsidian/plugins/metadata-extractor/metadata.json
+2022-03-01 22:59 - Downloading file log/daily/note/2022/03/2022-03-01.md
+2022-03-01 22:59 - Downloading complete log/daily/note/2022/03/2022-03-01.md
+2022-03-01 22:59 - Accepted log/daily/note/2022/03/2022-03-01.md
+2022-03-01 22:59 - Uploading file log/daily/note/2022/03/2022-03-01.md
+2022-03-01 22:59 - Upload complete log/daily/note/2022/03/2022-03-01.md
+2022-03-01 22:59 - Server pushed [LER0ever-iPhone] log/daily/note/2022/03/2022-03-01.md
 2022-03-01 22:59 - Fully synced

Sync Log on Mac:

Looks like it just blindly accepted the server side version without any warnings.

2022-03-01 21:59 - Fully synced
2022-03-01 22:03 - Uploading file log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:03 - Upload complete log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:03 - Server pushed [LER0ever-MacMini-2.local] log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:03 - Fully synced
2022-03-01 22:59 - Server pushed [LER0ever-iPhone] log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:59 - Downloading file log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:59 - Downloading complete log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:59 - Accepted log/daily/note/2022/03/2022-03-01.md
2022-03-01 22:59 - Fully synced

Hope this helps with debugging the issue, and please let me know if you need any additional information. I still have no luck in reproducing it on purpose, probably because the condition required to trigger the issue is not what I thought before (version resolving / race condition). This time I had a perfect network connectivity, with both devices in door and connected to WiFi. The Obsidian (desktop) are updated to 0.13.28, so the attempted fix in 0.13.27 is in place, at least for the Mac.

#16

The fix wasn’t specific to this issue and it hasn’t been released for mobile yet.

It’s possible that some plugins or some other software on your devices is changing the file behind your back.

Run obsidian everywhere without plugins and let us know if it happens and think if you have any other software operating on these files (like another sync service).

#17

Our other report from @chriswaterguy mentioned he has the Calendar and Tasks plugin, both of which were installed in @rongyi 's original debug log as well.

Is it possible that one of them (I feel like it could be the Tasks plugin) that’s causing this behavior?

To explain a bit further, the log seen here is identical to the one posted by @chriswaterguy in behavior - Sync seems to download the file when it comes online, then very quickly upload a different version which is smaller.

I’ve gone back to audit the sync code and I don’t see any possibility of the code hanging on to an older version after it has downloaded the latest version. The only possibility here is if a plugin loaded the file in question before Sync downloaded the new version, and then wrote back the old version after Sync has completed the download, which causes the rollback.

1 Like
#18

This post was flagged by the community and is temporarily hidden.

#19

Thanks for the update @WhiteNoise @Licat

It’s possible that some plugins or some other software on your devices is changing the file behind your back. Run obsidian everywhere without plugins and let us know if it happens and think if you have any other software operating on these files (like another sync service).

I can rule out third party software changing the file, since this happens on iOS. And I’m not using any other sync service besides Obsidian Sync.

I probably cannot experiment with no plugins at all, but I can try to disable some of them.

Is it possible that one of them (I feel like it could be the Tasks plugin) that’s causing this behavior?

I’ll start a plugin bisect by disabling some of the plugins that I don’t actively use / need including Tasks.

  • Annotator
  • File Tree Alternative
  • Graph Analysis
  • Highlightr
  • Linter
  • Tasks

I’ve disabled the above plugins, and will report back if the overwrite happens again.

3 Likes
#20

Happened again today with the above plugins disabled… So it’s not caused by Tasks plugin. I believe I have some clue about what’s causing the issue, but have to double-check.

File: log/daily/note/2022/03/2022-03-03.md

Sync Version History

image
image982×1366 112 KB

Sync Log on iPhone

2022-03-03 11:54 - Downloading file .obsidian/plugins/metadata-extractor/allExceptMd.json
2022-03-03 11:54 - Downloading complete .obsidian/plugins/metadata-extractor/allExceptMd.json
2022-03-03 11:54 - Accepted .obsidian/plugins/metadata-extractor/allExceptMd.json
2022-03-03 11:54 - Downloading file .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:54 - Downloading complete .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:54 - Accepted .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:54 - Downloading file log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:54 - Downloading complete log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:54 - Accepted log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:54 - Fully synced
2022-03-03 11:55 - Uploading file log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Upload complete log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Server pushed [LER0ever-iPhone] log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Fully synced
2022-03-03 11:55 - Connecting to server
2022-03-03 11:55 - Connection successful. Detecting changes...
2022-03-03 11:55 - Fully synced

Sync Log on Mac

2022-03-03 11:53 - Server pushed [LER0ever-MacMini-2.local] .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:53 - Merging conflicted file .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:53 - Downloading file .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:53 - Downloading complete .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:53 - Accepted .obsidian/plugins/metadata-extractor/metadata.json
2022-03-03 11:54 - Uploading file log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:54 - Upload complete log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:54 - Server pushed [LER0ever-MacBookAir] log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Server pushed [LER0ever-iPhone] log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Downloading file log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Downloading complete log/daily/note/2022/03/2022-03-03.md
2022-03-03 11:55 - Accepted log/daily/note/2022/03/2022-03-03.md

Enabled Plugins

	Plugins installed: 28
	Plugins enabled: 19
		1: Advanced Tables
		2: Calendar
		3: Quick Switcher++
		4: Natural Language Dates
		5: Custom window title
		6: QuickAdd
		7: Admonition
		8: Excalidraw
		9: Minimal Theme Settings
		10: Advanced URI
		11: Metadata Extractor
		12: Dataview
		13: Diagrams
		14: Style Settings
		15: Open vault in VSCode
		16: Local images
		17: Contextual Typography
		18: Global Hotkeys
		19: Templater

@chriswaterguy do you have the enabled plugin list when this issue happens?

1 Like
#21

@Licat It is now 100% consistently reproducible in my case. Bug report filed to the corresponding plugin repository: [BUG] External file updates lost during Capture prompt · Issue #210 · chhoumann/quickadd · GitHub . Could really use some suggestions on how to fix that either on the Sync side or the plugin side.

In summary, it turns out a plugin (Quickadd) has a long running operation (prompt for user typing). It reads the file before that, and writes the updated file afterwards based on the version it read. The Obsidian Sync updates the same file during that prompt period.

this.app.vault.read(file)
const newFileContent: string = await this.formatter.formatContentWithFile(content, this.choice, fileContent, file);
    // the above function waits for user input. 
this.app.vault.modify(file, newFileContent);

I’m not sure what’s the best practice here, deferring the file read operation to after the user input finishes seems like a solution, but I don’t know much about Obsidian’s API / plugin dev.

1 Like
#22

Ah yes, I see the bug report, and thank you!

There isn’t much we can do on Obsidian’s side unfortunately, but it should be fairly easy to fix it from the plugin’s perspective.

Most of the time, plugins are encouraged to read-process-write as quickly as possible to avoid conflicts. Reading and then waiting for user input is definitely before finally writing is definitely troublesome.

If a plugin requires user input, but must also read the file beforehand, we recommend reading again after user input just to verify the file hasn’t changed, and show a notice if the file has changed (or cancel the action) to avoid overwriting.

1 Like
#23

I’ve been noticing this behavior as well, primarily with my Daily note, which I start on my desktop usually, and then later in the day may add something from my phone.

I have the Daily notes plugin setup with a Template, and the setting to “Open daily note on startup” enabled.

I believe what’s happening is that my phone starts Obsidian up, and goes to open the Daily note. It doesn’t seem like the Sync has happened (or completed) yet, so it creates a new file using the defined Template. Sync then runs and sees a more recent version of the Daily note file, and uploads that, overwriting the version from my desktop.

I can mitigate for myself for the time being by not using the “Open daily note on startup” option for the time being.

So repo steps for me at least (happens every time):

  1. Make sure Daily note hasn’t been created yet on either device
  2. Have Daily notes plugin setup with “Template” set and “Open daily note on startup” turned on for both Desktop and Phone
  3. Open Obsidian only on Desktop, so that Daily Note is created
  4. Make edits to Daily note (and ensure it Syncs)
  5. Open Obsidian on phone
  6. Edits are overwritten with newly created Daily note from phone with Template

Sync Log

Desktop:

2022-03-04 16:22 - Connecting to server
2022-03-04 16:22 - Connection successful. Detecting changes...
2022-03-04 16:22 - Uploading file Journal/2022-03-04.md
2022-03-04 16:22 - Upload complete Journal/2022-03-04.md
2022-03-04 16:22 - Server pushed [MARS] Journal/2022-03-04.md
2022-03-04 16:22 - Fully synced
2022-03-04 16:22 - Uploading file Journal/2022-03-04.md
2022-03-04 16:22 - Upload complete Journal/2022-03-04.md
2022-03-04 16:22 - Server pushed [MARS] Journal/2022-03-04.md
2022-03-04 16:22 - Fully synced
2022-03-04 16:22 - Server pushed [Pixel 3] Journal/2022-03-04.md
2022-03-04 16:22 - Downloading file Journal/2022-03-04.md
2022-03-04 16:22 - Downloading complete Journal/2022-03-04.md
2022-03-04 16:22 - Accepted Journal/2022-03-04.md
2022-03-04 16:22 - Fully synced

Phone:

2022-03-04 16:22 - Connecting to server
2022-03-04 16:22 - Connection successful. Detecting changes...
2022-03-04 16:22 - Server pushed [MARS] Journal/2022-03-04.md
2022-03-04 16:22 - Merging conflicted file Journal/2022-03-04.md
2022-03-04 16:22 - Accepted Journal/2022-03-04.md
2022-03-04 16:22 - Uploading file Journal/2022-03-04.md
2022-03-04 16:22 - Upload complete Journal/2022-03-04.md
2022-03-04 16:22 - Server pushed [Pixel 3] Journal/2022-03-04.md
2022-03-04 16:22 - Fully synced

I can see that a merge is happening, but maybe because the file is considered to be newly created (and so all edits are “newer”), it uses that as the source of truth?

Obsidian Desktop v0.13.23
Obsidian Mobile (Android): v1.1.0

2 Likes
#24

that’s a different issue and will be fixed in 0.13.26 and 1.1.1

2 Likes
#25