Obsidian Sync incorrectly duplicates sections of files

[Sigrunixia]

I have a FR that could use some love.

Obsidian Sync: Add a notification warning when Sync Merge with conflicts happens

Use case or problem What’s happening now is that I’ll sit down at my computer after, say, having had Obsidian Sync turned off, and didn’t realize it, and I’ll work for awhile… and then realize sync was turned off (because I was messing around with folders names or whatever and shut down all of my sync services to avoid race conditions). I turn it on, and a bunch of files merge. Which, yay, no data loss! But those files get kind of borked visually (for example, “cssclass” goes in the middle now, …

[CawlinTeffid]

I gave it my heart years ago. But I’ll add my notification related comments from here. Thanks for digging that up!

[Neigh]

I’m chiming in as the frequency of this issue mangling my notes has gotten to the point where I am considering alternatives to ‘Sync’. Similar situation as others - always on desktop, and secondary use on mobile. It’s paramount that sync issues be communicated better to the user. I’m infinitely frustrated because today I just realised I’ve lost data from an old note that went unnoticed, and thus my faith in Obsidian Sync has taken a serious hit.

[CawlinTeffid]

For notification specially, see also Obsidian Sync: Add a notification warning when Sync Merge with conflicts happens.

[halkin]

I consistently have this issue as well working between my iPhone and Mac device. It usually syncs up fine up until the last few sentences I write, though sometimes more is missing. If I start writing on another device, the file is at risk of getting mangled up with duplicate text, and sometimes even data-loss.

It is so scary to risk losing or corrupting data on every device switch. This is the core feature of Sync, and it seems to be happening for so many people who pay for this feature. This bug report was opened in January. Is the issue being given appropriate attention by devs?

[halkin]

I lost half my daily notes today. I was able to recover them using version history, but this is a really serious issue for literally the main purpose of the Obsidian sync product that’s been open since the start of the year. If there is no plan to fix this soon, then I will unfortunately have to find another tool for syncing that does not corrupt my data.

[CawlinTeffid]

On the phone, do you often leave the app soon after making changes? That raises the chance of conflicts because Sync only runs when Obsidian is active. I use the CSS snippet in [Mobile] Make Sync icon always visible to help avoid that.

[halkin]

Yeah I definitely do leave the app after making changes. I don’t fully close the app though, so it’s still on in the background. That CSS snippet seems useful, I’ll check that out, thank you!

This issue doesn’t happen on other apps though, surely a background sync can be triggered on exit of the app?

[halkin]

Just tried it out, that CSS snippet is extremely useful thank you!

Looks like sync takes ages sometimes even for minor changes, which means waiting a while before closing the app. If sync time can’t be improved, then making sure it syncs in the background really seems critical.

[CawlinTeffid]

@halkin

I don’t fully close the app though, so it’s still on in the background

Obsidian doesn’t run in the background on mobile. In general, mobile operating systems freeze apps when they’re not visible. IOS is stricter about it, allowing a little time to finish things up when the app first goes to background. Android, I gather, may allow more, but I’m not sure Obsidian takes any advantage of it.

@halkin

Looks like sync takes ages sometimes even for minor changes

Anecdotally, Sync has been feeling slower for me recently (I don’t know if that’s generally true). Also bigger notes tend to take longer (even if the change is small), so that might be a factor.

[georgie]

I also have this problem. If I edit a file on my phone or iPad there is a high likelihood I get a duplicate section on my mac. I am also using sync.

[CawlinTeffid]

Since I switched to conflict files, I obviously don’t have this problem. But maybe this relates to the underlying problem: I’ve noticed that the conflict files almost never have any actual conflict. One is correct, and the other is an older version. I don’t know why Sync thinks there is an issue; I believe it has happened in cases where I didn’t edit the file on >1 device.

In the most recent case, on my laptop I made a large number of changes to a large note (updating the timestamp format in a project log). I paused Sync while doing it, tho possibly after having made some of the changes. Everything seems to have synced to my phone just fine, but I got a conflict file from my tablet. The conflict file was an older version, and the one with canonical name was the up to date version. I don’t think I edited the file on my tablet.

[yannik]

This thread is really concerning for me. A bug which causes data loss and duplicates is not fixed for 10 months (the bug probably exists even longer). Also, there is no notification in the Obsidian app if a merge conflict happens.

I thought about switching to Obsidian because it’s file-based and offers an official synchronization option, but this really holds me back.

People say you are in control of your data with file-based apps, but at the end, you have data loss because of unreliable synchronization options. This was the same with TiddlyWiki, which I used in the past.

This really got me thinking if I should rather use cloud apps. I may be more dependent on them and always need to be online, but at least I won’t lose any data because of synchronization errors.

[CawlinTeffid]

@yannik

This really got me thinking if I should rather use cloud apps. I may be more dependent on them and always need to be online, but at least I won’t lose any data because of synchronization errors.

Another option is to use third party sync your Obsidian notes. Sync your notes across devices - Obsidian Help

[yannik]

Thanks, but than I would be afraid that they’re even worse? I’m now using iCloud, but I read mixed opinions about it.

I just can’t get it why this has no priority for the Obsidian Team. What’s the point of using Obsidian and fill it with data if you can’t trust it to handle your data correctly?

[Sigrunixia]

Also, there is no notification in the Obsidian app if a merge conflict happens.

The sync log will indicate if a merge conflict happens, but no, no app-popups.

---

This issue is actually very hard to reproduce in testing environments, and it usually involves circumstances where syncing services have issues no matter which service it is.

No response does not mean we are not working on it or changes to address it, but it takes time and a lot of double and triple checking. There has been more than once we developed a potential helpful patch, and had to scrap it, because it messed up a lot of other functions.

Data loss is not Obsidian Sync exclusive, and this is why we advocate, no matter what you do, that you keep backups. It’s not us telling you this so we don’t have to develop fixes. It’s us telling you this because all it takes is a computer crash, a bad OS update, a hard drive failure, etc, to lose a lot of valuable work.

[WhiteNoise]

Let me add a quick note here. I plan to review all the sync threads in bug reports and overhaul the docs about this matter.

The issue of duplication, as far as we are able to reproduce it, happens when there are multiple independent changes to a note.
For example we begin with Note A that is synchronized in both device 1 and device 2.
If device 1 and device 2 both independently make changes to note A, without having the possibility to synchronize in between, then the automatic merge algorithm can produce undesirable outputs, often in the forum of duplicated sections.
However, there is no way really to fix this, because when there are multiple conflicting changes, only YOU (the user) know what’s right. No algorithm can merge them correctly.

There are currently two way to go about this:

1. You either let the automatic merge algorithm attempt. If it fails, you can manually look in the previous versions of Note A which are in the sync version history and file recovery version history.
2. Disable the automatic merging in case of conflict (We have recently introduce this). In this case, every time there is one of these “merge conflict with independent edits” an additional Note A-conflict will be generated and you have to manually review it Note A and Note A-conflict for changes..

[serious]

@WhiteNoise

The issue of duplication, as far as we are able to reproduce it, happens when there are multiple independent changes to a note.
For example we begin with Note A that is synchronized in both device 1 and device 2.
If device 1 and device 2 both independently make changes to note A, without having the possibility to synchronize in between, then the automatic merge algorithm can produce undesirable outputs, often in the forum of duplicated sections.

Just wanted to be clear that this differs from the behavior that I opened this bug report for, and that I’m still experiencing to this day. Notes are not modified independently by the user (me) on both devices. Notes are modified on only device 1, and Obsidian incorrectly invents changes on device 2 that are some combination of copied lines from the different versions of the note.

If this was clear and you’re just speaking to what you’ve been personally able to reproduce so far, then I’m being redundant, but I wanted to make it clear since this thread has seen a lot of replies and other people jumping on, some with the same or a very similar issue, and some maybe not.

[WhiteNoise]

One of the problem of this and the few other sync threads is that there is probably a superposition of different problems. So it’s difficult to scope them individually.

One of reasons we introduced the option to disable automatic merge in case of Independent edits is to figure out if some problems still appear when this is off.

So please, turn that off and let us know if you still have problems.

[lesiu]

I would hope here for a 3rd point:
Notify user that merge conflict happened, through the notification. Obsidian already notifies us when file cannot be synced due to other problems, let us know that it failed to merge files as well.

Cause like someone here pointed out before, sometimes you will not know about merge conflict until they view file in a week or two.