I just want to report that I just had another case of complete erasure of a note, zero byte file, all contents lost. Same as before.
It was a file I had been working on all afternoon, for about the last 5 hours. I had quit Obsidian a few minutes prior. Then I got a notification from my vault monitoring script that there was a zero byte file and lo and behold, there it was. I used snapshots to recover the last save point, and so only lost about 10 minutes worth of work.
I am still using the CM5 “legacy” editor I am using the new CM6 editor. No new plugins added in the recent weeks.
This is troubling, I don’t know what to do with this problem. I’ve reported it many times, and the bug always gets closed with “not enough info” or “reproduce it in the help vault” which is not practical as this is a very rare (luckily) and hard to reproduce bug.
I understand the default response to this is going to be “it must be a plugin”. But how come a safety net can’t be implemented in Obsidian itself that does the following:
when switching files, closing panes, or quitting Obsidian: "Do the contents of the buffer match the bits on the disk?" Y/N
if N → try to save again
if that fails, save the buffer contents to the “snapshots” so as to preserve the latest copy somewhere safe
pop up a warning and put something in a persistent log file (disk based so we can look at it after the fact, which is when we would usually realize this)
While that wouldn’t solve the bug, at least it would give us a chance of tracking it down and preserving the data. Why is this not possible?
A safety net is implemented. It’s the file recovery. You can lower the interval to 1 minute if you like.
If a file cannot be saved, you will already get popup warning.
I noticed in another thread that you get the file merge popup. Maybe some other program is the source of your problems?
What you are proposing will make the current merge approach of external changes not possible, because the file can be different due to a benign external change.
You store your vault in iCloud. I’ve seen many other people with issues on iCloud. It works seamlessly for most, but I don’t trust it, and I’ve said that multiple times. I would never trust iCloud as a working directory for an app that frequently saves files.
Are you willing to try moving out of iCloud? If you are using iCloud as a backup, maybe consider something different. I wouldn’t trust iCloud sync as a full backup anyway. Or push a non-working copy of your vault to iCloud as a backup if you want, but work outside of it.
It’s just a shot in the dark, but this is my guess. I’m biased against it, and I don’t know for sure.
I’m really sorry you lost data! Even 1 minute of loss can really shake your confidence.
Let’s assume you’re right, and this bug is caused by some other process on my system (I don’t edit my vault files in anything else but Obsidian btw). Why would Obsidian blindly “accept” a change that zeroes out the file while it’s running?
So somewhere between 3:17 and 3:19, something ate my file. It’s interesting because there’s a snapshot in Obsidian from 3:19 that is 3.42kb, so the zeroing must have occurred either simultaneously with that or within seconds after it.
Again I am just looking for some kind of straw to grasp onto to try and figure out what’s going on here.
@rigmaroleNo I don’t use iCloud anymore and have not for many months (since around May 21st).
since the original top post, I’ve updated my Obsidian to the 0.13.19 installer version with no .asar patch.
I have never ever gotten one of these. What do they even look like? If the problem is occurring during Obsidian’s quit/cleanup routines, how would this even be shown?
I haven’t gotten a file merge conflict in a while and in this case most definitely did not.
I realize that, but again this could be made much safer with 2 changes:
never just blindly accept a change which truncates the entire file - at least not without making an immediate safety snapshot regardless of the user’s snapshot interval
From Obsidian’s perspective, if it sees a file externally modified or “zeroed out” I don’t see any reasonable action for it to take other than to respect the external edit and I never ever want to incur it making needless extra reads and nag pop-ups. This extra machinery you are requesting seems to be a specific work-around for something that only maybe might solve your problem and it isn’t general.
I agree with the intuition that this is an iCloud issue. If you can’t move your working directory off iCloud, have you tried disabling “Optimize Mac Storage”? If you have already, still think this is an iCloud issue.
With this setting enabled, iCloud will unexpectedly zero out your files.
@pmbauer did you even read my post? I am not using iCloud.
Let me restate a few points:
I am not using iCloud
I do not ever edit any of my vault files outside of Obsidian
The strength of Obsidian lies in it’s flexibility and its extensibility. There are better text editors. There are better databases, that can handle millions of files. There are better mobile apps. But Obsidian is king at being pretty good at all of those things and really great at adapting to just about any workflow that one can dream of.
So while you might not consider this important, I do. That’s fine, that’s great. We can both win.
Let’s be clear about what I’m asking for here: I merely want to get whatever bug that’s causing my data loss fixed. I’m not the only one experiencing this, as evidenced in the other threads. But as it’s elusive, and not even necessarily fixable (could very likely be an upstream bug) I’m just asking for a few toggle switches in the meantime to help avoid this critical data loss.
I’m sorry. I did miss the part where you weren’t using iCloud anymore.
I do think it’s presumptive demanding the devs take time to implement your prescribed workaround given that no one can reliably reproduce the issue; there is no guarantee it will even fix your specific problem.
To recap, here are my suggestions for mitigating the data loss and hopefully getting to the bottom of it:
when switching files, closing panes, or quitting, add an additional safety check to ensure the buffer was successfully saved to disk
if the check fails, then commit a new snapshot
use persistent disk based logging so we can go back at least 24-48 hours and see if there were any errors related to saving data even after quitting Obsidian
don’t automatically accept a file edit which truncates the entire file - at least not without making an immediate safety snapshot regardless of the user’s snapshot interval
add a toggle to completely disable auto-external merging
I just had this happen. My notes for the last 10 sessions and next 5 or so sessions of D&D planning all vanished into the zero-byte hole. That’s days of planning and info I couldn’t get back if this happened again and wasn’t lucky enough to have an instance of Obsidian still open with the files in their original state - this is the only notes app I’ve ever seen do this in any way, under any circumstances. These files were never edited outside of Obsidian.
I’m moving all of my notes out of Obsidian immediately and not using it again. Whatever is going on to save data is clearly not sufficient or reliable. Obsidian is much lighter-weight and cheaper than Devonthink, but Devonthink has never lost my data desite years of iCloud syncing.
I’m so sorry this happened! But before moving out of the vault, did you try the File Recovery plugin? There is a chance you can recover the notes.
(Also, I see your other replies in all the other threads. It sounds like you have an iCloud bug, and it might not be related to Obsidian at all. As far as I know, DevonThink is a database. It stores the data in a completely different way.)
It sounds like you have an iCloud bug, and it might not be related to Obsidian at all.
It doesn’t really matter to me whose fault it is if I can’t trust data stored by Obsidian to be safe. If iCloud isn’t safe, then Obisidian shouldn’t offer it as a sync solution.
And honestly, from this thread, it sounds like iCloud sync, as used by Obsidian, is unreliable and might lead to data loss at any moment.
As far as I know, DevonThink is a database. It stores the data in a completely different way.
Sort of, it has a database but also syncs regular files. It has great support for Markdown, including backlinks as of recently I was pleased to discover.
Obsidian doesn’t offer it. The documentation simply lists the options available on each OS. Because of the way that Apple lock iOS down, iCloud is the only option apart from Obsidian Sync. But the files are just files that Obsidian works with, they don’t belong to Obsidian and other apps can edit them.
Many users say it works very well, but I wouldn’t trust iCloud myself.
SYSTEM INFO:
Obsidian version: v0.14.6
Installer version: v0.12.3
Operating system: Windows 10 Home Single Language 10.0.19044
Login status: not logged in
Insider build toggle: off
Live preview: on
Legacy editor: off
Base theme: dark
Community theme: none
Snippets enabled: 3
Safe mode: off
Plugins installed: 57
Plugins enabled: 33
1: Advanced Mobile Toolbar
2: Banners
3: Better File Link
4: Calendar
5: Customizable Page Header and Title Bar
6: Customizable Sidebar
7: Cycle through Panes
8: Emoji Toolbar
9: Find unlinked files and unresolved links
10: Grandfather
11: Map View
12: Mousewheel Image zoom
13: Natural Language Dates
14: Paste URL into selection
15: Periodic Notes
16: Privacy Glasses
17: Prompt
18: TimeStamper
19: Templater
20: Review
21: Reading Time
22: Random Structural Diary
23: Journey
24: Frontmatter Tag Suggest
25: Paste image rename
26: Better Word Count
27: Hotkey Helper
28: Auto Card Link
29: Auto Link Title
30: Obsidian Tabs
31: Hover Editor
32: Homepage
33: Sidebar Expand on Hover
Just chiming back in that I still suffer with this. It happened to me again yesterday on 0.14.10 (installer ver 0.14.6)
I recovered via snapshots but it’s really disconcerting. I originally blamed Electron for this but now that Obsidian has caught up with the latest versions of that, I’m no longer so sure.
I feel like the only hope of ever getting this solved is for a debug build of Obsidian to be produced that outputs much more detailed logging around filesystem operations. Only Licat would know if that’s even possible.
… you should see a complete log of all the syscalls and files obsidian is touching. If you are getting empty files with no action of yours and obsidian is actually at fault, there should be a corresponding syscall for it when it happens.
My best guess is still there is something specific to your system or usage of Obsidian triggering the behavior your are observing, but if you are convinced Obsidian is the key culprit, what I just described is a good way to collect raw data that might help your case.
