I have never ever gotten one of these. What do they even look like? If the problem is occurring during Obsidian’s quit/cleanup routines, how would this even be shown?
I haven’t gotten a file merge conflict in a while and in this case most definitely did not.
I realize that, but again this could be made much safer with 2 changes:
never just blindly accept a change which truncates the entire file - at least not without making an immediate safety snapshot regardless of the user’s snapshot interval
From Obsidian’s perspective, if it sees a file externally modified or “zeroed out” I don’t see any reasonable action for it to take other than to respect the external edit and I never ever want to incur it making needless extra reads and nag pop-ups. This extra machinery you are requesting seems to be a specific work-around for something that only maybe might solve your problem and it isn’t general.
I agree with the intuition that this is an iCloud issue. If you can’t move your working directory off iCloud, have you tried disabling “Optimize Mac Storage”? If you have already, still think this is an iCloud issue.
With this setting enabled, iCloud will unexpectedly zero out your files.
@pmbauer did you even read my post? I am not using iCloud.
Let me restate a few points:
I am not using iCloud
I do not ever edit any of my vault files outside of Obsidian
The strength of Obsidian lies in it’s flexibility and its extensibility. There are better text editors. There are better databases, that can handle millions of files. There are better mobile apps. But Obsidian is king at being pretty good at all of those things and really great at adapting to just about any workflow that one can dream of.
So while you might not consider this important, I do. That’s fine, that’s great. We can both win.
Let’s be clear about what I’m asking for here: I merely want to get whatever bug that’s causing my data loss fixed. I’m not the only one experiencing this, as evidenced in the other threads. But as it’s elusive, and not even necessarily fixable (could very likely be an upstream bug) I’m just asking for a few toggle switches in the meantime to help avoid this critical data loss.
I’m sorry. I did miss the part where you weren’t using iCloud anymore.
I do think it’s presumptive demanding the devs take time to implement your prescribed workaround given that no one can reliably reproduce the issue; there is no guarantee it will even fix your specific problem.
To recap, here are my suggestions for mitigating the data loss and hopefully getting to the bottom of it:
when switching files, closing panes, or quitting, add an additional safety check to ensure the buffer was successfully saved to disk
if the check fails, then commit a new snapshot
use persistent disk based logging so we can go back at least 24-48 hours and see if there were any errors related to saving data even after quitting Obsidian
don’t automatically accept a file edit which truncates the entire file - at least not without making an immediate safety snapshot regardless of the user’s snapshot interval
add a toggle to completely disable auto-external merging
I just had this happen. My notes for the last 10 sessions and next 5 or so sessions of D&D planning all vanished into the zero-byte hole. That’s days of planning and info I couldn’t get back if this happened again and wasn’t lucky enough to have an instance of Obsidian still open with the files in their original state - this is the only notes app I’ve ever seen do this in any way, under any circumstances. These files were never edited outside of Obsidian.
I’m moving all of my notes out of Obsidian immediately and not using it again. Whatever is going on to save data is clearly not sufficient or reliable. Obsidian is much lighter-weight and cheaper than Devonthink, but Devonthink has never lost my data desite years of iCloud syncing.
I’m so sorry this happened! But before moving out of the vault, did you try the File Recovery plugin? There is a chance you can recover the notes.
(Also, I see your other replies in all the other threads. It sounds like you have an iCloud bug, and it might not be related to Obsidian at all. As far as I know, DevonThink is a database. It stores the data in a completely different way.)
It sounds like you have an iCloud bug, and it might not be related to Obsidian at all.
It doesn’t really matter to me whose fault it is if I can’t trust data stored by Obsidian to be safe. If iCloud isn’t safe, then Obisidian shouldn’t offer it as a sync solution.
And honestly, from this thread, it sounds like iCloud sync, as used by Obsidian, is unreliable and might lead to data loss at any moment.
As far as I know, DevonThink is a database. It stores the data in a completely different way.
Sort of, it has a database but also syncs regular files. It has great support for Markdown, including backlinks as of recently I was pleased to discover.
Obsidian doesn’t offer it. The documentation simply lists the options available on each OS. Because of the way that Apple lock iOS down, iCloud is the only option apart from Obsidian Sync. But the files are just files that Obsidian works with, they don’t belong to Obsidian and other apps can edit them.
Many users say it works very well, but I wouldn’t trust iCloud myself.
SYSTEM INFO:
Obsidian version: v0.14.6
Installer version: v0.12.3
Operating system: Windows 10 Home Single Language 10.0.19044
Login status: not logged in
Insider build toggle: off
Live preview: on
Legacy editor: off
Base theme: dark
Community theme: none
Snippets enabled: 3
Safe mode: off
Plugins installed: 57
Plugins enabled: 33
1: Advanced Mobile Toolbar
2: Banners
3: Better File Link
4: Calendar
5: Customizable Page Header and Title Bar
6: Customizable Sidebar
7: Cycle through Panes
8: Emoji Toolbar
9: Find unlinked files and unresolved links
10: Grandfather
11: Map View
12: Mousewheel Image zoom
13: Natural Language Dates
14: Paste URL into selection
15: Periodic Notes
16: Privacy Glasses
17: Prompt
18: TimeStamper
19: Templater
20: Review
21: Reading Time
22: Random Structural Diary
23: Journey
24: Frontmatter Tag Suggest
25: Paste image rename
26: Better Word Count
27: Hotkey Helper
28: Auto Card Link
29: Auto Link Title
30: Obsidian Tabs
31: Hover Editor
32: Homepage
33: Sidebar Expand on Hover
Just chiming back in that I still suffer with this. It happened to me again yesterday on 0.14.10 (installer ver 0.14.6)
I recovered via snapshots but it’s really disconcerting. I originally blamed Electron for this but now that Obsidian has caught up with the latest versions of that, I’m no longer so sure.
I feel like the only hope of ever getting this solved is for a debug build of Obsidian to be produced that outputs much more detailed logging around filesystem operations. Only Licat would know if that’s even possible.
… you should see a complete log of all the syscalls and files obsidian is touching. If you are getting empty files with no action of yours and obsidian is actually at fault, there should be a corresponding syscall for it when it happens.
My best guess is still there is something specific to your system or usage of Obsidian triggering the behavior your are observing, but if you are convinced Obsidian is the key culprit, what I just described is a good way to collect raw data that might help your case.
I know it’s a rare condition, but it’s not unique to my system. Others have reported it too. I would love to find the root cause of course. I suppose I could disable SIP and run dtruss for days, weeks, or longer to try to hit that bug…all the while eroding my system’s performance by tracing everything 24x7.
But I feel like this post I made above would be a better intermediate step and still be useful. Obsidian should be able to tell on its own accord whether file writes are successful. If it can’t even do that, then I don’t see how we can trust it for any serious data storage.
The dtruss command looks useful, it looks like it could be shortened to simply
dtruss -p $(pgrep ^Obsidian$)
or even possibly just
dtruss -n Obsidian
Why is the Renderer process not worth tracing? I thought I remember reading that a lot of the I/O was handled by that subprocess…
…all the while eroding my system’s performance by tracing everything 24x7.
Just dtrace obsidian. You actively use Obsidian 24x7?
Why is the Renderer process not worth tracing?
I never said the renderer process wasn’t worth tracing. Maybe develop a case of curiosity and try it? I’m not seeing what you and some others are seeing, but if I were and I was inclined to keep using obsidian, I would do what I could to figure out what was going on. I’m not sure waiting around for your suggestions to happen is a sound strategy.
I saw in another reply that you tend to make external edits? Even if Obsidian is somehow at fault here (far from proven by these anecdotes) it’s very likely some pattern of use is causing this. I’d want to know what it was.
Had this happen again yesterday to a large/detailed note. 0.14.15. Was able to recover it using file recovery. But there’s still a gremlin lurking somewhere deep below.
I believe I may have a clue now as to at least one of the triggers that can cause this complete destruction of a file.
If I remember correctly, what happened here is I switched to Obsidian and started typing immediately as soon as I saw the window appear.
The sequence I think I typed was ⌘N to create a new note, followed by the keys G-o-o-g-l-e. I was intending to type “Google Drive”
I noticed that what actually happened was:
I ended up on a new note with the filename “og” (chars 3 and 4 of what I typed) and no content. Here you can see that in my recent files list:
When I switched back to the note that I happened to be on when I pressed ⌘N (filename: “Archiware Pure free backup solution for VMware ESXi.md”) I saw it had been truncated to just the letters Go, screenshot:
Another case of data loss, this happened yesterday (Sun Jun 26 at around 10:55am). Obsidian 0.15.2.
I opened a note via URI (this may be a clue? something unique about launching or opening a note via the ?open URI command?). The note’s title is “Python - List Comprehensions.md”
Once Obsidian had opened, I was greeted with a completely empty file. I popped the console open to see if there were any errors, clues etc. Only saw this:
In the end, it appears the file’s contents were simply 100% replaced by null. I used File Recovery to restore the data and went on with my day.
Since then I tried setting up a script on a loop that simply opens this URI, pauses for a few seconds, quits Obsidian, and then repeats. I ran this script for about 10-15 minutes and could not reproduce it. Does this mean it didn’t happen? Obviously not!
@Licat what do you think—anything about it being triggered via a URI action that could lead somewhere?
