I’m sorry. I did miss the part where you weren’t using iCloud anymore.
I do think it’s presumptive demanding the devs take time to implement your prescribed workaround given that no one can reliably reproduce the issue; there is no guarantee it will even fix your specific problem.
I will keep the BR open. Maybe we can gather some more data point.
Thanks for keeping this open @WhiteNoise
To recap, here are my suggestions for mitigating the data loss and hopefully getting to the bottom of it:
I just had this happen. My notes for the last 10 sessions and next 5 or so sessions of D&D planning all vanished into the zero-byte hole. That’s days of planning and info I couldn’t get back if this happened again and wasn’t lucky enough to have an instance of Obsidian still open with the files in their original state - this is the only notes app I’ve ever seen do this in any way, under any circumstances. These files were never edited outside of Obsidian.
I’m moving all of my notes out of Obsidian immediately and not using it again. Whatever is going on to save data is clearly not sufficient or reliable. Obsidian is much lighter-weight and cheaper than Devonthink, but Devonthink has never lost my data desite years of iCloud syncing.
I’m so sorry this happened! But before moving out of the vault, did you try the File Recovery plugin? There is a chance you can recover the notes.
(Also, I see your other replies in all the other threads. It sounds like you have an iCloud bug, and it might not be related to Obsidian at all. As far as I know, DevonThink is a database. It stores the data in a completely different way.)
did you try the File Recovery plugin?
Yes, that eventually worked.
It sounds like you have an iCloud bug, and it might not be related to Obsidian at all.
It doesn’t really matter to me whose fault it is if I can’t trust data stored by Obsidian to be safe. If iCloud isn’t safe, then Obisidian shouldn’t offer it as a sync solution.
And honestly, from this thread, it sounds like iCloud sync, as used by Obsidian, is unreliable and might lead to data loss at any moment.
As far as I know, DevonThink is a database. It stores the data in a completely different way.
Sort of, it has a database but also syncs regular files. It has great support for Markdown, including backlinks as of recently I was pleased to discover.
Obsidian doesn’t offer it. The documentation simply lists the options available on each OS. Because of the way that Apple lock iOS down, iCloud is the only option apart from Obsidian Sync. But the files are just files that Obsidian works with, they don’t belong to Obsidian and other apps can edit them.
Many users say it works very well, but I wouldn’t trust iCloud myself.
I have the same bug, it has already happened twice during the last 24 hours. It’s zeroed two of my files. Fortunately, there’s a latest copy.
I was shocked when a very important work file was absolutely empty. Checked other files, they are ok for now.
What could it be? How to fix it and protect files?
I don’t use iCloud or other clouds for syncing, I use it locally.
My current version is v0.14.6
Can you post your debug info? Run the command “Show debug info”.
Sure
SYSTEM INFO:
Obsidian version: v0.14.6
Installer version: v0.12.3
Operating system: Windows 10 Home Single Language 10.0.19044
Login status: not logged in
Insider build toggle: off
Live preview: on
Legacy editor: off
Base theme: dark
Community theme: none
Snippets enabled: 3
Safe mode: off
Plugins installed: 57
Plugins enabled: 33
1: Advanced Mobile Toolbar
2: Banners
3: Better File Link
4: Calendar
5: Customizable Page Header and Title Bar
6: Customizable Sidebar
7: Cycle through Panes
8: Emoji Toolbar
9: Find unlinked files and unresolved links
10: Grandfather
11: Map View
12: Mousewheel Image zoom
13: Natural Language Dates
14: Paste URL into selection
15: Periodic Notes
16: Privacy Glasses
17: Prompt
18: TimeStamper
19: Templater
20: Review
21: Reading Time
22: Random Structural Diary
23: Journey
24: Frontmatter Tag Suggest
25: Paste image rename
26: Better Word Count
27: Hotkey Helper
28: Auto Card Link
29: Auto Link Title
30: Obsidian Tabs
31: Hover Editor
32: Homepage
33: Sidebar Expand on Hover
We never quite figure out what’s the cause of luckman’s problem.
Regarding @max_samusevich, I would begin with
BTW: do you get errors in console when this happens ctrl-shift-i?
How much free space do you have on your drive?
Are you using any sync/backup solution?
Just chiming back in that I still suffer with this. It happened to me again yesterday on 0.14.10 (installer ver 0.14.6)
I recovered via snapshots but it’s really disconcerting. I originally blamed Electron for this but now that Obsidian has caught up with the latest versions of that, I’m no longer so sure.
I feel like the only hope of ever getting this solved is for a debug build of Obsidian to be produced that outputs much more detailed logging around filesystem operations. Only Licat would know if that’s even possible.
No need to wait for licat; a debug build may or may not even show what would be useful.
This will though. If you selectively disable dtrace SIP and run the following after launching obsidian
dtruss -p $(ps aux | grep -i obsidian | grep -vi helper | grep -vi grep | awk '{print $2}')
… you should see a complete log of all the syscalls and files obsidian is touching. If you are getting empty files with no action of yours and obsidian is actually at fault, there should be a corresponding syscall for it when it happens.
My best guess is still there is something specific to your system or usage of Obsidian triggering the behavior your are observing, but if you are convinced Obsidian is the key culprit, what I just described is a good way to collect raw data that might help your case.
I know it’s a rare condition, but it’s not unique to my system. Others have reported it too. I would love to find the root cause of course. I suppose I could disable SIP and run dtruss for days, weeks, or longer to try to hit that bug…all the while eroding my system’s performance by tracing everything 24x7.
But I feel like this post I made above would be a better intermediate step and still be useful. Obsidian should be able to tell on its own accord whether file writes are successful. If it can’t even do that, then I don’t see how we can trust it for any serious data storage.
The dtruss command looks useful, it looks like it could be shortened to simply
dtruss -p $(pgrep ^Obsidian$)
or even possibly just
dtruss -n Obsidian
Why is the Renderer process not worth tracing? I thought I remember reading that a lot of the I/O was handled by that subprocess…
…all the while eroding my system’s performance by tracing everything 24x7.
Just dtrace obsidian. You actively use Obsidian 24x7?
Why is the Renderer process not worth tracing?
I never said the renderer process wasn’t worth tracing. Maybe develop a case of curiosity and try it? I’m not seeing what you and some others are seeing, but if I were and I was inclined to keep using obsidian, I would do what I could to figure out what was going on. I’m not sure waiting around for your suggestions to happen is a sound strategy.
I saw in another reply that you tend to make external edits? Even if Obsidian is somehow at fault here (far from proven by these anecdotes) it’s very likely some pattern of use is causing this. I’d want to know what it was.
Had this happen again yesterday to a large/detailed note. 0.14.15. Was able to recover it using file recovery. But there’s still a gremlin lurking somewhere deep below.
I believe I may have a clue now as to at least one of the triggers that can cause this complete destruction of a file.
If I remember correctly, what happened here is I switched to Obsidian and started typing immediately as soon as I saw the window appear.
The sequence I think I typed was ⌘N to create a new note, followed by the keys G-o-o-g-l-e. I was intending to type “Google Drive”
I noticed that what actually happened was:
I ended up on a new note with the filename “og” (chars 3 and 4 of what I typed) and no content. Here you can see that in my recent files list:
File Recovery for og showing 0 B file:
When I switched back to the note that I happened to be on when I pressed ⌘N (filename: “Archiware Pure free backup solution for VMware ESXi.md”) I saw it had been truncated to just the letters Go, screenshot:
When I opened File Recovery to see if I could get the data back, I noticed 2 snapshots of the destroyed note had been created:
first, this one (11kb) which seems to have all the data, with simply the letter G inserted at position 1, above the YAML frontmatter:
then this one (2 bytes) which basically deleted all text and replaced it with Go:
here is this same snapshot scrolled to the bottom
I hope this may be useful to @Licat to provide a clue as to how this could happen? It feels like a race condition.
Another case of data loss, this happened yesterday (Sun Jun 26 at around 10:55am). Obsidian 0.15.2.
I opened a note via URI (this may be a clue? something unique about launching or opening a note via the ?open URI command?). The note’s title is “Python - List Comprehensions.md”
Once Obsidian had opened, I was greeted with a completely empty file. I popped the console open to see if there were any errors, clues etc. Only saw this:
Next I went to File Recovery, which looked like this:
scrolled down…
In the end, it appears the file’s contents were simply 100% replaced by null. I used File Recovery to restore the data and went on with my day.
Since then I tried setting up a script on a loop that simply opens this URI, pauses for a few seconds, quits Obsidian, and then repeats. I ran this script for about 10-15 minutes and could not reproduce it. Does this mean it didn’t happen? Obviously not!
@Licat what do you think—anything about it being triggered via a URI action that could lead somewhere?
Please, don’t tag the devs.
Nothing I can think of honestly. Sounds like a race condition but I guess it is hard to repro. The object in console is just a debug statement that we’ve received the URL action.
Regardless, I guess it would be pretty hard to say what’s going on with all those plugins also loaded, but I can definitely look into what’s going on for the previous report with the og later.
Thank you Licat. Of course I don’t know what’s really going on under the hood of Obsidian. But would it be a lot of trouble to add a bit of debug logging around any method that attempts to replace the editor content with null or ''?
If we could even determine whether this behavior is caused by a plugin (I use a pretty small set to be honest) or from within an internal Obsidian function I think that would be helpful in tracking down a fix or workaround.
