0-byte file (again) - note completely destroyed

  • macOS 12.1—12.6
  • Obsidian v0.13.19 (Installer 0.13.14) — 0.16.2 (Installer 0.15.9)

I just want to report that I just had another case of complete erasure of a note, zero byte file, all contents lost. Same as before.

It was a file I had been working on all afternoon, for about the last 5 hours. I had quit Obsidian a few minutes prior. Then I got a notification from my vault monitoring script that there was a zero byte file and lo and behold, there it was. I used snapshots to recover the last save point, and so only lost about 10 minutes worth of work.

I am still using the CM5 “legacy” editor I am using the new CM6 editor. No new plugins added in the recent weeks.

This is troubling, I don’t know what to do with this problem. I’ve reported it many times, and the bug always gets closed with “not enough info” or “reproduce it in the help vault” which is not practical as this is a very rare (luckily) and hard to reproduce bug.

I understand the default response to this is going to be “it must be a plugin”. But how come a safety net can’t be implemented in Obsidian itself that does the following:

  • when switching files, closing panes, or quitting Obsidian: "Do the contents of the buffer match the bits on the disk?" Y/N
  • if N → try to save again
  • if that fails, save the buffer contents to the “snapshots” so as to preserve the latest copy somewhere safe
  • pop up a warning and put something in a persistent log file (disk based so we can look at it after the fact, which is when we would usually realize this)

While that wouldn’t solve the bug, at least it would give us a chance of tracking it down and preserving the data. Why is this not possible?

8 Likes

A safety net is implemented. It’s the file recovery. You can lower the interval to 1 minute if you like.

If a file cannot be saved, you will already get popup warning.

I noticed in another thread that you get the file merge popup. Maybe some other program is the source of your problems?

What you are proposing will make the current merge approach of external changes not possible, because the file can be different due to a benign external change.

1 Like

You store your vault in iCloud. I’ve seen many other people with issues on iCloud. It works seamlessly for most, but I don’t trust it, and I’ve said that multiple times. I would never trust iCloud as a working directory for an app that frequently saves files.

Are you willing to try moving out of iCloud? If you are using iCloud as a backup, maybe consider something different. I wouldn’t trust iCloud sync as a full backup anyway. Or push a non-working copy of your vault to iCloud as a backup if you want, but work outside of it.

It’s just a shot in the dark, but this is my guess. I’m biased against it, and I don’t know for sure.

I’m really sorry you lost data! Even 1 minute of loss can really shake your confidence.

Thanks @WhiteNoise

Let’s assume you’re right, and this bug is caused by some other process on my system (I don’t edit my vault files in anything else but Obsidian btw). Why would Obsidian blindly “accept” a change that zeroes out the file while it’s running?

Here’s a screenshot of snapshots for this file:

The red outline shows the 0 byte version captured at 3:28pm. According to my time tracking software, I had not touched Obsidian since 3:17pm:

When I last used it, I was editing that file, and I am certain that its contents were intact.

I have a backup script that backs up my entire vault to a timestamped tar.zst archive once per hour:

Here’s the run from 2:28pm:

And here’s the one from 3:28pm with the 0-byte:

So somewhere between 3:17 and 3:19, something ate my file. It’s interesting because there’s a snapshot in Obsidian from 3:19 that is 3.42kb, so the zeroing must have occurred either simultaneously with that or within seconds after it.

Again I am just looking for some kind of straw to grasp onto to try and figure out what’s going on here.

@rigmarole No I don’t use iCloud anymore and have not for many months (since around May 21st).

since the original top post, I’ve updated my Obsidian to the 0.13.19 installer version with no .asar patch.

4 Likes

Followup to your other points @WhiteNoise :

It’s already at 1 minute.

I have never ever gotten one of these. What do they even look like? If the problem is occurring during Obsidian’s quit/cleanup routines, how would this even be shown?

I haven’t gotten a file merge conflict in a while and in this case most definitely did not.

I realize that, but again this could be made much safer with 2 changes:

  1. never just blindly accept a change which truncates the entire file - at least not without making an immediate safety snapshot regardless of the user’s snapshot interval
  2. a toggle to completely disable auto-external merging, as is being discussed over in Editor: Add a toggle to disable automatic merging of changes (non-obsidian sync)
1 Like

From Obsidian’s perspective, if it sees a file externally modified or “zeroed out” I don’t see any reasonable action for it to take other than to respect the external edit and I never ever want to incur it making needless extra reads and nag pop-ups. This extra machinery you are requesting seems to be a specific work-around for something that only maybe might solve your problem and it isn’t general.

I agree with the intuition that this is an iCloud issue. If you can’t move your working directory off iCloud, have you tried disabling “Optimize Mac Storage”? If you have already, still think this is an iCloud issue.

With this setting enabled, iCloud will unexpectedly zero out your files.

1 Like

@pmbauer did you even read my post? I am not using iCloud.

Let me restate a few points:

  • I am not using iCloud
  • I do not ever edit any of my vault files outside of Obsidian

The strength of Obsidian lies in it’s flexibility and its extensibility. There are better text editors. There are better databases, that can handle millions of files. There are better mobile apps. But Obsidian is king at being pretty good at all of those things and really great at adapting to just about any workflow that one can dream of.

So while you might not consider this important, I do. That’s fine, that’s great. We can both win.

Let’s be clear about what I’m asking for here: I merely want to get whatever bug that’s causing my data loss fixed. I’m not the only one experiencing this, as evidenced in the other threads. But as it’s elusive, and not even necessarily fixable (could very likely be an upstream bug) I’m just asking for a few toggle switches in the meantime to help avoid this critical data loss.

3 Likes

I’m sorry. I did miss the part where you weren’t using iCloud anymore.

I do think it’s presumptive demanding the devs take time to implement your prescribed workaround given that no one can reliably reproduce the issue; there is no guarantee it will even fix your specific problem.

I will keep the BR open. Maybe we can gather some more data point.

2 Likes

Thanks for keeping this open @WhiteNoise

To recap, here are my suggestions for mitigating the data loss and hopefully getting to the bottom of it:

  • when switching files, closing panes, or quitting, add an additional safety check to ensure the buffer was successfully saved to disk
  • if the check fails, then commit a new snapshot
  • use persistent disk based logging so we can go back at least 24-48 hours and see if there were any errors related to saving data even after quitting Obsidian
  • don’t automatically accept a file edit which truncates the entire file - at least not without making an immediate safety snapshot regardless of the user’s snapshot interval
  • add a toggle to completely disable auto-external merging
4 Likes

I just had this happen. My notes for the last 10 sessions and next 5 or so sessions of D&D planning all vanished into the zero-byte hole. That’s days of planning and info I couldn’t get back if this happened again and wasn’t lucky enough to have an instance of Obsidian still open with the files in their original state - this is the only notes app I’ve ever seen do this in any way, under any circumstances. These files were never edited outside of Obsidian.

I’m moving all of my notes out of Obsidian immediately and not using it again. Whatever is going on to save data is clearly not sufficient or reliable. Obsidian is much lighter-weight and cheaper than Devonthink, but Devonthink has never lost my data desite years of iCloud syncing.

I’m so sorry this happened! But before moving out of the vault, did you try the File Recovery plugin? There is a chance you can recover the notes.

(Also, I see your other replies in all the other threads. It sounds like you have an iCloud bug, and it might not be related to Obsidian at all. As far as I know, DevonThink is a database. It stores the data in a completely different way.)

1 Like

did you try the File Recovery plugin?

Yes, that eventually worked.

It sounds like you have an iCloud bug, and it might not be related to Obsidian at all.

It doesn’t really matter to me whose fault it is if I can’t trust data stored by Obsidian to be safe. If iCloud isn’t safe, then Obisidian shouldn’t offer it as a sync solution.

And honestly, from this thread, it sounds like iCloud sync, as used by Obsidian, is unreliable and might lead to data loss at any moment.

As far as I know, DevonThink is a database. It stores the data in a completely different way.

Sort of, it has a database but also syncs regular files. It has great support for Markdown, including backlinks as of recently I was pleased to discover.

2 Likes

Obsidian doesn’t offer it. The documentation simply lists the options available on each OS. Because of the way that Apple lock iOS down, iCloud is the only option apart from Obsidian Sync. But the files are just files that Obsidian works with, they don’t belong to Obsidian and other apps can edit them.

Many users say it works very well, but I wouldn’t trust iCloud myself.

3 Likes

I have the same bug, it has already happened twice during the last 24 hours. It’s zeroed two of my files. Fortunately, there’s a latest copy.

I was shocked when a very important work file was absolutely empty. Checked other files, they are ok for now.

What could it be? How to fix it and protect files?

I don’t use iCloud or other clouds for syncing, I use it locally.

My current version is v0.14.6

1 Like

Can you post your debug info? Run the command “Show debug info”.

1 Like

Sure

SYSTEM INFO:
Obsidian version: v0.14.6
Installer version: v0.12.3
Operating system: Windows 10 Home Single Language 10.0.19044
Login status: not logged in
Insider build toggle: off
Live preview: on
Legacy editor: off
Base theme: dark
Community theme: none
Snippets enabled: 3
Safe mode: off
Plugins installed: 57
Plugins enabled: 33
1: Advanced Mobile Toolbar
2: Banners
3: Better File Link
4: Calendar
5: Customizable Page Header and Title Bar
6: Customizable Sidebar
7: Cycle through Panes
8: Emoji Toolbar
9: Find unlinked files and unresolved links
10: Grandfather
11: Map View
12: Mousewheel Image zoom
13: Natural Language Dates
14: Paste URL into selection
15: Periodic Notes
16: Privacy Glasses
17: Prompt
18: TimeStamper
19: Templater
20: Review
21: Reading Time
22: Random Structural Diary
23: Journey
24: Frontmatter Tag Suggest
25: Paste image rename
26: Better Word Count
27: Hotkey Helper
28: Auto Card Link
29: Auto Link Title
30: Obsidian Tabs
31: Hover Editor
32: Homepage
33: Sidebar Expand on Hover

We never quite figure out what’s the cause of luckman’s problem.

Regarding @max_samusevich, I would begin with

  1. download and reinstall obsidian
  2. run it in safe mode for a while and see if it happens

BTW: do you get errors in console when this happens ctrl-shift-i?

How much free space do you have on your drive?
Are you using any sync/backup solution?

Just chiming back in that I still suffer with this. It happened to me again yesterday on 0.14.10 (installer ver 0.14.6)

I recovered via snapshots but it’s really disconcerting. I originally blamed Electron for this but now that Obsidian has caught up with the latest versions of that, I’m no longer so sure.

I feel like the only hope of ever getting this solved is for a debug build of Obsidian to be produced that outputs much more detailed logging around filesystem operations. Only Licat would know if that’s even possible.

No need to wait for licat; a debug build may or may not even show what would be useful.

This will though. If you selectively disable dtrace SIP and run the following after launching obsidian

dtruss -p $(ps aux | grep -i obsidian | grep -vi helper | grep -vi grep | awk '{print $2}')

… you should see a complete log of all the syscalls and files obsidian is touching. If you are getting empty files with no action of yours and obsidian is actually at fault, there should be a corresponding syscall for it when it happens.

My best guess is still there is something specific to your system or usage of Obsidian triggering the behavior your are observing, but if you are convinced Obsidian is the key culprit, what I just described is a good way to collect raw data that might help your case.