Idea for a plugin for Large File Management ("External Vault")

Plugins ideas
1

Hey Obsidian community,

I’ve been exploring ways to manage large file storage without overloading Obsidian Sync and wanted to share an idea I’ve envisioned. While using Obsidian Sync provides great convenience for versioning and consistency, its vault-size limitations and the performance impact of indexing numerous files can become restrictive. This is particularly true when handling many large non-Markdown files (PDFs, docs, ppts, maybe even audio and video clips).

The Problem:

  • Storage Limits: Obsidian Sync’s storage limits may not accommodate extensive files.
  • Cost: Services like iCloud offer a better $/GB ratio (e.g., 2TB for $10/month), but syncing changes of multiple small files can be slow and glitchy, that’s why Obsidian Sync is the best way out.
  • Performance: Indexing large numbers of files can reduce Obsidian’s performance.

The Proposed Solution:

Imagine a plugin that enables users to manage large files with an “external vault” system. This external vault could be linked to any folder on your computer, possible mapped to file services like Dropbox, iCloud, or Google Drive.

How It Would Work:

  1. Adding Files: User adds a file(s) with a plugin action. The file(s) is copied into the external storage location and marked as immutable (read-only).
  2. Creating a Reference Note: A corresponding Markdown note is generated in the primary Obsidian vault. This note mirrors the file’s metadata (e.g., file name, size, type), includes a tag like “#external-document”, and contains a link to the “real” file’s location, relative within the external vault.
  3. Editing: Users can edit the Markdown note within Obsidian as usual, maybe add some non-mutating annotations . The binary file remains unchanged, outside the main vault, preserving Obsidian Sync space and maintaining performance.
  4. Syncing: The immutable nature of these external files means they are only synced once, reducing sync delays. The Markdown “entry notes”, being text, remain light and easy to sync with Obsidian Sync.

Benefits:

  • Storage Efficiency: Large files don’t consume Obsidian Sync space, and theoretically the size is unlimited.
  • Integration Flexibility: Users can leverage the better cost-efficiency of external storage providers.
  • Annotations and Organization: Notes in Obsidian still provide an organized, annotated reference to the external files.

Cons:

  • iOS/macOS support: I doubt it will work smoothly.
  • Files immutability: At least, files should not be renamed by the user, otherwise the references will be broken. As for file content mutability, it’s possible, but conflict resolution can become a nightmare.

Questions for the Community:

  • Is there something similar already?
  • Any ideas how to make the concept better (smoother for users)?
  • Would such kind of plugin solve your challenges?
  • Are there technical or practical hurdles to implementing this idea?

Thanks in advance for your feedback!

2 Likes
2

I’m not totally sure if I would use this, but it is intriguing and appealing. The closest plugins I know are Remotely Save (sync whole vault using Dropbox, WebDAV, and other unsupported things) and Binary File Manager (create companion notes for attachments). Given that Remotely Save has already done the work of enabling connections to various services, it might make sense to explore if they would consider an alternate mode or variant plugin. (But note: some of Remotely Save’s features require payment to enable.)

On Apple devices it might be possible to implement some parts of this in a Shortcuts shortcut — like, share an image to the shortcut and it saves it in iCloud and makes a note in the vault with a link to the file (except if I remember right iCloud doesn’t provide cross-platform links other than web ones).