Hi, I’m building a plugin that relies on semantic search to propose content to the user. It requires indexing entire Obsidian vault by calculating embeddings for each note.
As an MVP, I chose to store embeddings in JSON file within the user’s vault. It worked reasonable well for small vaults, but on mobile devices with a larger number of files Obisian just crashes (like due to out-of-memory).
Have any of you figure out a clever way to overcome this? Perhaps an algorithm or storage strategy that avoids loading the entire index into memory at once?
Requirements:
- must work on mobile
- stores data in filesystem
- operates entirely locally