I’ve been thinking about a personal project that I might build for myself, and I have a feeling that some of you might find it interesting as well. The idea is to create an Obsidian plugin that seamlessly integrates a Retrieval-Augmented Generation (RAG) workflow. Essentially, whenever you update or create notes in Obsidian, the plugin would automatically chunk and embed them into a vector database. Then, by connecting to the OpenAI API (or another LLM provider), you’d be able to ask questions directly inside Obsidian and get responses that are enriched with the context of your own notes. Imagine having your entire knowledge base at your fingertips in a conversational format—no more switching tools, manually copying text, or struggling to recall where you stored that bit of crucial information.
I’m planning to start working on this in 2025, but before diving in, I’d love to hear your thoughts. What features or functionalities would you find most valuable? How would you envision using such a tool in your daily workflow? Are there any pitfalls or potential improvements you can think of right away?
Your input will help shape the early stages of development, so feel free to share any ideas, critiques, or suggestions you might have. Thanks in advance!
I think it’s a very interesting idea. I think it would be very useful if we could include an option where the plugin could even consider a part of a vault as a rag section to respond to analyses about that other section.
I have been experimenting with RAG in Obsidian with existing community plugins like BMO Chat, Smart Connections and CoPilot.
I’m finding that these plugins (in their current state) are not quite as useful for RAG as I would like them to be. BMO can only reference one note and CoPilot is a bit clunky with Indexing. Smart connections is hit and miss.
I have been experimenting with Msty but it’s closed source, runs as a separate app and I can’t read my notes whilst having a conversation with them.
Experimenting further with N8N workflows into Pinecone/Supabase to use in a Web Viewer instance inside Obsidian but reading markdown files isn’t something a Vector database can do without some functions to convert them.
I would be awesome to find a solution that works in Obsidian.
If I discover anything further I will let you know
Having only a shallow understanding of RAG technology, I think the following points need attention:
Chunking in RAG is a major challenge. Poor chunking can lead to inaccurate and less usable data retrieval. Perhaps we can refer to some NLP chunking methods? Or consider using the latest Agentic RAG? (If possible, it would be great to support Chinese, as English chunking rules generally don’t apply well to Chinese text.)
How to encode images: Is the plan to put this on hold or OCR or integrate multimodal approaches? Alternatively, we could consider leveraging some ocr open-source projects such as GOT-OCR2.0 or olmocr.
If possible: The plugin should offer a configurable local LLM, , defining the scope of RAG encoding, which can be invoked within a separate dialog box or in editing page. Moreover, it should allow users to manually configure various roles and choose whether to use RAG.
Also, consider: Updating the RAG database after file updates
I was surprised that there is no solid RAG implementation for Obsidian yet. There is even a Lanchain connecter for Obsidian (Obsidian | 🦜️🔗 LangChain) But I am not sure who/which plugin is using it.
Well I think there’s sort of an issue that’s outside the scope of Obsidian. I can tell ya from working with these in industry the past two or so years, it’s hard to abstract into a plugin alone.
I think realistically, the plugin actually would be simple which is why there is many - if you code your RAG yourself in Langchain, Ollama, and Llama Index (and this sounds daunting for non AI ppl but it’s not all that scientific - people have “anecdotal” things they’ve seen which is all we have because we can’t really say which one’s best, or assume the GPU power of a user).
It lends itself strongly to a VERY intuitive personalization at a user PC level no matter what OS. Imagine your obsidian notes covering and udpdated with notes or LLM powered logs for errors. Once it happens a few times, you take that info, use your RAG, and boom, in the context of portions of your knowledge base, you have a tool.
I might look into this as I just put the RAG together. I only have a 3080 at home though so I have to find some cloud GPU.
But to drive home the point, RAG is best when you have full control. And when the word is used, it gets confusing to understand how much of the AI or RAG the tool is doing. And who is hosting, where that data is going to, and explaining that so people know their obsidian notes may or may not be protected when analyzed by an AI endpoint. I have tried to stay local. If anyone wants tips or to understand the sliding scale of full control requirements and benefits vs moving hybrid to cloud setup with API AI bots, please DM me!
