Lilbee: a local AI search engine for your vault that runs its own models, crawls the web into your notes, and clicks back to every source

Share & showcase Plugins
,
1

Disclaimer

Is this project open source? Yes
Is this project completely free? Yes
Is this project vibe-coded beyond the author’s ability to comprehend how it works? No
Community Directory: lilbee

Hi all,

A bit about me first. I’ve been fascinated by local AI since I discovered Ollama a few years ago. It was really fun, but the novelty wore off fast when I realized I couldn’t do much with the models on their own.

So a few months ago I started exploring augmenting local models with a search engine over my own files (local RAG), and suddenly their utility became insanely powerful. Some of my earliest use cases were crawling horticulture forums and talking to them, and talking to my car manual, API documentation, Wikipedia, and my code. A friend of mine, a biomedical engineer, has been using it to talk to PhD research papers and video game manuals; he got further in Myst than he ever did before by talking to the strategy guide entirely offline. He also uses it to help configure his OS by feeding it the man pages and asking them offline. Another friend, a PhD chemist, used it to write a Python script with the Gwyddion SDK entirely offline, using DeepSeek with the user guide indexed in.

Once I saw my friends having fun too, I realized how useful this was I couldn’t stop, and I’ve been working on it obsessively, nearly daily, since late March. It finally feels worth sharing now that I’ve shown it largely works across multiple platforms. The result is lilbee.

As for why it exists: most tools in this space are heavy web apps you have to deploy and manage, and that’s just not my style. I’m primarily in the terminal, so I wanted a terminal-first tool to talk to my documents and code and I wanted a GUI integration for general Q&A and previewing source citations. I’d rather not run a full web stack or containers when a native executable or a Python package could accomplish the same thing. I was dissatisfied with the options and just wanted to talk to my documents from the terminal or something preexisting and simple to install. I have been a fan of Obsidian since 2021 and I chose Obsidian for this because it’s battle-tested and has a huge community, and because anyone already in Obsidian shouldn’t have to stand up a separate web app to talk to their stuff.

I don’t think there’s anything comparable right now that’s this lightweight and simple to onboard with in this space. Under the hood the plugin talks to a single executable that bundles a model manager, a search engine (local rag), and a web crawler (and more). I built the native model manager so the steps to get going stay contained to that one executable: from Obsidian it’s community store, install lilbee, and once you walk the setup wizard you have local AI over your vault.If you’re already using lilbee externally, you can just point the plugin at your external server.

If you want to see it before installing, the tutorial reel is the best overview and it’s linked in the project README at the top.

That’s the backstory. Here’s what it actually does:

It runs the models for you: browse a built-in catalog, pull one with a click, and it runs locally, so there’s no Ollama or LM Studio to install first and no account to make. It also works with your Ollama or LM Studio if you already have them, and a cloud model (OpenAI, Claude, Gemini) is there too, opt-in per role.

It crawls websites into your vault: point it at one page or a whole docs site and it saves them as markdown you can search and cite offline, even after the original changes or goes down. That’s how I talk to those horticulture forums and Wikipedia pages.

Every answer cites the file and line it came from, and you click the citation to land on the exact spot. That was the part I didn’t want to give up; I want to see where an answer came from, not just trust it.

It reads a lot more than markdown: PDFs, Office files, ebooks, spreadsheets, code, and scanned pages through OCR, 90+ document formats and 150+ programming languages. That’s how I read my car manual and search my own code.

You can run it managed or external: by default the plugin downloads and runs the engine for you, nothing to set up; if you already run lilbee yourself, just point the plugin at your own server.

Install: it is in the community plugin store. Settings → Community plugins → Browse → search “lilbee”. (Directory link is in the disclaimer at the top.)

Heads up: this downloads to your computer. lilbee is a local search engine with its own models, so the plugin fetches the engine (a few hundred MB) on first launch and the models you pick from the catalog (a few hundred MB up to several GB each) when you choose them. It’s all stored locally and runs on your machine.

It’s still early and in active development and I’d love feedback. Bug reports and ideas very welcome.

Thanks for reading!

1 Like
2

Question: what can it do better than a Hermes Agent with (py/js, etc.) skills (which Hermes now got a desktop app too and can otherwise be used via the Agent Client plugin via acp)?

3

Thanks for your response, I greatly appreciate it!

This isn’t competing with Hermes, it’s in a completely different space. lilbee is built on local AI first and has a privacy-first mindset: it runs the models on your own machine, indexes
your own files, and nothing has to leave your computer. Hermes is a cloud-connected agent that runs skills and keeps memory, so it’s in a different space altogether.

Where they differ most is document analysis. Hermes leans on the model for everything and leaves the embedding and document search to you, so it doesn’t index your files or retrieve from
them. lilbee is the retrieval stack itself:

  • a model manager: it downloads and runs the models for you, locally.
  • a search engine: it embeds and indexes your files into a local vector store, with reranking and answers that cite the file and line.
  • a web crawler: it saves whole sites into your vault as searchable, offline markdown.

So on the document side lilbee does way more, and it does it locally and privately. The agent side is coming too: agent integration built on opencode is in the works, so you’d get that
without giving up the local search engine underneath. And it already runs an MCP server, so an agent can use lilbee as its retrieval backend in the meantime.

I’m still a new user here so I can’t post clickable links yet, which is a shame because they’d make it much easier to see what lilbee actually is. If you want the full detail and the
tutorial reels, here are the spots (paste them into your browser):

  • Core engine on GitHub: github.com/tobocop2/lilbee
  • Obsidian plugin on GitHub: github.com/tobocop2/obsidian-lilbee
  • Full tutorial reels: obsidian.lilbee.sh/tutorial (plugin) and lilbee.sh/tutorial (engine)
4

A user recently reported that the plugin wasn’t working for them on Fedora. They shared their diagnostic report with me and it wound up exposing an issue with the managed lilbee server within the obsidian flatpak sandbox.

I’ve since corrected the issue as of lilbee 0.6.66b495

It’s not a plugin update, but a core server update, so I just wanted to provide some clarity here for anyone that has installed the plugin with a flatpak managed Obsidian.

Any users using the obsidian flatpak and managed mode with this plugin can update the lilbee server from within the plugin itself. See screenshot below and make sure your server is at least atv0.6.66b495

Screenshot 2026-06-12 at 6.22.20 PM
Screenshot 2026-06-12 at 6.22.20 PM2298×1902 440 KB

5

Quick follow-up since my account is new and I couldn’t post links earlier. Here they all are in one place so they’re easy to find.

Try it

lilbee (the engine behind it)

Source

Coming very soon: multi-GPU

Right now lilbee targets a single GPU and does all inference in process. I picked in-process early on for zero HTTP overhead, back when this was just a tool for my laptop. Then I started pointing it at much bigger jobs, so I rebuilt it to scale.

In the next week or two, ideally :sweat_smile:, lilbee goes from single GPU to a multi-GPU cluster, plus opencode integration. I’ve been on this for about a month. It’s already running large models in practice, for example minimax m2 spread across 3 GPUs, and I’m currently using my dev branch to rasterize and embed roughly 300 GB (around 3 million files) across multiple GPUs.

The update lets the plugin share work across every available GPU, and it also brings document-processing optimizations you’ll notice even on a single GPU. Tracking it here: Multi-GPU model serving and opencode integration by tobocop2 · Pull Request #267 · tobocop2/lilbee · GitHub

6

For transparency, the crawling feature had some stability issues on lower end machines, that I resolved in the latest release of the plugin and server.

* introduced a lightweight crawling mode.

By default, the plugin was crawling with javascript, which involved spinning up a headless chromium. This is awesome if you are targeting pages with dynamic content but it shouldn’t be the default for static pages. The default is now http only.

latest versions of the plugin and server appear to now be stable across all of the supported surfaces of the plugin. If using external server mode, make sure to update to the latest version and if using managed mode, just make sure the server shows that it’s at the latest version in the plugin settings.

Screenshot 2026-06-17 at 12.46.48 AM
Screenshot 2026-06-17 at 12.46.48 AM1920×1198 308 KB

Screenshot 2026-06-17 at 12.04.47 AM
Screenshot 2026-06-17 at 12.04.47 AM1920×1199 251 KB

7

I’m thrilled to share that multiple GPU support is coming very soon hopefully :crossed_fingers: and it’s insanely exciting for me to get this out as soon as possible. This is from my development branch at the moment but I’m anticipating that I will be ready to release this either this week!! Attached are real live demos.

Here’s a sneak peek where I’m running Qwen3-235B, a 235-billion-parameter model, split across three NVIDIA A100 80GB GPUs, with a Qwen3-Embedding-8B embedder mirrored onto every card.

In the first clip, lilbee indexes a codebase with all three GPUs embedding at once, and you can watch each card’s load move live. In the second, the 235B answers a question about that code and cites the exact source it came from.

At this point, I’ve effectively eliminated a need for ollama or lm studio for my needs and I’m excited to share this with others.

This plugin at the moment is just model management and Grounded Q&A on your stuff. However, https://lilbee.sh is also getting agent integrations for opencode and hermes, but for now I am keeping it to Encyclopedic Q&A because I that’s my preferred use case. I prefer keeping my agents in the terminal for the time being but an agent feature could come eventually.

My primary focus lilbee, which is what drives this plugin so I will do my best to establish parity between the two.

bigmodel-embed

bigmodel-gpu-bars

8

I use Lilbee for local searches and find it quite reliable.

1 Like
9

Thank you!!!

10

It’s been a while, and way way more than multiple GPU support has been added.

One thing first: all of this is in the dev builds right now.

Go to Settings, turn on “Include dev builds”, then install the newest build from the Server version picker.

This is temporary. Keeping the engine out of the stable release is what let me keep iterating without breaking the plugin for anyone, and I’m moving it all to stable very soon.

Pair a coding agent with your vault

This is something I just added to the plugin this week. Install a coding agent plugin like Claudian next to lilbee and pair them in one click. The agent uses the models lilbee is already running and reaches your library through lilbee, so it can crawl a page in, search your notes, answer with citations, and write a note or code back into your vault. No API key, no account, nothing leaves your machine.

The demo below used Qwen3 Coder 30B. Zero data centers, full data sovereignty, and it cost me zero tokens. That isn’t even the newest model, which is kind of the point:

local LLMs are very useful once you augment them with a search engine, a web crawler, and an agent.

The engine

The engine has been drastically improved and can scale way beyond your laptop. Ingest runs across every GPU in the machine instead of pinning one. NVIDIA, AMD and Intel all work, and there’s a view showing what’s loaded where and how busy each card is. The server can run on a different machine too, so you can point Obsidian at a real box and keep working on your laptop.

Extraction

The extraction layer has also improved and you should see an order of magnitude improvement in extraction speed from your documents. Tables and layout survive now, and
scanned pages go through OCR.

Search

The retrieval pipeline has been drastically improved too, so you should see better search results. I graded it properly this time and the numbers hold up. If you’re curious about the methodology it’s here:

I’ve ingested tens of millions of files into lilbee so far and it holds up at that size.

The wiki

I’ve finally stabilized the wiki as well. It reads your vault and writes a page for every concept and person that keeps coming up, with links back to the notes it came from. The pages are ordinary markdown notes with [[wiki links]] between them, so they show up in your graph and backlinks like anything else you wrote. Ask for a subject it hasn’t covered and it writes that page on the spot. The wiki is now pretty cool, but it’s very GPU demanding so just a fair warning there.
The wiki demo can be viewed here:

Plugin

Conversations are saved now, so you can go back and pick one up later, and long ones keep going instead of falling over. The model list pulls live picks from Hugging Face instead of a list I kept by hand. The server has been stabilized in general as well.

I’m probably missing a few more things, but here’s the gist: the engine got much faster and much bigger, and the plugin is now largely caught up.

I will formally release the engine soon, which will remove that ‘dev builds’ step I mentioned in the beginning.

11

It’s been a while, and way way more than multiple GPU support has been added.

One thing first: all of this is in the dev builds right now.

Go to Settings, turn on “Include dev builds”, then install the newest build from the Server version picker.

This is temporary. Keeping the engine out of the stable release is what let me keep iterating without breaking the plugin for anyone, and I’m moving it all to stable very soon.

Pair a coding agent with your vault

This is something I just added to the plugin this week. Install a coding agent plugin like Claudian next to lilbee and pair them in one click. The agent uses the models lilbee is already running and reaches your library through lilbee, so it can crawl a page in, search your notes, answer with citations, and write a note or code back into your vault. No API key, no account, nothing leaves your machine.

The demo below used Qwen3 Coder 30B. Zero data centers, full data sovereignty, and it cost me zero tokens. That isn’t even the newest model, which is kind of the point:

local LLMs are very useful once you augment them with a search engine, a web crawler, and an agent.

The engine

The engine has been drastically improved and can scale way beyond your laptop. Ingest runs across every GPU in the machine instead of pinning one. NVIDIA, AMD and Intel all work, and there’s a view showing what’s loaded where and how busy each card is. The server can run on a different machine too, so you can point Obsidian at a real box and keep working on your laptop.

Extraction

The extraction layer has also improved and you should see an order of magnitude improvement in extraction speed from your documents. Tables and layout survive now, and
scanned pages go through OCR.

Search

The retrieval pipeline has been drastically improved too, so you should see better search results. I graded it properly this time and the numbers hold up. If you’re curious about the methodology it’s here:

I’ve ingested tens of millions of files into lilbee so far and it holds up at that size.

The wiki

I’ve finally stabilized the wiki as well. It reads your vault and writes a page for every concept and person that keeps coming up, with links back to the notes it came from. The pages are ordinary markdown notes with [[wiki links]] between them, so they show up in your graph and backlinks like anything else you wrote. Ask for a subject it hasn’t covered and it writes that page on the spot. The wiki is now pretty cool, but it’s very GPU demanding so just a fair warning there.
The wiki demo can be viewed here:

Plugin

Conversations are saved now, so you can go back and pick one up later, and long ones keep going The model list pulls live picks from Hugging Face instead of a list I kept by hand. The server has been stabilized in general as well.

I’m probably missing a few more things, but here’s the gist: the engine got much faster and much bigger, and the plugin is now largely caught up as it can be and now you can use lilbee’s models, search engine, and web crawler behind an agent directly from Obsidian if you pair with Claudian.

I will formally release the engine soon, which will remove that ‘dev builds’ step I mentioned in the beginning.

12

Thank you!!

1 Like
13

I’ve promoted my engine from a dev release to a stable release, so everyone that uses this plugin will now automatically get the latest and greatest engine.

For better context on what’s happened with the engine, see here: Release v0.6.90b420 · tobocop2/lilbee · GitHub

Also, this part “Go to Settings, turn on “Include dev builds”, then install the newest build from the Server version picker.” is no longer relevant now that the engine has been promoted to a stable version so anyone using this plugin as of today will get the latest stable version.

14

I’ve made a few quality of life improvements and bug fixes to the plugin today:

Plugin Github (0.6.92) :

Full detail in release notes:

One big improvement though is that the plugin now relies on my lilbee npm package

This package is a shim for the lilbee server. I’ve ported the binary management code from the plugin to this package. I received requests to publish to npm and I figured it was a good excuse to migrate the plugin in the process. It brings improved CPU and GPU detection to the plugin and also has enabled me to gut some custom binary management code.

This is an invisible change to the plugin for the most part but a nice bonus is that pre AVX2 CPU’s will now get the proper compatibility build of lilbee in the latest plugin version. The plugin didn’t support these old CPU’s previously.

All panels now are easily close-able and a few other bugs were fixed

I made it simpler to update the server when the server changes. There’s now an alert ribbon icon as well as a popup modal that can be easily dismissed or disabled.

Server Github:

Full detail in release notes

The server changes the most, far more than the plugin so I don’t post updates here on every server change, but the cause of several of the plugin bug fixes were server side.

The latest server supports a lot more hugging face models. The model estimation math that is used before downloading or loading a model was broken for a lot of hugging face models. I don’t own the model estimation math, I defer to a third party library. I’ve fixed all the bugs in the library and validated it on hundreds of model families from hugging face.

Additionally, I’ve improved the search engine. One of the bugs reported from a plugin user was a retrieval bug that was very visible on small models. It was specific to followup questions and one of the default server settings that detects followup questions and combines them with the previous chat history into the search query. This functionality is now disabled by default.

The windows build had several issues that are now resolved all related to how the executable was being packaged and distributed.

There were also a few inert settings that never made it to the server so those are fixed now.

In the past few weeks my main focus has been on extraction and search improvements as well as improving support for as much of the hugging face model catalog as possible.

Alot of my work has been in the model fit estimation library used by lilbee. So if you don’t see a lot of activity on the plugin side it’s because the plugin is much simpler and most of the active development goes to the server. The plugin is the GUI integration for the server, so I’m just mentioning this so it’s clear why the plugin isn’t as frequently updated as the server. I don’t post updates here often, but the server is very actively updated and maintained so don’t let my plugin inactivity fool you.

To folks using managed mode:

an older server will not automatically get the latest and greatest server, so make sure to go to

Community Plugins → lilbee → Settings → Server Version

and update to the latest server to get all of the above changes.

I’m also a maintainer at xberg, the extraction library behind lilbee’s search engine. There are a lot of extraction improvements and new file formats coming in the next few weeks as well. I don’t really want to spam here with every server update, but if you are interested, you can track lilbee on github

I will post here anytime there are meaningful updates impacting the plugin though.