Bring Open Source Enterprise Search tool to your vault

You want to be able to ask to your vault: Do I have something for x?. If x is somewhat of a noun or tag, then built-in search works great.

But why limit to keyword based search ? There are NLP tools out there that can possibly help. For e.g. if i want to find Do i have something for typing speed? and some note contains a mention of stenography without the word typing in it, i might want to review it.

I came across GitHub - deepset-ai/haystack: Haystack is an open source NLP framework that leverages pre-trained Transformer models. It enables developers to quickly implement production-ready semantic search, question answering, summarization and document ranking for a wide range of NLP applications. which seemed like perhaps it can be leveraged.

Any experts out there think something like this can work ? What challenges will i face if i attempt or any known previous attempts ?

Thanks!!

4 Likes

There have been a few discussions about improving search, and so far the best plugin I have found in Omnisearch. It’s not NLP though, and I would love to see someone implement something like you suggest.

Related discussions:

1 Like

Omnisearch dev here. While I’m not really knowledgeable on NLP, I researched the subject a bit, and there are a few things I think I understand (feel free to correct me):

  • NLP is all about context, so it needs pre-trained sets that take a lot of space to understand what you’re asking. We’re talking several GBs for English alone.
  • It doesn’t look trivial at all to run this kind of application standalone, and even less as a plugin. There’s a reason it’s mostly online services and Jupyter notebooks.

Making a user-friendly consumer product on top of that is no small feat. If it were doable, it’d be a lot of work. And honestly if I were to do it, that would almost certainly be an online paid service.

2 Likes