Bring Open Source Enterprise Search tool to your vault

[dmk1913]

You want to be able to ask to your vault: Do I have something for x?. If x is somewhat of a noun or tag, then built-in search works great.

But why limit to keyword based search ? There are NLP tools out there that can possibly help. For e.g. if i want to find Do i have something for typing speed? and some note contains a mention of stenography without the word typing in it, i might want to review it.

I came across GitHub - deepset-ai/haystack: Haystack is an open source NLP framework that leverages pre-trained Transformer models. It enables developers to quickly implement production-ready semantic search, question answering, summarization and document ranking for a wide range of NLP applications. which seemed like perhaps it can be leveraged.

Any experts out there think something like this can work ? What challenges will i face if i attempt or any known previous attempts ?

Thanks!!

[SimonW]

There have been a few discussions about improving search, and so far the best plugin I have found in Omnisearch. It’s not NLP though, and I would love to see someone implement something like you suggest.

Related discussions:

Data Donation request

Hey @acamara , Big fan of the idea of suggesting links between notes. In fact when first exploring software like Roam and Obsidian I assumed this was already a feature. The closest I’ve seen to replicating this is a python script that tries to find the similarity between notes based on term frequency x inverse document frequency matching. Would be keen to hear what you think of it.

Sort search results by relevance (and what relevance is)

There were more suggestions as to how to present search results. I suggest having option to sort by relevance. It would solve more feature requests - see below. What relevance might mean - I suggest it encompasses following and in following order (which is of course subject of further discussion): filename (if searched word is found in filename, it is definitely highest relevance and should be presented first - it is default functionality in apps like Notational Velocity etc) - (more this for…

[scambier]

Omnisearch dev here. While I’m not really knowledgeable on NLP, I researched the subject a bit, and there are a few things I think I understand (feel free to correct me):

• NLP is all about context, so it needs pre-trained sets that take a lot of space to understand what you’re asking. We’re talking several GBs for English alone.
• It doesn’t look trivial at all to run this kind of application standalone, and even less as a plugin. There’s a reason it’s mostly online services and Jupyter notebooks.

Making a user-friendly consumer product on top of that is no small feat. If it were doable, it’d be a lot of work. And honestly if I were to do it, that would almost certainly be an online paid service.