Multimodal semantic search

I’m the only one who has trouble finding notes in obsidian sometimes?
I think I’ll implement a plugin to do semantic search

Example: “show me the note about brown bears” (the note contains brown bear images and related text)


I started tinkering on this using

The code for the obsidian plugin is 80% done

(here is fake API)

My current advance regarding the semantic search part:

  • some issues on building the dataset of caption <-> text, so it’s not taking the whole vault when indexing, i.e. search results not great yet
1 Like

Hey, I advanced a bit, now using basically a fine-tuned sentence embeddings model fine-tuned in an unsupervised fashion on my vault, see details:

1 Like