Multimodal semantic search

I’m the only one who has trouble finding notes in obsidian sometimes?
I think I’ll implement a plugin to do semantic search

Example: “show me the note about brown bears” (the note contains brown bear images and related text)

I started tinkering on this using

The code for the obsidian plugin is 80% done

(here is fake API)

My current advance regarding the semantic search part:

  • some issues on building the dataset of caption <-> text, so it’s not taking the whole vault when indexing, i.e. search results not great yet