Find similar notes (Python script)

[macedotavares]

@raudaschl, in fact, I’m moving this to the Plug-in Ideas section. We may get some useful input and traction there. Thanks!

[raudaschl]

That’s pretty awesome.
Having used this for a few days now for personal and work the value is really high.
I’ve discovered so many connections I have forgotten.
I feel the obsidian search is a bit weak for finding relevant notes based on keywords.
This fills part of that gap.

[raudaschl]

So I have been able to experiment with this on and off the last few weeks.

I’ve made some amendments to the script and here is what I’ve learned.

1. Added Lemmatisation but did not seem to make much difference in the relevancy of the results with Jaccard Similarity
2. However, I did introduce TFxIDF cosine similarity and that not only improved the recall of the number of results but when combined with the lemmatisation made the results far more relevant
3. I then blended the TFxIDF score with the keyword similarity score and adjusted the score weightings
4. The results I’m seeing is better recall (now seeing results for some items where as before I was seeing none) and improved relevancy

I’ve attached the code below which allows you to compare the ranking of the two methods side by side.

The new script does require new libraries and is a bit more resource-heavy.

One thing that surprised me is how this can tell me how novel a new note or idea is!
I can see an application for that when ideating, refactoring or reflecting.

I might set up some experiments around entity extraction next to see if this improved the relevancy further.

In terms of productionizing I’m not sure how to move this forward. The Obsidian to Anki Plugin seems to have found a way to blend python and nodejs.
It may be worth looking into that in more detail.

similarity.py.zip (2.4 KB)

[macedotavares]

Oh, this has got to be good

I’ll take it for a spin! Thanks!

[cristian]

Hi @marcelotavares, @raudaschl !, nice scripts you have there :D.

Last month I played with the library networkx to get some vault metrics and things like Pagerank (GitHub - cristianvasquez/prototype_05). I was happy with it.

We could have a ‘python lab’ plugin to make these experiments easier, providing an interface that executes the python scripts we’ve configured and return results in some usable way. We can use the plugin to experiment further and come up with better scripts.

In my case, I’m trying to figure out how:

• Inspect similar notes to the current one.
• Propose new links while I’m writing
• Find two notes that I should merge into one.

I think python scripts are a flexible way to do it. I also have a feeling that the best scripts will depend on the contents of the vault.

[raudaschl]

Hey @cristian

I love this idea of a python plugin!
How do we even go about doing such a thing?
I’ve been having issues getting node to even run a python script.

On the points you are trying to figure out I think the above similarity scripts are a good starting point.

Another way this could be achieved is via topic modelling.
I’ve been experimenting with creating models based on bi and trigrams within the notes - so each topic is based on keywords that commonly associated with eachother from the entire vault.

It’s based on this script I wrote to identify trends in COVID-19 preprint papers. Good news is it works on any plain text, so its great for obsidian.

The sooner we make python easier to implement in Obsidian the sooner I think we will see really what networked thinking augmented with machine learning can do.

[cristian]

@raudaschl Let’s do it then!

@raudaschl

I’ve been having issues getting node to even run a python script.

I executed some python in the past from Obsidian, with this plugin GitHub - cristianvasquez/obsidian-snippets-plugin

For the lab, I’m trying to imagine something simple to use and extend.

At least I think that in a ‘lab’ plugin, you can associate commands to scripts, to experiment.

What do you imagine? are similar things out there?

[raudaschl]

This is impressive @cristian
To be honest I’ve only ever imagined having a dedicated button or typing in some command into the note to activate a python script and paste its output.

I see the python script as a function. It could either paste onto the page or possibly create a new note with the output.

The biggest pain point for me right now is an activation that uses the current note I’m working on as the source input without having to pull up terminal/VS code, drag the file path, run the script and study its output.

What would be cool is if you could have a place for scripts, link to them from the settings page and then assign a way of activating them. Go to the note I want to be the focus, hit an activation command and it automatically passes the contents of the note/file path of the note to the script and prints its output in the file.

What I’m aiming for is one script that could easily be debugged in a terminal but is instantly productionalisable within obsidian.

[macedotavares]

Hey, @raudaschl , @cristian !

I’ve been very silent lately (because work ), but I wanted to drop by and say this is discussion is looking really promising.

@raudaschl

The biggest pain point for me right now is an activation that uses the current note I’m working on as the source input without having to pull up terminal/VS code, drag the file path, run the script and study its output.

This was the hardest, when using Alfred. My workaround was to have the action paste a unique random string into the note and then run a shell find command with it as the input.

@raudaschl

What would be cool is if you could have a place for scripts, link to them from the settings page and then assign a way of activating them.

This would be awesome. I have another Alfred Workflow with a bunch of “text factories” that are basically recipes for sequential regex replacements, and I use them in a similar way: select text, type shortcut, select recipe from list, run recipe.

Doing part of the heavy lifting and having a better integration with Obsidian (i.e. without resorting to external apps) would be huge for me.

Again, I may be in radio silence, but I wanted to show my support and throw in my two cents.

Cheers!

[cristian]

@raudaschl

What would be cool is if you could have a place for scripts, link to them from the settings page and then assign a way of activating them. Go to the note I want to be the focus, hit an activation command and it automatically passes the contents of the note/file path of the note to the script and prints its output in the file.

I would like that using and sharing scripts with others is easy, just cloning a github repo. Perhaps the plugin could just detect different directories with scripts and add them as options.

I wonder how an interface to this would look like, how to sync etc. I can work a little on this in the weekend

[raudaschl]

@macedotavares

This was the hardest, when using Alfred. My workaround was to have the action paste a unique random string into the note and then run a shell find command with it as the input

Thats really clever.

recipes for sequential regex replacements, and I use them in a similar way: select text, type shortcut, select recipe from list, run recipe.

I would be interested in knowing more about these

@macedotavares

Doing part of the heavy lifting and having a better integration with Obsidian (i.e. without resorting to external apps) would be huge for me.

I fully agree - it would be the same for me.
I’m hoping that @cristian and I can come up with something.

[raudaschl]

@cristian

I would like that using and sharing scripts with others is easy, just cloning a github repo. Perhaps the plugin could just detect different directories with scripts and add them as options.

I think this would be amazing.

@cristian

I wonder how an interface to this would look like, how to sync etc. I can work a little on this in the weekend

Annoyingly I don’t think there is anything similar available we could use as a reference.
But as I find myself using python scripts more and more the value is only growing for me and I’m thinking of new ways it could be employed.
For example, as I generate lists of related notes I would like to create indexes from the tags associated with them or be easily able to jump from one related note to the next easily.

I could actually have a go at this starting with some user stories and possibly some mockups.

[raudaschl]

Hey all,

So I had a first pass at what the behaviour for such a plugin could be with some user stories.
Let me know what you think.

Obsidian Python Plugin User Stories

MVP

Python script using a single note

As a user
When i’m working on a note
I want to be able to run python scripts that will use that note as input
So that I see and potentially paste the output based on the title, text and metadata of that note

Passing variables to Python script

As a user
I would like the following variables passed to an activated python script while working on a note: filename, filepath, contents of the note, date created, date modified
So that I can easily use these variables in my script

Python script output

As a user
After I run a python script and get an output
I would like the output to be pasted into my active note
So that I can easily see the output or enhance my current active note

Python script management

As a user
I want to be able to add or remove python scripts used in my Obsidian vault
So that I can easily manage which scripts are available to use

Python script activation

As a user
I want to be able to trigger a python script using a keyword shortcut or button while working on an active note
So that I can easily see or paste the output of the python script without having the leave my note

V1

Select Python Version

As a user
I would like to configure which python version installed on my machine my script will use
So that I can easily experiment in the same environment using terminal or VSCode without having to use Obsidian all the time

Custom variables for Python Script

As a user
I would like to be able to pass custom variables to a script when it is run
So that I can change how I want my script to behave e.g. showDetails=False

---

I think this would be a great place to start.

[macedotavares]

@raudaschl

I would be interested in knowing more about these

Here you go!

The “recipes” folder contains all the rule sets that I’ve used so far. Some are from my time with Bear, but may retain some usefulness. Others are more specific and experimental. You can create or delete as many recipes as you want. They’re simply .txt files.

• Each recipe is a set of “find and replace” pairs that use regex.
• Inside recipes, each line is a transformation, with the first half being the “find” expression, a separator string (-> surrounded by a space on each side) and then the “replace” expression.
• Some expressions are commented (comments are whatever you write between a # and the end of the line). This feature was work in progress and may need to be redone or removed.

Let me know if you have any questions or comments.

Cheers!

[macedotavares]

This looks great!

@raudaschl

I would like the output to be pasted into my active note

There are 4 output behaviours that I can think of right now:

1. Replace the whole text
2. Replace only the text that was selected upon running the script
3. Insert the script output at cursor position.
4. View only, without modifying the note.

Would this be set by the plugin or an option for each script?

[raudaschl]

I really like how this automatically pulls an index from a list of scripts which is then easily run.
I can see why something of python scripts could be so useful

[raudaschl]

I had not considered these options actually.
The ability to specify in the plugin how you would like the output to be applied could be really interesting.
For the MVP I thought just appending the text at cursor point would be a good start.
But I especially like:

1. Replace the whole text ← Would like to head a use case for this one
2. Replace only the text that was selected upon running the script ← this could be really cool, not sure this could be expanded to only sending the selected text to the script as input

[trashhalo]

You might be able to embed your script into a plugin by using Pyodide — Version 0.17.0dev0 it appears to support nltk out of the box.

[cristian]

I wrote a proof of concept plugin to invoke experiments.

I only implemented a view called ‘result-list’ for similarity

To use it I configure the scripts in a JSON file:

In this case, there are two ‘similarity’ algorithms

1. dummy python script that returns random notes (obsidian-lab/randomScore.py at master · cristianvasquez/obsidian-lab · GitHub)
2. dummy javascript script that returns random notes

The results appear in a pane, with a score that appears at hover.

This is just a POC, I’m still trying to figure how to simplify this.

[cristian]

@raudaschl

Hey all,

So I had a first pass at what the behaviour for such a plugin could be with some user stories.
Let me know what you think.

Obsidian Python Plugin User Stories

Thanks for the stories!

What do you think of:

Inputs:
- The script id (mandatory).
- The vault path (mandatory).
- The active note name (optional).
- The current text selection (optional).
- The frontmatter data (optional. Contains date modified, tags etc).

Output options:
- A list → GUI:[Appears in a pane, as a clickable menu].
- An object → GUI:[Output to be shown in pane].
- A text → GUI:[Output to be pasted into my active note].

And an operation to ‘reindex’.

After experimenting a little with my scripts with the POC, I’m not so happy :/. I think I’ll try an implementation with a little HTTP-service to see how it goes.