Plugin to enable shell scripted functions from the community

[spacejunky]

I would like to throw an idea into the group and try and refine it a little before !suggesting it.

Outline:

A plugin to process certain “special” notes and run (their contents?, a configured piece of text) as a shell script, possibly appending the resulting stdout to the note

• when they are opened?, when obsidian starts?, periodically?

Motivation:

• many things to be done with pandoc conversion of foreign formats to md for import
• ditto for note export to other formats
• and some things which are now plugins could be done that way (like the Daily note with template, that admittedly was what started me thinking this way). This would open plugin/customisation possibilities for many more users, I suspect.
• it seems many people have scripts to process their vault in various ways
• at least in my office some standard reports were generated as markdown

Pro / Con

1. we can of course always just open a shell to do the same thing
   • but that’s a bit more work and slightly inconvenient
   • and if I have several such daily tasks the “automatic open what was open before” function when obsidian starts would deal with them all for no time investment. Very very convenient.
2. the scripts themselves are going to be OS dependent
3. people could get themselves in trouble (but see 1 above)

[spacejunky]

Just as an example, the request in one of the other topics for composite day notes could be done using this proposal (or a system with unix like utilities, which could also be windows with msys, and probably a Mac too?) by the following one liner:

 find Daily -name "*.md" -ctime -3 -exec echo \![[{}]] ';'|sort

Maybe some parameters for number of days and path but I’m sure you get the idea!

[SeanWcom]

I would love something like this. My team manages 60+ Kubernetes clusters and I’m constantly running ad-hoc ansible commands. But my memory is junk, which is why I write everything down. I would love to be able to not only write it down, but easily execute the command.

[scmwiz]

@spacejunky

Just as an example, the request in one of the other topics for composite day notes could be done using this proposal (or a system with unix like utilities, which could also be windows with msys, and probably a Mac too?) by the following one liner

Good idea! This can be done by setting up a cron-job to run at midnight. More useful with wysiwyg editing of embedded notes but even without that one can open up last-n-days in a separate pane.

[spacejunky]

@SeanWcom

I would love something like this. My team manages 60+ Kubernetes clusters and I’m constantly running ad-hoc ansible commands. But my memory is junk, which is why I write everything down. I would love to be able to not only write it down, but easily execute the command.

That’s an interesting idea. So far I hadn’t been thinking about the scripts doing anything other than working on the contents of my vault.

I guess for this case you would need execution to be triggered by some very explicit mechanism (hot key for example) rather than any of the otheroptions (at obsidian startup, periodically, on note open)?

[spacejunky]

@scmwiz

Good idea! This can be done by setting up a cron-job to run at midnight. More useful with wysiwyg editing of embedded notes but even without that one can open up last-n-days in a separate pane.

Yes it could. I must say though that evn after quite a few years I still have trouble remembering the crontab syntax. Some easier way of triggering things would be helpful (for me at least) …

And cron & ff is definitely *nix only, no?

[spacejunky]

An update to the initial proposal to give you something moreto chew on …

Version	Changes	Date
1	Original idea drop	20/May/2020
2	Flesh out with my overnight thoughts	21/May/2020

Outline:

A plugin to process certain “special” notes and run an identified part of their contents, a configured piece of text as a shell script.

“Special” notes are located anywhere in the Vault tree, distinguished by naming convention “”<some_name>.script.md". They are expected to create/update notes only in that folder, or sub-folders thereof (this is necessary so that we can avoid confusion if scripts depend on each other - see later dicussion on [[#Execution order]]). I am going to call these “ScriptNotes” from now on …

Output could be, at discretion of the ScriptNote author,

• appended to the ScriptNote in an “# Results” section (TBC)
  • these would probably be logging sort of things, I guess
• piped to a file in the Vault
  • these would be result type things, presumably in md format

Motivation:

• many things to be done with pandoc conversion of foreign formats to md for import, ditto for note export to other formats
• some things which are now plugins could be done that way (like the Daily note with template, that admittedly was what started me thinking this way). This would open plugin/customisation possibilities for many more users, I suspect.
• it seems many people have scripts to process their vault in various ways
• at least in my office some standard reports were generated as markdown

Implementation basics

Script execution context

• To allow experimentation to figure out the usefulness of the whole concept I propose a first implementation as something that could be inserted in a “Batch file”/startup script that ends with loading obsidian. This is non-invasive both of the core devs timeand resources and the integrity of the Obsidian application.
  • and it lets us trial this before the plugin API is available
• each ScriptNote will be read to extract the script to be executed in the section headed “# script” (TBC), and will be run with the current directory set to that containing the ScriptNote.
  • This is not security but more just a suggestion to think carefully before fooling with notes higher in the Vault hierarchy.
    • security through “making it a PITA” is maybe a good phrase
  • this makes ScriptNote’s somehow location independent
• a consequence of this choice for initial experimentation is that for now the scripts cannot be triggered by anything except a restart of Obsidian
  • or manual running from the CLI too, I suppose
  • for now no hotkeys, no command palette entries …

Execution order

• I believe we need to establish a predictable and defined order of execution so that, if it is useful, scripts may depend on the results of other scripts (e.g. log summaries, consolidated views of related actions)
• as this initially brought up painful memories of makefiles (!) I propose something simpler
  • we start at the leaves of the Vault tree and work upwards, making sure that all leaves are done before we move up to the next level …

Technology

• To be as widely useful as possible we should probably bear in mind some things
  • obsidian runs on windows, linux and MacOS
  • obsidian is (I believe) written in javascript/Typescript and I assume that future integrated plugins will need to be implemented in the same way …
  • But if I’m going to the initial experimentation work we should bear in mind some of my constraints, of which the most relevant is that I have a big ZERO experience in either of these languages
• So my compromise proposal is to write the experimental version in dart!
• Why?
  • dart supposedly, and in alpha/beta at the moment, compiles the same source to windows, linux & macOS native excutables
  • dart is somewhat similar to javascript, I believe, enabling a slightly easier integration if at some later date the core devs think it’s a good idea!
    • dart also compiles to javascript, although I have no idea about the maintainability and “integrate-ability” of the resulting code.
    • Probably this isn’t a profitable avenue to pursue …
  • My current (semi-)professional work is in the flutter/dart domain, so I have the tools and my working memory already setup.

Pro’s & Con’s of this whole proposal

1. (con) we can of course always just open a shell to do the same things
   • (pro) but that’s a bit more work and slightly inconvenient
   • (pro) and if I have several such daily tasks the “automatic open what was open before” function when obsidian starts would deal with them all for no time investment. Very very convenient. Only the case if it becomes an integrated plugin.
2. (neutral) some of the scripts are going to end up being OS dependent
3. (con) people could get themselves in trouble (but see 1 above)
4. (pro) we could easily enough make some sort of ‘cookbook’ collection for the community

[sam.baron]

This is an interesting proposal. Trilium Notes has something like this where you can have a “html” note and a “javascript” note as children of a custom “render note”. You can write javascript against the internal API to access your data and present it however you want, but you have to execute it manually. I’m not sure if you could write to notes with this feature.

Also surprised to see someone mention Dart. I worked a job a few years back where I worked on a web app using Angular and Dart. It’s a decent language, well supported by Google, but obscure, and in my opinion, an unnecessary abstraction on javascript. I suspect the Obsidian devs aren’t going to go for this anyway. They are committed to javascript/typescript with as few dependencies as possible.

[spacejunky]

@sam.baron

Also surprised to see someone mention Dart. I worked a job a few years back where I worked on a web app using Angular and Dart. It’s a decent language, well supported by Google, but obscure, and in my opinion, an unnecessary abstraction on javascript. I suspect the Obsidian devs aren’t going to go for this anyway. They are committed to javascript/typescript with as few dependencies as possible.

I know what you mean, and clearly if/when this would become a “real” plugin it’ll have to be done in javascript/typescript.

For something initially running outside of Obsidian (for experimental purposes, as I said, until if/when happens, which is what this proposal is about) we need some form of multi-platform scripting language. Setting up to use javascript for that is beyond me …

Hence the idea of dart. I’m also happy to go for python, if the rareness of dart is a problem. Or with a certain reluctance java …

[spacejunky]

@sam.baron

but obscure, and in my opinion, an unnecessary abstraction on javascript

And a couple of side comments from one language nerd to another:

• I think since the arrival of flutter you will find that dart is no longer quite so obscure
• I found the prototype-inheritance model of javascript to be really quite perverted, which is one of several reasons why I chose to avoid getting involved with javascript whenever I could
• I personally prefer static type checking,which was the other big no no for javascript. I guess that Typescript solves that but I haven’t had the timeor inclination tolook into that so much

:-))

Nigel

[sam.baron]

@spacejunky Yes, I’m aware of Flutter. My old company was at the forefront of using it. In fact, one of my former co-workers left and went to work for Google on their developer programs team. I agree that Flutter is getting more popular as a cross-platform, mobile app framework, but I don’t see Dart being used much beyond that. Dart is a solid object-oriented language, but I think you’ll find TypeScript measures up well. The one benefit of Dart is that the tooling/ecosystem is simpler than JavaScript/TypeScript (almost every other language tooling is simpler, haha).

I guess in the end, nothing would stop you from writing compiled scripts from Dart, but you would be limited to who could contribute. I haven’t touched Dart in about two years

[spacejunky]

@sam.baron

nothing would stop you from writing compiled scripts from Dart, but you would be limited to who could contribute. I haven’t touched Dart in about two years

Fair enough!

So can I write CLI executable scripts in Typescript? The docs & examples all seem to imply the existence of a browser, or something else with a dom, to provide an execution environment. Can you give me a clue where to look?

In fact, as I said in the proposal, I’d be happier if they aren’t compiled - people shouldn’t have to download opaque executables unless there’s a very good reason. Something more along the lines of the traditional

#!/usr/bin/tsc
. . . scriptcode goes here . . .

is what I’d be looking for …

[sam.baron]

In that case, I think Python or JavaScript/TypeScript is probably the way to go. Almost everyone is going to have Python running and most technical people are going to have NodeJS. I’ve found Python easier for something like this, but if Obsidian ever has an external API, it will probably be in JavaScript.

[VictorGrycuk]

Not sure if what I need is what OP originally wanted, but what about the possibility of having a button next to the code fence to run whatever its inside?

For example, in my project we use several deployment scripts that I keep them in a note ready to copy and paste to run them. In the note I already have the path to PowerShell along with the command and parameters to run. Having a Run button next to the code fence would be a killer feature.

However, I am not familiar with how feasible would be to implement this.

[ksandvik]

Voodoopad actually has a similar feature where its notes could be executable scripts, bash, python and so on.

[spacejunky]

@VictorGrycuk

Not sure if what I need is what OP originally wanted, but what about the possibility of having a button next to the code fence to run whatever its inside?

No, it wasn’t what I had in mind, but it surely is a cool idea!

Nigel