New Properties and templater prompts and commands?

First of all. Disclaimer: I’m not a coder, so from where I’m sitting (on the fence), great work! :slight_smile:

I tried out the second, merge_frontmatter_function.

My testing is not comprehensive.

I really like the option of being able to select functions. It’s like plugin settings.


1

If I may make a suggestion (not sure how easy it would be to implement)…

From what I understand about the workings and YAML treatment of Obsidian (largely through the workings/settings of the fantastic community plugin Linter and the chats I’ve had with the developer/maintainer of the plugin – I sent him on more wild goose chases than he had bargained for), some values are Boolean, some are text strings, some are multi-line, some are not…etc. The Properties UI panel sometimes used to alert me to invalid properties such as topic and parent_topic fields being not properly addressed – not anymore, it seems.

So my question is, is it possible to merge the add and update functions – based on whether the keys in question are sporting single-line status fields?
I have a feeling that Obsidian still has some quirks handling these cases and you need to wait with the final solution but we should keep our options open – otherwise the user must be aware that the status properties will NOT be updated unless he adds update. As I managed it using some conditionals in my use case:

const fileContent = await app.vault.read(currentFile);
	const formattedDateTime = await tp.date.now("YYYY-MM-DDTHH:mm");
	const properties = [];

	// Check if dg_upload or status exists in frontmatter
	if (!fileContent.includes('dg-updated:')) {
		// If "dg-updated" doesn't exist, add it
		properties.push(
			{ key: 'dg-created', value: formattedDateTime },
			{ key: 'dg-updated', value: formattedDateTime },
        	{ key: 'dg_upload', treatment: 'update', value: 'VNC done' }
		);
	} else {
		// If "dg-updated" already exists, just update the timestamp
		properties.push(
			{ key: 'dg-updated', value: formattedDateTime },
        	{ key: 'dg_upload', treatment: 'update', value: 'VNC done' }
		);
	}

	// Add or update other properties even if they existed before
	properties.push(
        { key: 'status', treatment: 'update', value: 'dg_uploaded' },
		{ key: 'share', value: true },
		{ key: 'dg-publish', value: true }
	);

	// Call merge_frontmatter with the properties array
	await tp.user.merge_frontmatter(tp.config.target_file, properties, tp);
});

I’ve sent myself on a wild goose chase on this too, actually.
My original YAML keys to do with two types of status were dg_upload_status and status. No matter what I did (a week ago as well, when I wanted to make a processFrontmatter version of my search-and-replace version of one of my key templates), they keys’ values were not updated.
My first thought was it is some kind of Obsidian properties issue again, along the lines of this, meaning that I cannot have properties that has a conflict in names – I even went ahead and changed dg_upload_status to dg_upload everywhere in my vault, when I realized that I need to use update treatment…so finally at least I got it working. But as I said, these two should somehow be merged on keys that can take only one value (if it’s not easy to implement, leave it, just make sure you alert people to the quirks of this).
On date modified, I had to add update treatment as well to make it work. So it works as advertised, is all I’m saying.

There is no ambiguity on Booleans (default add without having to specify anything else works). Didn’t really test tags, as they were solved by you before and I use tags less and less…

2

A minor thing. If the file had no YAML but some content, there was no empty line added after the second instance of ---. In my case I had to add the empty line before my Heading 1 title:

I added this into my Templater js md file:

setTimeout(async () => {
    const currentFile = app.workspace.getActiveFile();
    let fileContent = await app.vault.read(currentFile);

    // Replace if the pattern matches
    fileContent = fileContent.replace(/(---)\n(^#{1,6}\s.*)/gm, (match, p1, p2) => {
        return `${p1}\n\n${p2}`;
    });

    // Modify the file with the updated content
    await app.vault.modify(currentFile, fileContent);
}, 250);
  • This is very rare of course, but I thought I’d mention it.

I hope some of my ramblings made sense. And as I said, I have a hunch some smoothing out still needs to be done under the hood as well before you can finalize the script(s).

Conclusion

The method offered is an upgrade on the formerly introduced methods. I was going to add a GIF here showing my earlier failings using a script from a week ago, but it seemed like too much work.
This definitely seems like an upgrade in functionality on this simpler method, which will not always work as expected, as I mentioned.

P.S.:
I’m going to start updating my templates using this and if in the meantime I see anything, I’ll let you know.

Cheers

1 Like

Hi Gino,
Thanks a lot for testing and feedback! I’m glad you’re putting it to use in your workflow :slight_smile:

Regarding 1, all the functionality you coded is already built into the script (unless I misunderstood something).

By default, missing keys are always added. It’s only in the case of existing keys that the treatments come into play. You can see how that’s handled here:

// Check if property exists, if not, add it

if (!fileCache || !fileCache.hasOwnProperty(prop.key)) {
    console.log(`${target.basename} doesn't contain ${prop.key}`);
    console.log(`Adding property: ${prop.key} to ${target.basename} with value: ${prop.value}`);
    frontmatter[prop.key] = prop.value;

// If property exists, handle it according to treatment type   

} else {
    console.log(`${target.basename} contains property: ${prop.key}`);
    if (treatments.hasOwnProperty(prop.treatment)) {
        treatments[prop.treatment](frontmatter, prop, value);
    } else {
    // Default treatment
    treatments['add'](frontmatter, prop, value);
    }
}

So the following should do the same as what were trying to prep before letting the script handle it:

<%*
tp.hooks.on_all_templates_executed(async () => {
	const formattedDateTime = tp.date.now("YYYY-MM-DDTHH:mm:ss");
	const targetFile = tp.config.target_file;
	const properties = [
    	{key:'dg-updated', value: formattedDateTime, treatment: 'update'},
    	{key:'dg-created', value: formattedDateTime},
    	{key:'dg_upload', value: 'VNC done', treatment: 'update'},
    	{key: 'status', value: 'dg_uploaded', treatment: 'update'},
		{key: 'share', value: true, treatment: 'update'},
		{key: 'dg-publish', value: true, treatment: 'update'},
    	{key: 'bool', value: true, treatment: 'update'}
    ];
	await tp.user.merge_frontmatter_function2(targetFile, properties);
});
-%>

See demo:
output

Here, I run the script several times with the same parameters and treatments to showcase how the behavior changes, depending on the treatment, whether the property exists, and whether the values are different.

So if you want something added if it doesn’t exist, but not updated if it does, use add or omit treatment to get the default (same, same). update will add if it doesn’t exist, update if it does (unless the values are the same, in which case, it won’t do anything).

Please let me know if I misunderstood what you meant.

Btw, I noticed that tp is now redundant as a parameter since I changed the script to accept the TFile as a parameter. I’m updating the scripts above to reflect that. I also removed some minor redundancy from the update case.

Regarding point 2:
I couldn’t test this due to the insider bug in processFrontmatter. But thanks for sharing your solution!

1 Like

Been a busy day, so I’ve not been able to get around to getting you some feedback, but I’ve noted your efforts, and will give some feedback in the not so distant future.

Don’t be too eager to remove the tp variable. One of the reason it’s often present is that it allows for the user functions to be used outside of Templater. For example it’s possible to access Templater functions from Dataview, but that do require for the tp variable to be set by Dataview. So if you remove it, you also remove the possibility to use your functions from Dataview, and that would be a little sad. I can imagine using this kind of function from within dataviewjs in various contexts…

2 Likes

I was talking about any status properties that had and accepts one value. My conditional (which is kind of pointless as I already had removed one condition and over time I extended the scope that this particular template was originally intended to cover) also shows what I had in mind.

  • EDIT: well, there used to be two if's there but one had to go. Anyway. I had to update two kinds of statuses, not make a new property.

I’m not sure (or: up until now I wasn’t entirely sure) how accepting one or more values (which ultimately would – in my mind – necessitate using the add or update treatment as per your settings) works under the hood.
Possibly like this (anyone is welcome to counter these if I’m mistaken):

  • If you have key: value format, then Obsidian treats it as text in a single-line array (“array” in my mind means a series of more than one, but that’s the term Linter uses in its settings, in which one can define which property one wants multi-line). The icon in the Properties UI looks like this:
    Screenshot from 2024-02-16 23-25-20
    The icon seems a little similar to that of list. The important thing is that it is a ‘text string’. In my case, it is a status (either one of the two): status can be either…or, not usually and logic-wise.
  • Among the multi-line properties there is list – again a word denoting or at least hinting at a series of more than one – speaks for itself and the icon is probably more graphically spot on.
    The multi-line array is clearly seen in the source view where each value is headed with the required syntax -:

    When Obsidian knows or “made up its mind” about these types as lists, it will (or at least should) allow ‘updating (the list) by adding’ – I am using the function names used for treatments on purpose here.


    Similarly to list items, Obsidian also has on board default properties tags and aliases which – already shown by the use of the much-debated plural (“more than one”) in the property names – will be used as multi-line properties, which is shown by the syntax used; even if there is (currently) only one tag(s) and alias(es) in the YAML, it will not be populated in key: value format (with the key and value on the same line), but in multi-line format:
    Screenshot from 2024-02-16 22-44-04

So to get back to the difference and similarity of add and update, it depends on the property type. Where the status was set (by whom and where? does Obsidian look for ‘status’ word?) as a single-line property value, Obsidian will not budge on (default) add, as it is not a multi-line (list). It expects your update treatment. When it’s a list, as with tags, your default add will update the list.
I don’t want to go into dates and Booleans here as these cases are more straightforward, although my date modified field was only updated on update treatment (which is fine, nothing wrong with that).

From the above, I reckon that it could be programatically achieved that single-line properties and dates could be updated on default add as well, but as I said, it’s not a big deal if the script stays as it is. What I have in mind again is the people who will in 2-3 months time will start using this and you could save (ourselves) some headaches going forward. I may be going over the top on this, granted.

What I haven’t tried yet is your override parameters. So all in all, I feel a bit overwhelmed by the fantastic effort yet – so bear with me.

  • People out there will be following suit for the reasons mentioned above. The js script needs to be done and filed away once and the template md files are quickly done by anyone not too savvy about coding. This will hugely make it easier with non-tech users.
  • Some concern is 1) reliance on Templater which can break or be abandoned 2) processFrontmatter – which is used in the js script – issues cropping up here and there; then we have to go back to search and replace scripts which I recommend everyone to not delete but keep with telling filenames.

  • I used value last and it worked like that as well. Great stuff.

  • Probably there is a better solution. It was a quick ad hoc treatment on my part.

So, again, chapeau from me and I’ll be advertising the method once it passes (scrutiny from) under some Norwegian clouds first to be blown back down south again… :slight_smile:

Thanks, I appreciate it!

In this case, it was only present because I needed it for tp.config.target_file inside the script. Now, no templater functions are used inside the script.

You can still use the user script together with dataview if you just access tp in the dataviewjs script using the methods described here: Templater snippets

Perhaps you can expand on it, but I’m not seeing how that is contigent on tp being passed as a parameter to the script. On the other hand, without the tp dependency, this script can be run by anything that runs JS. An then again, adding tp back is a minor change.

Text properties will accept multiple values and change into list properties, if you pass multiple values using processFrontmatter.

Obsidian_Zpr7TXP5yf

If you really meant a single-line array, like Linter supports, then this is not correct. Obsidian doesn’t support single line arrays. It will reformat to multi-line lists / arrays. A text property is not an array (doesn’t accept multiple values), but I think you were actually getting at that. As my test above shows, text will become list if you force multiple values onto a text property.

Using add on a property whose existing value is an array will push the new values to the existing array:

Obsidian_PnX0l4scjw

This only takes into account the existing values in the target note, not how the property is configured in the property UI. A value can only be considered an array if it’s formatted like this:

image

This is not nessarily the case even though the property UI says that the property type is list.

Using update on a property whose value is an array will overwrite the existing values with the new values:

Obsidian_0I6onFWtgF

Perhaps update should be called overwrite? In my mind, update updates existing values (but overwrite is just as descriptive, if not more), while add adds to existing values, but doesn’t update anything.

This seems to be a misconception, because add can handle both single values and arrays:

'add': (frontmatter, prop, value) => {
    if (!value || (value && override === 'destructive' && value !== prop.value)) {
        console.log(`Assigning ${prop.value} to ${prop.key} in ${target.basename}`);
        frontmatter[prop.key] = prop.value;
    } else if (value && Array.isArray(value) && prop.value != null) {
        arrayPush(target, frontmatter, value, prop);
    }
},

However, since add is meant to be the safe option that doesn’t overwrite existing values, it will only add values, if the key doesn’t have an existing value. The destructive override changes this, so that add will overwrite single values. Perhaps that’s the behavior you are looking for?

I mean, that is exactly the intended behavior and usecase for update.

Well yes, with the destructive override, but not by default, because this is meant to be an option that only adds, and does not remove / overwrite data.

I appreciate you having and discussing these concerns with me.

Sounds like you would benefit from the destructive one. And of course, no worries!

There’s really no reliance on Templater, because the script is all javascript and Obsidian API. But of course, it’s usefulness is amplified by the Templater templating environment. However, you can run it using QuickAdd or whatever runs JS. Of course, that leaves point 2, but I hope the devs appreciate how people are using processFrontmatter.

Yeah, this is because these keys are named, and not index based, so it provides this flexibility.

Thanks! And, hehe, yeah, it will be interesting to hear what @holroy has to say, and perhaps hear more about those dataviewjs ideas as well.

2 Likes

Yeah, I didn’t take into consideration the [field1, field2] array, which I hardly ever used, especially since the Properties launch, where the list values superseded those; in coding, apparently you can still continue to make use of these.
It’s really useful, what you put up here in animations as well.

I saw that, I meant the hook call, the way we invoke it.
Also, can we use const currentFile = app.workspace.getActiveFile(); instead of const targetFile = tp.config.target_file;?

Yeah, as soon as I mentioned above, I knew I should not haste with the post. But God had his plans with me…

In any case, all of the above goes to show that… (“Oh-oh, yes, I’m the great pretender”)…I am a great—interviewer who brings the most out of my subject because if I get everything right, all the general public would see here on your part is “Yeah, Gino, you rock.”
:)))

So, with my job duly done here…
…I’ll be “Up on the roof” – to quote another oldie…

Thanks again for the great explanations – I’m sure even older pieces of furniture round here were enlightened by all this.

1 Like

Although this doesn’t sound like something I’d do in my workflow, examples would be nice when you get the chance, because this is important for automation.

  • Coupled with DataView, which example or boilerplate based on this will probably done by @holroy, I presume.
    Bulk modifying potentially hundreds of files based on search criteria which cannot be done outside of Obsidian is an important next step.

It seems like Gino has covered some of the bases, but I’ve not read all of this very thoroughly, so I might repeat some things, and miss out on somethings already covered, but here goes my comments at the first level, and suggestion for improvements at lower levels:

  • Regarding switch or function, I’m a little undecided, but intuitively I find both approaches hard to read as both the switch and the function definitions hides away what’s happening when. It gets convoluted to follow the logic of the script.
    • Define handleProperty as its own function outside of mergeFrontmatter(), and use a switch within it
    • Use the forEach() variant without await’ing in the main callback function used with processFrontMatter().
  • The handling of the safe option also makes your code a lot harder to read, so I would preferably bail out sooner if its present.
    • Within handleProperty() do a simple test at the start and bail out if the safe option is set, possibly with a message stating that you bailed out of doing treatment
  • The add handling and the default handling is not identical, although I think I saw somewhere that you said they should be. That’s a little dangerous, as you then can experience different behaviour depending on how you call it.
    • Make the logic for both of these operations the same
  • You’ve tested some of the cases of the property being an array or a simple list, but not thoroughly, so you might be changing an array directly into a simple value when doing update, or you’ll not add anything if the property has a single value but rather change it?. Similar cases exists if you do custom on array values…
    • Not sure how to tackle this, as the exceptions are many to the most basic rule, but be sure someone will trigger that strange combination
    • I’m thinking that an add on a single value should change it into an array
  • How does your script handle changes to array values? You can add values _if its already an array, but not when its a single value. And can you update, and/or delete elements from arrays? Or do custom stuff to them? Or increment?
  • For debugging purposes it’s nice to have loads of console.log() messages, but in a “production” environment less so.
    • Provide an option to disable all log messages, but the error related ones
  • Dependency on variables into async method is inherently unsafe. You set the fileCache early on, and then after loads of awaiting here and there, you suddenly refer to it again. By now it could be changed or gone out of scope (not entirely sure on this one, as it depends on garbage collection/asynchronous call handling within javascript). In general it’s kind of unsafe to do stuff like that. Same logic applies to target and value at least. They’re suddenly appearing in a context where I’m not sure you can rely on their value, and where you don’t see where they came from easily.
    • Clean up the code, so that each function either defines locally, or through parameters all of its variables. No global stuff (except for function definitions in general)
  • This might be related to the previous item, but why do you refer to fileCache within the Promise.all() when it already has the frontmatter passed along to that callback?

I’m sorry if this was hard reading, as I do believe there could be a good reason for doing something like you’re doing. Especially when we talk about mass updating across multiple files. However, that is also the case why one needs the script to be extremely precise and not leave certain property value/type/treatment combinations to cause different behaviours.

And I’m a strong believe that readable code is usable code, and being able to follow the logic (and separations of concern) is vital aspects to making it readable. Sadly, I’m finding either version somewhat hard to read, especially due to the mix of defining functions and callbacks and whatnots, before they’re actually called.


Hmm… Right before posting I now see something about dataviewjs ideas? What was that? Maybe it was me mentioning that a script like this could be useful to call from within dataviewjs as part of a larger file update thingy? Like if one wanted to change all the daily notes from an earlier stage with a mismatch in templates, it could be useful using a tool like this to update stuff.

I love app.fileManager.processFrontMatter but can it be used to create new notes from a template? I unsuccessfully tried. Maybe an example may help.

I have used app.fileManager.processFrontMatter to recalculate formulas and write the results back in properties such as numFigures, numCallouts, etc., based on changes in the notes themselves.

Yes, if you use a timeout, or even better, the tp.hooks.on_all_templates_executed module from Templater (docs).
See these two examples here and here.

1 Like

That last link of yours, to zachyoung, seems to indicate that app.fileManager.processFrontMatter can’t create the file by itself. But after the templater hooks have completed, which indirectly depends on Templater to create the file (or Obsidian if created through a link), then processFrontmatter can be called to do further work.

1 Like

You’re right, I read @msfz751’s question too quickly and thought he was asking whether processFrontmatter could process newly created notes.

The best way to do that is using tp.hooks, if you are working with Templater. So since you already gave the right answer that processFrontmatter can’t create notes, my alternative suggestion would be to use tp.file.create_new to create new files using a specific template.

The file is already created, so no need to create another file…

I don’t think that’s the case:

I’m going to leave this thread now. I feel we’ve diverged to far from the original request, which also has been marked with solved. There could be made a multitude of new requests based upon this thread, but I’m out for now.

Quits Obsidian on iPad alas, so had to revert to old versions for now.

Oh really? I haven’t gotten around to testing the new vesion on mobile. I am going to check it out on the mobile console when I have time sometime this week.

1 Like

Shouldn’t this be <%_* tR = "" _%>? It didn’t work for me until I removed the +. Otherwise, it’s not removing anything.

I’m going to have to read this a few more times to get the full jest of this.

…but what I’m understanding is: if you use Templater ‘code’ in your YAML, you have to create a full block of code, including all properties you currently have in the Template.

For example, I use these along with some ‘standard’ ones:

modification date: <% tp.file.last_modified_date(“dddd Do MMMM YYYY HH:mm:ss”) %>

UID: <% Math.round(((Math.random()10**15)(tp.date.now(“YYYYMMwwDDHHmmss”))/(Math.random()*10**15))) %>

File Creation Date: <% tp.file.creation_date() %>

The ‘modification date’ will never again update though…