API: processFrontMatter removes string quotes, comments, types, formatting

Developers: Plugin & API
1

Calling processFrontMatter will destroy all previous formatting, and most YAML features/syntax. As you will see from the expected vs actual output, the behavior is quite odd.

Steps to reproduce

  1. Modify the sample’s ‘sample-editor-command’ code in the onload method here to the content below
this.addCommand({
	id: 'sample-editor-command',
	name: 'Sample editor command',
	editorCallback: (editor: Editor, view: MarkdownView) => {
		console.log(editor.getSelection());
		editor.replaceSelection('Sample Editor Command');
		let file = this.app.workspace.getActiveFile();
		this.app.fileManager.processFrontMatter(file, (frontmatter) => {
			console.log('this is the frontmatter', frontmatter)
		})
	}
});
  1. Create a file with the following frontmatter content
---
description: 'course; subject, note'
id: "deadbeef-123-$$"
date: !!timestamp 2022-01-25
date2: 2022-01-25

# a comment

forced-string: !!str "i should have bookend quotes"
status: !!null Null

nameArary: [six, seven, eight, nine, ten]
nameArrayQuoted: ["six", "seven", "eight", "nine", "ten"]
nameBlockStyleQuoted:   
  - "six"
  - "seven"
  - "eight"
  - "nine"

a: 123                     # an integer
b: "123"                   # a string, disambiguated by quotes
c: 123.0                   # a float
d: !!float 123             # also a float via explicit data type prefixed by (!!)
e: !!str 123               # a string, disambiguated by explicit type
f: !!str Yes               # a string via explicit type
g: Yes                     # a boolean True (yaml1.1), string "Yes" (yaml1.2)
h: Yes we have No bananas  # a string, "Yes" and "No" disambiguated by context.
---
  1. Run the ‘Sample editor command’ command

Did you follow the troubleshooting guide? [Y/N]

Yes

Expected result

Front matter is not modified

---
description: 'course; subject, note'
id: "deadbeef-123-$$"
date: !!timestamp 2022-01-25
date2: 2022-01-25

# a comment

forced-string: !!str "i should have bookend quotes"
status: !!null Null

nameArary: [six, seven, eight, nine, ten]
nameArrayQuoted: ["six", "seven", "eight", "nine", "ten"]
nameBlockStyleQuoted:   
  - "six"
  - "seven"
  - "eight"
  - "nine"

a: 123                     # an integer
b: "123"                   # a string, disambiguated by quotes
c: 123.0                   # a float
d: !!float 123             # also a float via explicit data type prefixed by (!!)
e: !!str 123               # a string, disambiguated by explicit type
f: !!str Yes               # a string via explicit type
g: Yes                     # a boolean True (yaml1.1), string "Yes" (yaml1.2)
h: Yes we have No bananas  # a string, "Yes" and "No" disambiguated by context.
---

Actual result

A string’s quotes can be removed, newlines are removed, comments are removed, types are removed, timestamp types are modified, and inline strctures (arrays) are converted to block style.

---
description: course; subject, note
id: deadbeef-123-$$
date: 2022-01-25T00:00:00.000Z
date2: 2022-01-25
forced-string: i should have bookend quotes
status: null
nameArary:
  - six
  - seven
  - eight
  - nine
  - ten
nameArrayQuoted:
  - six
  - seven
  - eight
  - nine
  - ten
nameBlockStyleQuoted:
  - six
  - seven
  - eight
  - nine
a: 123
b: "123"
c: 123
d: "123"
e: "123"
f: Yes
g: Yes
h: Yes we have No bananas
---

Environment

SYSTEM INFO:
Obsidian version: v1.3.7
Installer version: v1.1.9
Operating system: Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 22.6.0
Login status: not logged in
Insider build toggle: off
Live preview: on
Legacy editor: off
Base theme: dark
Community theme: none
Snippets enabled: 0
Restricted mode: off
Plugins installed: 1
Plugins enabled: 1
1: Sample Plugin v1.0.0

Additional information

Possibly related: `processFrontMatter` creates undesired line break for long strings

6 Likes
2

I think we need to set the expectation first. As long as things are semantically the same, it’s not a bug and you should not expect for it to remain untouched.

The dates is something we handle internally.
For comment support, you can search/open a feature request.

What else in the above list is semantically different?

3

The issue you are referring to is related to the fact that used library js-yaml doesn’t have a round-trip feature.

I am working on the similar feature for Obsidian Linter plugin

2 Likes
4

Comments are data, and removing them is loosing that data. You should do a better parsing

3 Likes
5

I think we need to set the expectation first.

I agree, and sorry, I know my post was a bit of a hodgepodge, but I didn’t know what to focus on since there are multiple surprising changes, but not very meaningful to talk about each in isolation. Since it seems this is all under the control of js-yaml, and expected behavior, then perhaps this isn’t worth much discussion.

In case it might be useful, let me mention how I came across these issues, and how they impact me:
I want to have a plugin that appends or modifies any number of fields in my frontmatter. One explicit example: after completing a “review” of a file, add the fields: last-reviewed-datetime, review-duration. Right now, it appears this is not possible to do without the entire frontmatter being modifed. This is not ideal when tracking file history in version control, or working with other people’s files where you don’t want mess with their formatting.

2 Likes
6

So we’ve had long debates about this, and it mostly comes down to a trade-off between allowing user to store arbitrary comments and formatting in the YAML, vs making it not painful for plugins to interact with YAML.

I will say that it is rather unfortunate that YAML became the most popular way markdown documents store metadata. YAML is difficult to write by hand because it provides way too many options to write the same thing, and there’s a lot of “gotchas” such as reserved keywords, the many many ways one can write a multiline string, etc.

Unfortunately, this is really a place where standard markdown forces us to make a hard choice. The decision we made is for plugins to be able to interact with frontmatter easily. We do this by converting YAML into a JS object, which means it lose all of its YAML specific formatting and comments. This makes it easy for plugin developers to read and modify properties of a note, without requiring them to interact with an excessively complex YAML syntax tree, which would require them to understand how YAML is parsed internally.

I do understand that a very small minority of people make heavy use of YAML, and we are unable to accommodate their needs for such levels of customization. In these cases, you’ll either have to manually use js-yaml to work off the parse tree (I’d actually be curious to see how you make use of it), and avoid having Obsidian touching the YAML altogether.

10 Likes
7

That is understandable, focus on the majority of your userbase.
However, maybe Obsidian could make easier for plugins to deal with js-yaml.
I’m not saying you build an entire abstraction layer, but it could be cool if you make the js-yaml api available under some namespace, that way plugins will not need to bundle their own version of the library.

3 Likes
8

Perhaps you have heard of “templates.”

In the past, it could parse correctly.

In fact, there’s not so much “unfortunate” here. All of this is not a reason to change user data; there’s no need for someone to tell us how to take notes, what format to follow, and enforce conversions. I’m not sure if this aligns with “for you.”

I’m not a technical person, but since my YAML can be recognized and forcibly converted, could an API be provided to plugin developers instead of using them as a scapegoat to shift the conflict?

In previous versions, my aliases and tags could be correctly parsed, so I don’t think there’s a need for it to be forcibly formatted. Could an option be provided to completely disable Properties? It greatly affects my batch processing. I don’t want my data to be silently changed every time I insert a template or perform any operation, and if there’s YAML in my template, it can’t even be fully inserted.

Alternatively, could you consider my following suggestion: YAML is left to Obsidian, but please optimize “Inline Fields” so that aliases and tags within it can be parsed.

1 Like
9

I’ve been having a similar issue. Here’s a link to my post and current workaround.

2 Likes
10

I’m currently using Obsidian Linter pluginr, which, compared to Obsidian itself, gives me reminders after each change.

1 Like
11

Wouldn’t Obsidian achieve the same goal by only modifying the contents of the properties that the plugin developer is updating, and leave the rest of the YAML contents as-is?

You would still abstract the read/write functions for the developer (e.g. I don’t have to interact with a complex syntax tree), but the majority of the YAML would stay untouched, exactly how it was wanted by the user.

4 Likes
12

Thanks for the insights! I’d still greatly appreciate it if, at a minimum, comments in the YAML frontmatter could be preserved.

If assisting plugin developers is the goal, removing comments from the frontmatter - which can otherwise be used to “hide” various plugin template engine instructions (such as for the Zotero Integration Plugin) - is also detrimental to plugin developers.

Henceforth, it would be great if your implementation of js-yaml could preserve the comments at a minimum, as suggested by @AlanG

1 Like
14

Can we at least get a warning added to processFrontMatter - Developer Documentation to indicate that this will damage user data?

3 Likes
15

well d and f have had their type information changed for one,
and the comment has somehow gone missing,
and the name array has been changed from the brackets style to the newline-dashes style.

Imagine if the linter you use to write the code for Obsidian itself, deleted all commends and re-wrote literals like this. That would be unacceptable! Be kind and considerate to the user data you are taking care of, please. Do not mandate to your users that your way of writing their data is superior to theirs.

1 Like
16

Please let me use my software how I want to. If my choices in how I use my software cause it to break or slow down, that should be my decision whether that is acceptable, not yours.

You could also send the plugin some in-memory-only version of my frontmatter, leaving what I wrote untouched but still sending them your idealized version.

At the very least give me some setting/option/preference to preserve this stuff, please. To me, it’s unacceptable for you to destroy my data just to allow some hypothetical 3rd-party plugin dev to have an easier time reading something I didn’t write.

1 Like
17

@KyNorthstar nobody forces you to use processFrontMatter() function if it doesn’t fit your needs, you can use Vault.process() and deal with the entire note and serialize your front matter preserving your comments and other important data.

As far as I see, Obsidian internally uses GitHub - eemeli/yaml: YAML parser and stringifier for JavaScript to parse/stringify YAML.
I guess if you find a library that fits your needs and has comparable performance, you could suggest the devs to switch to it

1 Like