Update Properties of Files from Search Results in Bulk

This could be something people might be interested in.
There are things you can query within Obsidian only, e.g. (lack/number of) backlinks. You cannot do this in VSCode or other text editors.

So how to update Properties (YAML fields) only in certain files, in the results of a query (in this case, a DataView query)?

Late last night, I fiddled with the Projects community plugin, looking for a solution.
(As far as I know, the MetaData Menu plugin makes it possible to update DataView table properties one by one, but I might be wrong.)

The challenge is to update Properties or fields of potentially hundreds of files. For example, tags. But only tags of files in which we matched some query criteria.

E.g. I fed this DataView query to Projects:

TABLE file.inlinks as Backlinks, length(file.inlinks) as Total, join(file.tags, " ") as Tags
FROM "X"
WHERE !econtains(flat(list(file.tags)), "#dg_uploaded")
AND !econtains(flat(list(file.tags)), "#nopublish-formatted🟢")
AND length(file.inlinks) = 0
SORT length(file.inlinks) DESC
  • This will give you a table listing files with no backlinks filtered by tags not present in these files.
    I chose the folder X for the sake of simplicity, because I knew a limited number of files will crop up here.

Unfortunately, in Projects, I didn’t find a way to update the tags in bulk:


I don’t see an option for it.

A workaround would be to add a field, which could be used to match with regex captures outside of Obsidian (with VSCode, Notepad++, whatever), then update the tags with regex and delete the dummy field created in Projects.

So my question is: how do you update Properties in batch based on custom queries with DataView or otherwise?
Am I missing something?
Are we bound to some workarounds and regex hocus-pocus?

The hocus-pocus part:

  • I botched this up, as I should have catered to different cases for formatted and unformatted, but you get the idea.

If you search for summing columns in tables you’ll soon find a boilerplate that I’m using a lot in different contexts, and which should be applicable here as well.

```dataviewjs 

const result = await dv.query(`
... insert your query here ...
`)

if ( result.successful )  {

  ... Do something with result.values

} else
  dv.paragraph(`~~~~\n${ result.error }\n~~~~\n`)```

In your case the handling of the values cloud be trigger the app.FileManager.processFrontMatter() to change every file in your result set.

In other words, use an ordinary query within dataviewjs, and process the result using app.FileManager.processFrontMatter().

2 Likes

Here is a fully working example:

```dataviewjs
const tp = app.plugins.plugins['templater-obsidian'].templater.current_functions_object

const result = await dv.query(`
  LIST
  WHERE file.folder = this.file.folder
    AND file != this.file
`)

if ( result.successful ) {
  const values = result.value.values
  for (const f of values) {
    const tFile = await tp.file.find_tfile(f.path)
    await app.fileManager.processFrontMatter(tFile, (fm) => {
      console.log(fm, Object.keys(fm).length)
      if (Object.keys(fm).length == 0) {
        fm['initiallyEmpty'] = true
      } else { 
        if (fm['one']) {
           fm['one'] = fm['one'].toUpperCase()
        }
        delete fm['toBeDeleted']
      }
    })
  } 
} else
  dv.paragraph("~~~~\n" + result.error + "\n~~~~")
```

Here the query simply lists the file in my current folder, excluding the file of the query. Then it proceeds to uppercase the one property, delete the toBeDeleted property, and if no properties add the initiallyEmpty property. I’m using a little trickery to include the find_tfile() from the Templater plugin, so this query relies on both Dataview and Templater. Not sure if (or how) I can locate the tFile using other means.

To showcase my test setup, I ran this query:

```dataview
TABLE file.frontmatter
WHERE file.folder = this.file.folder
  AND file != this.file
```

The before run showed this:
image

And the after run has this output:
image

And as expected the wanted changes has occurred.

Some caveats using stuff like this:

  • Be sure not to cause any recursive editing of properties which can occur if your current file holds a query displaying values/files you’re going to change. The update of the files trigger a refresh of the queries in the current file, aka also running your update frontmatter script again…
  • You might experience issues if some of your files are open in other tabs, but it should work
  • Please do multiple dry-runs checking what you want to change, before actually doing the changes. Both to verify that your query actually hits the correct files, and that the changes you intend to do are the correct changes to do. A tool like this can explode/change your vault drastically
  • Please do make a backup of your vault before doing a larger run!!!
3 Likes

Thanks for taking the time to look into this. I’m afraid I could only skim the code as I’m nearing bed time and otherwise not too js-savvy. Although yesterday I made the DVJs version of the query too (wasn’t hard as I had 5-6 different versions of much the same thing) and was looking to cook something up with my Dumb and Dumber partner, the bad-hair-day C-Three.5Pio but I bailed out, thinking there must be some other way.

I mean my write-up sounds like a niche case, but it isn’t. Anybody who uses Obsidian for a database will time and again find the need to modify the files in the results of a query – whether to scrap, archive or whatever.
There is not an innate Obsidian way of doing this and it’s up to the users to carve out some use cases for themselves from various third-party plugins.

I actually find the capabitity of the Projects plugin quite handy. You can batch modify the files of your query (based on various criteria) by adding a field (I used Dummy for key and 123 for field value) and with this way you can temporarily mark your files (as in Notepad++ you can mark files, for instance) – I say temporarily, because I don’t need a YAML key after my date created and modified keys.
But I thought I’d write this up so people with zero coding and regex skills might benefit from some answers.

I expect Obsidian to be able to work a bit like the Projects plugin (but better) drawing on its own Databases core function within 6-10 months time, because modifying properties in batch is a necessity. (Not sure what Notion or other software can do.)

Anyway, I’ll look into what you were up to with your code.

Cheers

P.S. The (flat(list(file.tags)) I also nicked from one of your posts, BTW. I couldn’t make it work any other way.

The logic I understand and I like the succinct way it handles the task at hand.

Using your example files and script, it does work.
Otherwise it throws

Evaluation Error: TypeError: Cannot read properties of undefined (reading 'replace')

So seems like this script only updates the frontmatter if there no extra keys and values in the frontmatter?

Also, would it need extra care with regard to tags being in a list, I wonder?

That should work just fine, as I also had those kind of properties in my test setup. So your error is related to something else. Could you show us your version?

I’ve not tested with lists, but that should work just fine I reckon. I’m more curious on how links would be treated, and might need to check that out.

I added some other frontmatter keys and values and was working after I wrote these lines… I mean your script with the test files and frontmatter values.

Then on my own script, the error changed to

Evaluation Error: TypeError: Cannot read properties of undefined (reading 'file')
    at eval (eval at <anonymous> (plugin:dataview), <anonymous>:15:28)
    at async DataviewJSRenderer.render (plugin:dataview:18670:13)

Script used:

```dataviewjs
const tp = app.plugins.plugins['templater-obsidian'].templater.current_functions_object;

const result = await dv.query(`
  TABLE file.inlinks as Backlinks, length(file.inlinks) as Total
  FROM "999"
  WHERE !econtains(flat(list(file.tags)), "#dg_uploaded")
    AND !econtains(flat(list(file.tags)), "#nopublish-formatted🟢")
    AND length(file.inlinks) = 1
  SORT length(file.inlinks) DESC
`);

if (result.successful) {
  const values = result.value.values;
  for (const f of values) {
    const tFile = await tp.file.find_tfile(f.path);
    await app.fileManager.processFrontMatter(tFile, (fm) => {
      console.log(fm, Object.keys(fm).length);
      if (Object.keys(fm).length === 0) {
        return;
      } else {
        if (fm['tags']) {
          fm['tags'] = 'nopublish-formatted🟢';
        }
      }
    });
  }
} else {
  dv.paragraph("~~~~\n" + result.error + "\n~~~~");
}
```
  • I don’t want to create but update the tag anyway; do we need to delete first? Or use search and replace? I usually always use search and replace for frontmatter fields as well because Templater frontmatter changes were causing those annoying Merging changes… pop-ups.

Frontmatter of result in DV query done separately (one test file in temp folder 999 used in js script):

---
title: Ad
aliases:
  - ad
tags: 
  - unformatted⚪
share: true
dg-publish: true
dg_upload_status: down
date created: 2022-12-14
date modified: 2023-10-22
---

Was doing some back and forth with chat robot earlier this afternoon, but I think it got sidetracked on the first error message, then I had other things to do.

I tried some 6-8 variations on the script, changing the frontmatter as well, so I don’t really know anymore.

I can hazard a guess that Templater cannot access the tp.file from the DV query?

I think the main problem is that your query returns a list of links, and a second column, instead of a single list of file links. This means that the f.path isn’t valid and it v seems to create some havoc elsewhere.

So you either need to change your query so that it is using LIST on a single level, or you need to change and add another level of looping around the file&frontmatter stuff.

Yes, I was thinking that too. But when hours before I was using other types of queries (which may not have qualified either, granted) and it was those which spewed the other error.

As for the other thing, create vs. update (update is like rename: delete + create, right, says the non-programmer in me), minutes after my post I got thinking I shouldn’t use the tags I’m using; rather I should use status key with one field. Tags have their own advantages and before Properties came along, we could only list tags. I could convert those tags to status fields to more easily manipulate them (without having to resign myself to search&replace jobs), but that would entail rewriting some 10-12 Templater scripts (wouldn’t take long, of course).

I will do that tomorrow (today; in Europe), and will get back to you.

Cheers

After rereading your script I do see that you’re setting the value of fm["tags"], not changing the list which would be needed to preserve other tags.

With that in mind I think it’s safer to use a dedicated status property to avoud interference with other tags. Then you can change it to your hearts desire and not worry about side effects related to other potential tags.

If you want to keep using the tags property you’ll also need to do the remove and insert operations in order to preserve other tags.

Yep.

And I don’t really need to rewrite my Templater scripts as they are all search and replace operations.
Only need to do regex replace for whole vault, but I’ve done that like 15 times over the last year… :slight_smile:

I converted the tags in question to status fields, then I tried the script, to no avail.

Even the example script using test files in yesterday’s test vault didn’t work.
I tried making a Templater Js script with the DV API, but couldn’t make it work. For that I needed to move the file into my Templater folder. When I reverted to the DVJs version, and moved back the file into the folder where the test files resided, it worked again.

It is some kind of bug with DataView. I tried on Linux, then on Windows, and then I found this thread:

So in any case, the logic sticks and it works, but there are many ifs and inconsistencies.

There is usually a better explanation than it’s just a bug, but hard to tell when we don’t see any query.

Your query, your properties, even the file names were the same.
It is some kind of indexing issue (both Linux and Windows environments were freshly indexed after making full-vault changes). Strange thing is, the testing was done in a separate test vault.

I tried a non-tp way (Obsidian API) as well, with similar reading error.

I like to end all things on a good note. In spite of trying and failing to iron out the kinks of the programmatic solution, and still saying it is a neat but not too user-friendly way of handling it, and bearing it mind I had the general user base in mind when I started out making this thread, I’d like to refer back to the GUI solution with the Projects plugin (which seems far more integrated to core Obsidian than the other plugin mentioned by me above).
Unfortunately, it doesn’t have a docs or how-to page, and I don’t mean to go into details, either.

But the usage is simple: you can use it without DataView installed or enabled but the real power is with DataView and even if a red rectangle appears about it being a read-only status, you can create and even update fields in batch safely, with the same Add field button.
Status fields are updated without creating duplicate YAML keys (or properties). A slight beef is that you cannot create new tags if there is one already in your list of tags. It will be updated with the new one given.

I’m wondering, but it’s hard to test and debug when it’s not happening in my vault, that your error is caused not by a bug in dataview but rather by trying to access file details of part of a non-existing file. At least that is what the error message eludes at…

I am using the Obsidian program as interface. In the folder structure, the files are there. Plugins are installed and enabled. DataviewJs example script kindly shared by you (and working twice before) is there, example files with properties used by you are there. I even went out of my way to put in all properties used in if clauses in the script, although even I know how if clauses work: if there is no match, it goes to the next step.

So the file read issue comes from somewhere and after 2-3 hours of trying to find a cause so I can get past this and maybe file away a nice snippet I could use sometime later, I had to leave this behind, being out of ideas now.

Even the trick of moving the file out of the folder and moving it back (as a way to update the index?) didn’t work a second or third time. There is something off.

If you’ve localised the issue to be within a given folder of files and queries, would it be an option to zip that stuff together and post it here, so someone could look at it?

(And I have to ask, you are on the last version of Obsidian and Dataview and so on? )

I could do that, actually, but I probably won’t.

As for the second question, after doing my bit with the chat robot trying to find the problem, the latter part of the day was spent on GH issues pages of the Metadata Menu, Projects and DataView plugins, and I actually copied out some URL’s where there is talk of issues or stuff not working, etc., but I am simply not cut out for investigating – simply because my main vault rarely needs needs update jobs and I’m not too personally interested in the quirks of databases, etc.

So yes, I even thought about rolling back versions of DataView and Templater as I have latest versions.

I rolled back DataView version for a try – nothing.

I used the script in a Sandbox vault, on four files having the exact same properties in files – same error.

I forgot to mention with Chat-GPT I made various scripts with console logs after each line and checked the Obsidian console for progress. It found the files but cannot update frontmatter.
I am doing this again in Sandbox vault:

Script used:

```dataviewjs
const tp = app.plugins.plugins['templater-obsidian'].templater.current_functions_object;

// Get files from a specific folder or directory
const folderPath = "Script try";
const files = app.vault.getMarkdownFiles().filter(file => file.path.startsWith(folderPath));

console.log('Number of files to process:', files.length);

for (const file of files) {
  console.log('Processing file:', file.path);

  const tFile = await tp.file.find_tfile(file.path);
  if (tFile) {
    console.log('Found file:', tFile.path);

    try {
      await app.fileManager.processFrontMatter(tFile, (frontmatter) => {
        console.log('Existing frontmatter:', frontmatter);

        // Update the 'one' property
        frontmatter['one'] = 'second';

        console.log('Updated frontmatter:', frontmatter);
      });
    } catch (error) {
      console.error('Error processing frontmatter:', error);
    }
  } else {
    console.log('File not found:', file.path);
  }
}
```