I had a look, mostly for what you got out of Datacore and how.
I was so impressed I hardly looked around thoroughly enough before coming back here to say thank you.
I’m glad you liked it! I’d be happy to answer any questions you have. Also, the ‘Page Task’ along with Datacore is pretty cool—you can even expand it to run different commands based on properties. The script that changes the icon when you press Ctrl + Space can be found in the Homepage and under SYSTEM > TEMPLATE > CODE.
I’ll probably start tinkering with it tomorrow (today) and ask you about stuff.
For now, just one thing: any major updates on your vault/GitHub repo, especially technical, use this thread to let people know.
On my hobby laptop, I tested your vault for performance. First with 15k files, then your own leaner vault.
I hesitated about being a busy-body about this, but thought I’d mention it anyway, mainly because I’m not a software engineer or web developer and I don’t know the answers.
If in your vault I remove the excluded files (actually, folders) in Files&Links and run a search for path: SYSTEM/TEMPLATE/FORMAT /dat, results are pretty slow coming in with all your CSS enabled.
Once I remove the larger CSS files from .obsidian (and there is also a CSS folder somewhere else), the 200-odd results come in instantly without any stutter of the progress bar in the Search Modal.
I’m just mentioning this because the hefty CSS files in .obsidian are obviously built upon previously made ITS and efemkay snippets and there could be something wrong with them. I suspect issues with the view.css files in the SYSTEM/TEMPLATE/CSS folder as well.
If you don’t see any issues, try adding 5-15k files and try the search.
I just tried much same search for 1586 results in my own vault with all your CSS removed and there was no sign of a lag or stutter.
Tasks also has issues with large vaults, apparently.
So far, based on tests, having Datacore on a large vault doesn’t cause problems.
thank you so much for this. I’ll work on improvements maybe over the weekend. These are very valuable observations. Any idea on where I could get realistic dummy data? I was looking to run test sets to measure performance metrics: 5k, 10k, …, 100k notes.
With this I could hopefully arrive with a solution for very large vaults using the template.
I already have something in mind with tasks and why they are slow on very large vaults but for now focus will be on getting that dummy data.
I never tried it, personally.
Then a regex replace Python script could be used to add YAML to all files, based on filename, creation time, etc.
I have also found that Datacore works with snake_case property keys well but not with camelCase…
Created an account here just to comment about how insane this vault is. really mindblowing. Playing with it today but just want to share my appreciation for the work here.
I’ve seen you finalized your vault and possibly do not take FR’s on your GH.
Can I ask here why in the Map of Content file one can edit the property (cell) values if the changes don’t persist and reflect on the given file’s frontmatter? (the mobile version of M of C doesn’t have this).
Are you planning to add this support?
Thanks
Compare DV implementation here - never tried it but this would be awesome with Datacore.
Sorry, that completely flew out of my head. Anyhow I built a fix for your concerns:
Properties are not being updated
Properties cannot be selected for mobile version
You can download the updated version of the Map of Concent and Mobile Map of Concent
Requirement:
Install and enable the MetaEdit community plugin.
Changes Made:
We created an updateNoteProperty helper function that connects to MetaEdit’s API (app.plugins.plugins[“metaedit”]) to handle the actual note updates.
We built an EditableCell component that manages the editing state and shows a confirmation dialog before making changes, using useState and useRef hooks to track values.
We modified the table rendering to use EditableCell instead of just displaying values, allowing users to click any cell (except file names) to edit properties, which then triggers the MetaEdit update process through our helper function.
Tried it.
Awesome!
This will really rock Datacore users’ experience.
I wonder though:
Why is MetaEdit needed when processFrontMatter is available from Obsidian API?
The confirmation pop-up is nice but could be considered unnecessary as one can re-edit the single cell value if some type was made. When making a lot of changes, this would hinder a good workflow. On pressing Enter, the change would be registered and a notice could be flashed in the upper right-hand corner to let the user know.
I am of course asking these on behalf of the wider community.
I am having issues with the Obsidian API, so I just resorted to Metaedit API which was a suggestion from this article. If you manage to find a solution with Obsidian API, would love to have a look at the code.
There is a small issue with the code, when you edit again the value displayed will be rendering the 1st Update value (but the note property will have the most recent update value). While it is better to just have the notif at the side, this will do as a quick fix to prevent users from making unnecessary/accidental updates. Will send here if I find a solution.
For those of you who are curious with the startup time, here is a breakdown:
If you’re not using todoist, removing the plugin will bring down the time to just 2 seconds
Excalidraw also takes time to load on startup
Note:
From my tesing Lazy loading can bring things down to 1 second startup time even with all the plugins but it causes issues with the Metabind buttons that are nested from tabs
Obsidian start-up time breakdown
Obsidian version: v1.7.7
Operating system: Windows 11
That’d be robot-made code, so I’ll pass.
I may try anyway when I get the time.
Just remembered that MetaEdit has not been updated for quite some time, but pretty sure it uses Obsidian API under the hood, so no biggie. Just want to cut down on installed plugins, is all.
I have found some plugins must be added as Instant loaded, e.g. to counter Commander custom-icons on mobile toolbar issues, etc.
Still, the majority of plugins can be loaded on startup.
I tried your vault with 15k notes (extensive links usage) on a faster laptop (Intel i5, 16GB RAM) on Windows 11, with all the CSS I considered “heavy” enabled and most if not all of the plugins that I knew to be necessary for all the functionality users will need.
I was running multiple DVJs queries, Datacore, Omnisearch queries (users with larger vaults may want to increase the searchDelay to 1000ms or more in the data.json of the Home Tab plugin with the plugin disabled for the duration of this edit), flipped stuff around and was watching the Task Manager.
I felt a lag with page scrolling and typing when the RAM consumption went up to 4-6GB. After disabling Omnisearch (and a couple of other plugins that I found substitutions for, namely, Editing Toolbar and Style Settings), the sluggish behaviour is gone. Mind you, I have several plugins (DV, DC, Obsidian Query Language, Graph Analysis, etc.) that do indexing so other people may not need to chuck Omnisearch (which I will miss mostly for the fuzzy search capabilities, otherwise I sometimes use the ripgrep function of Another Quick Switcher with regex).
BTW, Editing Toolbar has a fix in the form of a main.js, when downloaded and installed, CPU usage will go down in the main process of Obsidian.
Style Settings substitution is just pure CSS, not another plugin, of course.
In the meantime, I have also found that extensive backlinks and the enablement of the Graph core plugin slows down Backlinks results generation. Had to disable the Graph plugin. This is a known issue. For Local Graphs, users can try the Juggl plugin as a replacement.
In the end, I again had to strip off the extra functionality (bar Datacore, which I took on earlier). And even after discarding my changes, the vault was sluggish. I have an idea that either Omnisearch or quite possibly the Tasks plugin corrupts the index* and that’s why the Backlinks are super slow and over time the general UI handling starts being off.
So if I may be inclined in the future to try testing again, I’d start with these two plugins removed.
Update 1
I have gone ahead and did some more testing.
My Backlinks results super slow issue comes from either of the following Multi Columns.css files (I tried these on alternate occasions): MCL Multi Column (from the current vault, possibly from here, but in we find a variant (I didn’t diff them) of it in another Demo Vault here).
Anyone can try it for themselves. Enable either of the snippets from either of the Demo vaults. Open a note in your own old vault with many backlinks upon startup. Backlinks come in fast. Now open a note where you have callouts. Go back to the earlier note and watch the results come in in Backlinks. Slower now with the progress bar stuttering. This on a freshly installed Windows 11, with freshly indexed Obsidian 1.7.7., with no community plugins.
We see many issues at Efemkay’s GH site. One of these from a can-do person with diagnostic skills feature performance issues:
Update 2
I have finalized my existing vault with most of the functionality of the demo vault by OP.
Unfortunately, trying an earlier version of the Multi Columns snippet didn’t solve the slow-vault problem in the long run.
I used the Obsidian Columns plugin, which took some fiddling around; luckily, the Meta Bind buttons were outside the scope of the columns, otherwise these plugins don’t play well together.
Instead of Omnisearch (which doesn’t work well with the Lazy Plugins loader),
I used Another Quick Switcher plugin’s ripgrep function (users need to install it; in the settings, it is called simply grep, for some reason, though).
Luckily, the files in the CSS folder don’t cause performance issues.
I didn’t look for replacements for other Efemkay snippets.
Update 3
About the Image Toolkit plugin: it also has some issue at my end as the CPU on the main process shows 6% when enabling it. The other plugin with the same functionality – Awesome Image – doesn’t cause this problem.
Apparently, Tasks causes no problems whatsoever on large vaults anymore.
I hope this helps users of large vaults with your building blocks.
Thank you again for your great contribution!
* Built-in Obsidian ‘Rebuild Vault Cache’ wasn’t even enough; I had to wipe the whole config folder in ‘C:\Users\username\AppData\Roaming\obsidian’ and re-add my vaults, and now it’s back to normal.
