#69

Could the owners of Obsidian please pay more attention to this issue? @Licat @Silver

To me it’s the future, targeting future-proofness of the product as far as scalability & performance goes, and targeting a market (larger customers) with bigger fish to fry and this bigger software budgets…

1 Like
#70

There is no maximum number of notes in a vault. Yes, of course there is performance impact the more notes you add (doh!), but is not that simple. Obsidian does many things (the graph, the linked mentions, the unlinked mentions, etc) and not everything is impacted the same way and not everything depends only the number of notes, interconnections matters, and the attachments too. Some functions suffer more than others.

When you say other software can handle more notes, that is possible. But what features do these software exactly offer?

There is definitely (and always) room for improvement. However, to be clear many things are already “cached” (organized in databases). Adding a full text search engine, which I am sure will happen at some point in the future, will speed up search but have no effect on improving the graph processing on 1M notes. And it’s not a free lunch either, the indexer will have to run constantly in the background (Some people don’t like seeing an app using 5% cpu at “idle”).

My personal opinion from observing the community discussion is that the vast majority of our user are fine with the current level of performance and demand other things (better mobile, a wyswyg editor, sync sharing, etc). Those things will be addressed first.

2 Likes
#71

Also: please don’t tag the devs begging for attention like this. There are over 2000 other feature requests on this forum and all of them are as important to the person who asked for them as this is for you.

2 Likes
#72

WhiteNoise, thank you for the prompt response. I understand the points. I understand your valid reminder of the existence of tradeoffs. Consider that the reason there is not a clambering for more scalability/performance, is that Obs have not catered to-- and not made visible inroads into-- the enterprise/performance market. Yes, the ‘mom and pop’ shops, writers, independent academics, students, that is a larger bottom-of-the-pyramid market. Many of them are indeed leading the way towards innovative feedback and ideas for usability and plugins, even developing them. But beyond that market horizon is enterprise-class performance, and enterprise-class profits for Obsidian. Not abandoning who you are as a product and a vision, but just redirecting more percentage of planning and R&D horsepower toward scalability and viability in the broader and more lucrative SMB & corporate market.

#73

This depends by what you mean with “enterprise/performance market”. Obsidian can be used today for team documentation writing, maybe team project management if the users are inclined to use a text based solution. These use cases will benefit from shared sync.

If by enterprise/performance market you mean “a big company that needs a CRM software”, then Obsidian is not good for them and probably will never be good for them.
I believe the bigger you go the more structured information you need. They need a real database-based app with a fixed schema not text files.

1 Like
#74

[…] “enterprise/performance market” […]

Obsidian as it currently stands may get used in an enterprise setting but it is just some personal software which, as long as it doesnt ever need admin rights, will likely fly under the radar. It isnt something I would classify as ‘enterprise software’. That would be things like a CRM. Or Office 365. Or Jira. My team has actually has been using Jira in a very (form driven) Zettelkasten(ish) way which works surprisingly well considering that wasnt the intent for the software. All of these will have a large annual maintenance fee and a team that supports it.

Back closer to the topic:
It is a given that there will be an issue (ie application performance) when user expectations dont match the architectural decisions. It is also a given that a business needs to keep the value proposition high enough that they retain the (paying) customers. By these means the business can set priority on value-add items (ie the wyswyg editor, iOS Shortcuts, and etc). For my part I, and I suspect WahWah as well, just ask that this doesnt end up at the bottom of the pile.

Personally, Im really hoping Ill want to use AND be able to use Obsidian for the next decade or 3

1 Like
#75

As for enterprise documentation, we break them down to manageable depots as nothing really scales and it becomes subject to problems.

For Obsidian, for huge collections is a matter of building separate vaults for sub-directories and open each up per demand.

BTW one of my vaults has now 12k entries and works just fine. Helps to use a fast SSD drive separate from the main boot drive and also make sure the right file system is created on that drive.

PS: Integrate ripgrep as default search engine – same Visual Studio Code did - and the search performance is super-fast. But for the time being the existing search performance is fine for me.

1 Like
#76

As for long term scalability and Obsidian. These are text files in a file hierarchy, if Obsidian stops working due to millions of files, the data is still there and one could use grep and other plain text editors, until those neither scale up. Next step is to install an Enterprise text database, similar to what Google and search companies use.

#77

There is no need to integrate ripgrep because chrome is implementing the same regular expression algos directly in V8 and we will get those improvement downstream at some point.
Also ripgrep doesn’t run on mobile.

#78

I have been using Obsidian for only the past month or so. I migrated 27,000 evernote notes (which is only about half of my evernote database). Performance is fine on my laptops, even my Surface Go with only 4mb ram, with a few caveats:

  • it took hours to index, and appeared to be hung many times, but once I just left it alone it finished
  • search is SLOW. Especially compared to evernote, which searches the same notes in seconds. Seems like the index could be improved to speed this up. Gmail has no problem with many times as much content.
  • search sometimes totally fails to find a note that is easy to find in evernote. My working theory is that terms that appear more than x characters into the note are not indexed. I have tested this by searching for a unique term in a note that appears toward the end, and another unique term at the beginning of the same note, and seeing obsidian find the latter but not the former.

And also, I cannot open this vault at all on mobile. It indexes about 350 notes then crashes every time. I assume that’s because of my small amount of ram and, believe it or not, I just ordered a Samsung S21 to see if I can open this same vault on a better phone :slight_smile:

2 Likes
#79

Evernote is a web database. Gmail is a web database.
You’re not going to see the same performance on a local machine however optimised its search is.
Quite a few technologies have switched almost entirely to the web for this reason (grammar processors, translators, speech to text etc).

If the computer is powerful enough, then it should be sufficient for the maximum number of notes most people will make. But limits there will be.

#80

This would be absolutely awesome.

#81

ripgrep is written in Rust - 100x faster than any Javascript. And you’ve be amazed what runs in mobiles nowadays.

#82

Properly tuned Javascript is maybe 10%/20% behind native speed essentially because it is converted to run V8’s c++ code.

#83

The issue is memory consumption with Javascript objects that makes most browsers page and slow down things, including eating up batteries. Hence many browser implementations are now trying to optimize this by wasm. By using rust as wasm frontend. But might use rust in native mode anyway.

#84

So I imported the entire pathfinder 1e wiki into obsidian using a custom script to convert the HTML files I had into markdown files.
The entire vault contains around 35k files.
The indexing took about 10 minutes and the navigation is surprisingly fluid. The search takes quite a bit of time to find all the results, but that is expected.
The graph view on the other side is not usable with so many files. After loading for about 5 minutes obsidian black screened and I had to force close it.
So I guess I found the limit xD

This is the last image I have of it before the black screen.
(looks kind of cool, though I would love a full render of the graph)

pathfinder_graph_1
pathfinder_graph_11555×1221 156 KB

System specs:
RTX 3070
Ryzen 3800 XT
32 GB RAM

5 Likes
#85

Thanks much for that data point. Bad news that it can’t handle that load well. And you’ve got an ample system. Even worse, there’s not much chatter about development activity related to performance improvements at volume and scale, at least nothing I can see. Gorgeous graph!

#86

how’d you actually do this? I wondered if something like this can be done for other systems/editions. Would certainly be interesting as a quick lookup :stuck_out_tongue:

Also, dang that image looks sweet already.

#87

only the global graph view doesn’t work at scale, the rest seems to work even at 100k notes Interlude: Obsidian vs. 100,000 - by Alexander Rink. Also apparently Obsidian actually performs better at scale than competitors TfT Performance: Interim Results - by Alexander Rink

4 Likes
#88

This sounds like good news. I’m skeptical, but Rink looks quite thorough in his methodology, and even that he wants to – or sees a need to properly wade into performance waters. Great link. Thank you.