#63

LOL - the answer there is no, I’ve searched for years.

1 Like
#64

that’s a shame, I’ve also been looking for a while. Really hope one day there’s a good alternative for non-mac users

#65

While I don’t know all the secrets Devonthink offers, maybe some parts of it (namely indexing) can be tackled with fzf and rofi? There’s also zzzfoo that uses recoll?

1 Like
#66

Interesting… discussing a theoretical issue that probably nobody in real life has.

Can you imagine all the notes you could have written while discussing this.

#67

Unhelpful. Just because you may not have and issue, doesn’t mean others don’t. Many may want to deploy obsidian in the enterprise, so scalability and performance are actual concerns, interests, and unf roadblocks to adoption.

3 Likes
#68

FWIW, I was reading this thread because after migrating from Evernote (yay YARLE), Ive noticed how sluggish Obsidian got with some operations (ie renaming a file, deleting a file, searching)

For reference, according to Vault Statistics, I have 10704 notes, 17557 links, 81460 files, and 70756 attachments. My vault is divided up by years and, currently, goes back to 2011. This is also on my iCloud drive. Ive concluded that their mobile sync is not great.

Prior to Evernote, I was storing a lot of my stuff in email. These go back to 1995 and I still use them for reference. Ive been contemplating pulling them in too but its entirely possible that I could hit 200k files.

If you havnt guessed, Im on the collector end of the spectrum and, prior to Obsidian, all the linkage has always been in my head (and/or via tags after they became a thing). Im workin on that.

So yes, please test and tune for 5M+ files and add some form of caching process to speed up operations. :slight_smile:

3 Likes
#69

Could the owners of Obsidian please pay more attention to this issue? @Licat @Silver

To me it’s the future, targeting future-proofness of the product as far as scalability & performance goes, and targeting a market (larger customers) with bigger fish to fry and this bigger software budgets…

1 Like
#70

There is no maximum number of notes in a vault. Yes, of course there is performance impact the more notes you add (doh!), but is not that simple. Obsidian does many things (the graph, the linked mentions, the unlinked mentions, etc) and not everything is impacted the same way and not everything depends only the number of notes, interconnections matters, and the attachments too. Some functions suffer more than others.

When you say other software can handle more notes, that is possible. But what features do these software exactly offer?

There is definitely (and always) room for improvement. However, to be clear many things are already “cached” (organized in databases). Adding a full text search engine, which I am sure will happen at some point in the future, will speed up search but have no effect on improving the graph processing on 1M notes. And it’s not a free lunch either, the indexer will have to run constantly in the background (Some people don’t like seeing an app using 5% cpu at “idle”).

My personal opinion from observing the community discussion is that the vast majority of our user are fine with the current level of performance and demand other things (better mobile, a wyswyg editor, sync sharing, etc). Those things will be addressed first.

2 Likes
#71

Also: please don’t tag the devs begging for attention like this. There are over 2000 other feature requests on this forum and all of them are as important to the person who asked for them as this is for you.

2 Likes
#72

WhiteNoise, thank you for the prompt response. I understand the points. I understand your valid reminder of the existence of tradeoffs. Consider that the reason there is not a clambering for more scalability/performance, is that Obs have not catered to-- and not made visible inroads into-- the enterprise/performance market. Yes, the ‘mom and pop’ shops, writers, independent academics, students, that is a larger bottom-of-the-pyramid market. Many of them are indeed leading the way towards innovative feedback and ideas for usability and plugins, even developing them. But beyond that market horizon is enterprise-class performance, and enterprise-class profits for Obsidian. Not abandoning who you are as a product and a vision, but just redirecting more percentage of planning and R&D horsepower toward scalability and viability in the broader and more lucrative SMB & corporate market.

#73

This depends by what you mean with “enterprise/performance market”. Obsidian can be used today for team documentation writing, maybe team project management if the users are inclined to use a text based solution. These use cases will benefit from shared sync.

If by enterprise/performance market you mean “a big company that needs a CRM software”, then Obsidian is not good for them and probably will never be good for them.
I believe the bigger you go the more structured information you need. They need a real database-based app with a fixed schema not text files.

1 Like
#74

[…] “enterprise/performance market” […]

Obsidian as it currently stands may get used in an enterprise setting but it is just some personal software which, as long as it doesnt ever need admin rights, will likely fly under the radar. It isnt something I would classify as ‘enterprise software’. That would be things like a CRM. Or Office 365. Or Jira. My team has actually has been using Jira in a very (form driven) Zettelkasten(ish) way which works surprisingly well considering that wasnt the intent for the software. All of these will have a large annual maintenance fee and a team that supports it.

Back closer to the topic:
It is a given that there will be an issue (ie application performance) when user expectations dont match the architectural decisions. It is also a given that a business needs to keep the value proposition high enough that they retain the (paying) customers. By these means the business can set priority on value-add items (ie the wyswyg editor, iOS Shortcuts, and etc). For my part I, and I suspect WahWah as well, just ask that this doesnt end up at the bottom of the pile.

Personally, Im really hoping Ill want to use AND be able to use Obsidian for the next decade or 3

1 Like
#75

As for enterprise documentation, we break them down to manageable depots as nothing really scales and it becomes subject to problems.

For Obsidian, for huge collections is a matter of building separate vaults for sub-directories and open each up per demand.

BTW one of my vaults has now 12k entries and works just fine. Helps to use a fast SSD drive separate from the main boot drive and also make sure the right file system is created on that drive.

PS: Integrate ripgrep as default search engine – same Visual Studio Code did - and the search performance is super-fast. But for the time being the existing search performance is fine for me.

1 Like
#76

As for long term scalability and Obsidian. These are text files in a file hierarchy, if Obsidian stops working due to millions of files, the data is still there and one could use grep and other plain text editors, until those neither scale up. Next step is to install an Enterprise text database, similar to what Google and search companies use.

#77

There is no need to integrate ripgrep because chrome is implementing the same regular expression algos directly in V8 and we will get those improvement downstream at some point.
Also ripgrep doesn’t run on mobile.

#78

I have been using Obsidian for only the past month or so. I migrated 27,000 evernote notes (which is only about half of my evernote database). Performance is fine on my laptops, even my Surface Go with only 4mb ram, with a few caveats:

  • it took hours to index, and appeared to be hung many times, but once I just left it alone it finished
  • search is SLOW. Especially compared to evernote, which searches the same notes in seconds. Seems like the index could be improved to speed this up. Gmail has no problem with many times as much content.
  • search sometimes totally fails to find a note that is easy to find in evernote. My working theory is that terms that appear more than x characters into the note are not indexed. I have tested this by searching for a unique term in a note that appears toward the end, and another unique term at the beginning of the same note, and seeing obsidian find the latter but not the former.

And also, I cannot open this vault at all on mobile. It indexes about 350 notes then crashes every time. I assume that’s because of my small amount of ram and, believe it or not, I just ordered a Samsung S21 to see if I can open this same vault on a better phone :slight_smile:

2 Likes
#79

Evernote is a web database. Gmail is a web database.
You’re not going to see the same performance on a local machine however optimised its search is.
Quite a few technologies have switched almost entirely to the web for this reason (grammar processors, translators, speech to text etc).

If the computer is powerful enough, then it should be sufficient for the maximum number of notes most people will make. But limits there will be.

#80

This would be absolutely awesome.

#81

ripgrep is written in Rust - 100x faster than any Javascript. And you’ve be amazed what runs in mobiles nowadays.

#82

Properly tuned Javascript is maybe 10%/20% behind native speed essentially because it is converted to run V8’s c++ code.