#46

Not at all; in fact, quite the opposite. I’m here because databases box you in, require you to mold your brain and workflow to a rigid structure, and generally make you feel like a data entry clerk, not a creator, a conjurer of possibilities, and an architect of new relationships. I don’t want a database. I want something quick and scalable, that’s free-text and flexible. I don’t want ‘mandatory fields’, in the classic sense, but I do want a back-end invisible technology that will allow me rapid access to the most un-rememberable and unreachable ends of my note landscape. So I can quickly connect something from 7 1/2 years ago with something I just wrote, just by remembering a bare smidgen of detail, barely, and nothing else to go on. That requires full text search, because I don’t know where that smidgen was.

3 Likes
#47

If you follow the Discord, you will also have noticed how often Licat talks about optimising by partial loading of the data.
As I said, at some point I would expect a plugin with an index (actually index implies database, so a plugin with a database); maybe just for search, maybe for other functions. When that happens, you’ll be able to measure the impact on general performance.

#48

You have full text search now. An index is a database - you talked about just adding an extra column to the table. It’s grep against an index. There’s no free lunch: whichever approach you take there are some gains and some losses.

#49

Yes, but it’s not indexed. That’s the whole point.

#50

Thanks for the heads up on this, @Dor. I am concerned about scalability and would love to know more about the vault organization strategies to which you are referring.

Can you recommend a good starting point for learning more?

Thanks in advance.

#51

Probably see:

2 Likes
#52

@ryanjamurphy ~ Many thanks, Ryan! Very much appreciate this helpful tip for learning more . . . .

Thank you, @Dor, for taking the time to provide this great overview.

1 Like
#53

Just to add to the discussion. I have over 9000 densely linked notes with little to no performance issues (late-2018 MBP, 8GB RAM, 256SSD, 2.3 GHz Quad-Core Intel Core i5). Only issues are initial load of vault graph view. Other than that no issues.

ObsidianNovember12
ObsidianNovember121530×1548 619 KB

DEVONthink could certainly handly WAY more than this - hundreds of thousands, based just on files (though it is databased in ways).

4 Likes
#54

Thanks for this update. I would like Obsidian to be able to handle the load that you mention devonthink handling. 9000 is great, certainly impressive, but in many other industries, still a very low number. Would be absolutely lovely to see attention and resources temporarily diverted from the nifty-cool-shiny-handy-smart feature updates, to just a little — just a little-- time toward adding a robust, scalable, industrial-strength indexing (and other) under-the-hood technology to let the thing shine at much higher numbers of notes. If we can see future-proofing at 1mm to 5mm notes, then noone will have to even bat an eyelash at 200k… You want loads more buy-in from corporate customers for enterprise licenses and tons of sync subscriptions? Make these changes, I suggest.

3 Likes
#55

You’re definitely not wrong there. I would imagine that would be dev in tandem with native applications. Because electron would be too intense on resources. Different project, but look at TheBrain 1mm+ notes are feasible.

#56

My two cents on this: The devs consistently spend time optimizing for “large” vaults, which is why the app almost doesn’t feel like an electron app. Given that it’s in beta, I would argue that optimizing now for millions of notes falls in premature optimization.

There are other features that are important for the sustainability of the app and other features that would make the app even better for the vault sizes we’re seeing today.

I’m not saying that this is not important, but it might be good to adjust your expectations on this. I’m sure it will come in time! If you do have some performance requests with large vaults, I’m sure the devs will appreciate more data points!

3 Likes
#57

Well, not what I wanted to hear, but it is what it is, and so I thank you for your input.

I would venture to think that you’re not seeing vault sizes that big, because the product is so new, and noone has built up a 5mm note vault. But – not because they are not needed or wanted, that’s not necessarily the case, I believe.

In the (hopefully short) interim between now, and screaming hot performance at vault-wide full text searching at 2mm notes, is a workaround to just split the vault? Ten vaults with 25k notes each? But then these cannot be searched all at once, right?

#58

ripgrep is a good example of how a highly optimised tool (multi-threading) could search and find results from hundreds of thousands of files within seconds. Visual Studio Code is also using ripgrep as their default search tool. The tool is isolated in the VSC code so there’s the option for the Obsidian team to integrate it in.

Most other indexing tools and systems have to deal with binary data, as Obsidian only uses text files the search solution does not require indexing.

5 Likes
#59

Well I’ve heard about vaults of varying sizes, anywhere between 200 to 40k notes. I believe there is one bible vault with around 80k notes too. You might want to ask some of those users directly in the discord server about search or how they manage their large vaults.

1 Like
#60

Question of atomicity

#61

My first thought is that you may want to try out DevonThink, which is a knowledge management software that may serve this purpose well. Basically I use DevonThink to collect everything, and use Obsidian to organize my own thoughts/notes.

The best part is, I can add the Obsidian vault to DevonThink, so it can interact with “raw inputs” without any headache.

#62

that definitely sounds like an awesome worklow, seems like devonthink is mac only. Do you perhaps know of any alternatives for us Windows / Linux users?

#63

LOL - the answer there is no, I’ve searched for years.

1 Like
#64

that’s a shame, I’ve also been looking for a while. Really hope one day there’s a good alternative for non-mac users

#65

While I don’t know all the secrets Devonthink offers, maybe some parts of it (namely indexing) can be tackled with fzf and rofi? There’s also zzzfoo that uses recoll?

1 Like