Testing several versions with Dropbox - 10GB / 10K files vault - vanilla - no plugin

Help
1

What I’m trying to do

I would like to install Obsidian, open a dropbox folder with 10-100GB of mixed data including :

  • media files
  • java / programs source code
  • linked notes

This study has been carried out as I was never able to use v1.5.12 and see my folder tree inside Obsidian

Things I have tried

… to be continued

2

What I’m trying to do

I would like to install Obsidian, open a dropbox folder with 10-100GB of mixed data including :

  • media files
  • java / programs source code
  • linked notes

This study has been carried out as I was never able to use v1.5.12 and see my vault folder tree inside Obsidian app.

This particular 1.5.12 version has been initializing ok on an empty local vault on Dropbox but not on the existing folder - where it hanged on loading cache / workspace. Tried this one several times. Killed process through Activity Monitor

Hardware is Mac i7 2.3 + 16GB RAM - loads of space on hdd.

Things I have tried

So to refine and attempt the testing I iterated the following scenario :

  • delete Obsidian.app
  • delete .obsidian in vault folder location
  • delete ~/Library/Application Support/obsidian
  • install obsidian through Mac installer
  • open folder as vault
  • no pluggins
  • no change of settings
  • created a note > observed lag on note title renaming - then updating ok
  • restart app to observe full reindexing (expected behaviour ?)
  • dropbox folder setup to be local to ssd drive

v1.5.12 > see above
v1.5.8 > hanged on loading Workspace > killed process
v1.5.3 > opened ok - initial index in less <2’ / restart + full reindex in <2’
v1.4.16 > could not boot the app
v1.4.14 > opened ok - initial index in less <2’ / restart + full reindex in <2’ / note creation lag

Same behaviour on v 1.4.12, 1.4.11, 1.4.10, 1.4.5, 1.3.7, 1.2.8 & 1.1.16

Other observations

When creating a sample vault with low number of data - indexing is invisible
=> It seems to do it so quickly that there is a optical illusion of no reindexing but reindexing happens

Questions
1/ what is expected between initial start and subsequent starts ?
I understood :

  • Start 1 > initial indexing - ok “happens only once”
  • Start n > in my view there should not be another Initial full indexing - but intermediate one only if files have changed between sessions

2/ what could explain the lag when renaming a note in obsidian ? the workflow is the following

  • create note
  • change title
  • press Enter to edit content > it takes time to have the cursor going on the note body section to add text
  • does not happen on folder sitting on hdd without being syncing on DB - can there be a thread locking mechanism interfering ?
3

How do you ensure, that the local Dropbox folder contains copies of all files?

In MacOS the behavior of cloud drives has recently changed, so that files sometimes look more present than they actually are.

And where do you store the local Dropbox folder?

4

I have checked all the settings on dropbox to make local copies - see links bellow.
The folder is in the standard /Users//Library/CloudStorage
This is forced by Dropbox last version

For the time being - there is a regression compared to 1.5.3 which I am reverting to for now so the initialization can happen and I can start building the vault.

Also Obsidian app is set not to sync with iCloud

It’s possible that MacOS / Dropbox have introduced some change in their “apis” that are affecting Obsidian behaviour - this is a real show stopper for those later versions including 1.6.2 which has the same behaviour as 1.5.8 /1.5.12.

Still versions < 1.5.3 are initializing correctly, albeit the full re-index on each app restart - which is less of a problem compared to not initialising at all.

This one seems tricky - good luck

5

So I am progressing - the topic is that I have a folder with java and other source code in the vault folder.

This is containing 200K files so probably a challeng for the local obsidian db to maintain the index in shape.

That being said I experimented the following on 1.5.12

=> copy my dropbox content to the Local Document folder (no icloud no dropbox)
=> open that folder as a vault
=> opened ok with indexing 10K files out of the ~200K

Closed obsidian

Restarted > reindex happened

Then I removed the ~200K sources files out of the vault

=> obsidian indexed in a blink - but I still think the full reindexing happen on all the program startup - not just initial

=> the main difference is when vault is on non Dropbox folder, even if big, it can start (5.12). When on DB with local file - it does not.

Is this expected ? will it be investigated by the team ?

6

Last update

Installer v1.5.3 and app 1.5.12 or 1.6.2 > the vault is loading an reindexing at every restart

But the program copes with one of my folder at 200K in the vault.

I put dropbox sync hon hold while opening obsidian.

Weird

I’ll keep the topic open but not too long if things get stabilized

7

060624 - new experiments of the day

1/ 1.5.3 installer + 1.5.12 app is quicker to load vault from existing folder than 1.5.12 installer + app which never gets initialized on my big vault

2/ ignoring a folder in a vault generated from an existing folder doe not have benefits on performance - ie creating an new note is taking 1 second+ to be committed on file system

3/ creating a new vault from scratch in dropbox root => creation of a new note is immediate w/o lag
=> size of folder has a real impact on the i/o generated for new note

4/ when using a bigger vault - CPU is 100%+ / go normal when creating a vault from scratch

next we’ll see if usable on M3 mac to come

8

Some more thoughts:

a) Maybe I have overlooked it, but have you tried a setup without Dropbox? My approach to debugging an Obsidian vault typically starts like this:

  • Create a new local vault on my Mac. Make sure that it is in a location that is not being synced with any cloud. (Depending on your MacOS settings, MacOS might be syncing Desktop and Documents folders to iCloud without you noticing it.)
  • Use Mac’s Finder to copy some of files from an external source to the new vault. See how Obsidian handles them. Then add more files. See how Obsidian handles those. Repeat until all files are copied to the new vault.
  • Only when I’m sure that Obsidian can process this vault offline, I start exploring sync options.

b) Explore sync options with smaller vaults. Does Obsidian work at all with Dropbox? Obsidian can’t access Dropbox servers directly. So it works with a local copy that is managed by Dropbox’ official app or some other apps that you might use to sync Dropbox servers with your devices. There’s a lot that can go wrong here, even with tiny vaults.

c) Number of files. In my experience Obsidian can handle 10K files easily. But I don’t recall any success story that involves 100K files or more.

d) File names and types of files. Obisidan seems to handle some file names and some types of files better than others.

e) There’s a feature request that addresses the use case of large folders with mixed data. Many problems should disappear, if it were possible to exclude certain file types, file names or whole folders from indexing:

1 Like
9

Hello @harr

Thank you for your precious help !

I did follow the exact same approach as you to pinpoint the root cause.

Solution is point c)

So I have put my source code project out of dropbox itself - after some thinking I also wondered how my usual IDE (Intelij) would behave with classes source on a cloud provider - even with local copies files
That source folder was 200K files.

As I write I just retried the opening of my “root” folder with complete directory structure and usual workload files and docs > works like a charm - we’re back to business.

Will close the topic for now

Thanks for your undefeated support - much appreciated !

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.