Ignore/exclude completely files or a folder from all obsidian indexers and parsers

+1 No exclude for indexing is a blocker for the use case I care most about.

1 Like

An include config file would be just as vital. I have an existing research folder with all sorts of guff in it. I could sensibly and efficiently start using Obsidian to organise it, if only I could tell Obsidian to only include/index .md files, and files of other patterns I determine. There is such a variety of guff in my research dir tree that exclude rules would be onerous.

Exclude rules are still necessary, however, for dirs Obsidian doesn’t have permission to read (upon which it chokes, fatally), and dirs that are full entirely of chuff without any meaningful notes or research artifacts.

2 Likes

What is the point of offering the option of using an existing folder as a vault if include/exclude rules cannot be setup to manage it?

Politely adding my +1 to this. Similar use case, I’d like to store git repos in my vault, but node_modules kills the performance.

1 Like

I’m all for the include/exclude feature. I would also like a mechanism to (deemphasize/exclude/not index) folders based on if they contain a certain file, like how .nomedia makes the media scanner on Android ignore its containing folder.

+1 for this feature. In my scenario, often I want to paste some long text (e.g. very long logs from some code), and do not want to make Obsidian super slow because it will index everything.

+1 to this feature request!

my use case is having a repository with an Astro project that i have markdown files for blog posts.
it would be great to be able to create and edit those markdown files in Obsidian and then just go to VS Code to push it to the repo, while ignoring the node_modules folder entirely and not loosing any performance.

i already tried multiple regex to exclude paths with “node_modules” but none worked, Obsidian still slowed down.

1 Like

+1 for this!
Initially I’ve searched a possibility to name my attachment folder .attachment so it would be hidden by default from everywhere, but it breaks pasting into documents.

This option is reasonable alternative for my needs

This feature is a must-have. So, I decided to find a solution that works for me in Obsidian until I have finished developing my own “Everything App” :wink:

I’m using an overlay filesystem, with my Obsidian vault as the “lower” directory, and an “upper” directory that contains everything that I don’t want to be visible, searchable and indexed by Obsidian. I.e. GIT repositories, source code, binaries and whatnot.

This is how I mount it in Linux:
sudo mount -t overlay overlay -o “lowerdir=/path/to/vault,upperdir=/path/to/vault.diff,workdir=/path/to/vault.work” /path/to/everything

From the shell, from IDEs, and whatever other tools you use, you can use /path/to/everything. In Obsidian, you open /path/to/vault like usual.

Under the hood, the overlay filesystem will ensure that anything you create somewhere under /path/to/everything will be stored in the vault.diff (upper) directory. The “work” dir should just be an empty directory (initially), that the overlay filesystem will use internally for some operations.

Full documentation here:
https://docs.kernel.org/filesystems/overlayfs.html

There might be some overlay filesystem implementations (maybe FUSE-based?) for macOS and Windows as well, but I haven’t looked into it since it’s not really relevant to me personally. But for Obsidian users on Linux, hopefully this can be helpful! :slight_smile:

3 Likes

I’m learning about this overlay filesystem. I’m reading that the lower layer is read-only. Doesn’t this prevent you from using Obsidian to update notes?

Nope, the lower directory is only read-only from the context of the overlay mount directory :slight_smile:

Some key points:

  • Use the Obsidian vault as the lower directory
  • Keep using Obsidian via the lower directory directly
  • Create “invisible” files/directories via the overlay mount

In the overlay mount, you have the identical directory structure and the same files as in the Obsidian vault, but you are free to create new files and directories which will only be stored in the “upper” directory

The overlay mount is basically just a merged version of the lower/upper directories, and any changes you make in the overlay mount will only be made in the upper directory

Example:

Let’s say you have a vault, named FooVault at ~/FooVault

You would keep using Obsidian directly on ~/FooVault

Create the following directories for the upper/work dirs:
~/FooVault.diff
~/FooVault.work

These directories are to be used by the overlay mount, but you would generally not access them directly. Just create them for now.

Then you want a directory through which you would be able to create directories and files within the context of the vault, but without actually being stored in the vault. Let’s just go with:

~/foo

After creating the ~/FooVault.diff, ~/FooVault.work and ~/foo directories you run:

sudo mount -t overlay overlay -o “lowerdir=$HOME/FooVault,upperdir=$HOME/FooVault.diff,workdir=$HOME/FooVault.work” ~/foo

Let’s say that within FooVault you have a directory such as:

~/FooVault/Projects/Bar

Within this directory you have a bunch of notes and other things related to the Bar project.

This directory and all the files will also be visible at:

~/foo/Projects/Bar

Let’s now say that you have a GIT repo with source code related to the Bar project, you can now store this in:

~/foo/Projects/Bar/src

Under the hood, this will create:

~/FooVault.diff/Projects/Bar/src

When you are using your terminal/IDE/whatever, you’ll access it via the overlay mount at ~/foo, which will see everything combined from both ~/FooVault and ~/FooVault.diff

But when you are using Obsidian (+ in case you want to edit notes or reorganize your vault using any other tool) you would do it via ~/FooVault directly and not via ~/foo

The main thing to keep in mind, besides doing things that should be “invisible” to Obsidian within ~/foo and making any modifications to the vault itself (adding/editing notes, moving vault folders around) via ~/FooVault, is that in case you want to do something like:

Moving ~/FooVault/Projects/Bar to ~/FooVault/Projects/SomethingElse

From the context of the overlay mount at ~/foo you would now have

~/foo/Projects/Bar/src (mapped to ~/FooVault.diff/Projects/Bar/src) as well as ~/foo/Projects/SomethingElse

So to account for that, in case you’re moving folders around in ~/FooVault you need to move them in ~/FooVault.diff as well

1 Like

Hmm that part seems scary. Not sure I full understand. But good to keep in mind if I ever look at overlays.

Thanks for sharing this clever idea!

1 Like

Also note that some of the issues mentioned in the following help topic also relate to this FR: Stuck on Loading cache - #76 by FernandoIbrahim
I came across the topic since i already forgot that obsidian indexes everything and was wondering why it was taking so long to start after running experiments that created 125k files with a total of 20GB. Now I either have to clear the cache before starting obsidian and have it gobble up RAM while using a core full time while trying to index all the files in the background or throw my current folder structure overboard…
This is the second use case that I ran into personally where the option to exclude files would be great.

I wanted to throw in supporting this feature request. It feels like a must-have to me for software with the scope of Obsidian.

Logically speaking it’s a bit strange that Obsidian doesn’t have this feature already, considering that it already restricts what types of files it shows you. Why does Obsidian index or otherwise pay attention to files that are not shown within its user interface?

My own use case is that I have a folder for a project, and within that folder I have a number of git repositories. It’s very convenient for me to have the files organized this way, and it’s also very convenient to have the top-level folder be an Obsidian vault. However, whenever I actively work on one of the git repositories, Obsidian eventually hangs, I suppose because too many files are changing too quickly for it. Being able to exclude the folders, or even just the .git subfolders, would fix this issue and allow me to work the way I want.

3 Likes

I also think, this would be a huge feature, which is probably comparatively easy to implement.

This feature is basically already implemented via Files and links > Excluded Files. Maybe just copy this setting, name it Ignored Files and skip those files and directories altogether.

For large vaults this could mean a significant improvement.

2 Likes

Another use case, that this feature request would resolve: hiding temporary files created by Microsoft Office. Why should Obsidian index or display them at all?

4 Likes

I just want to weigh in on the issue: I think it can be a real show stopper even for potential commercial users as I could be. The users who use Obsidian as doc management system on their big monorepos with built binaries, dist/, node_modules/ etc. For me it makes it a real pain to open a vault as it reindexes hundreds of modules which don’t belong to my code.

Is there a way I can contribute to have it implemented?

Also, I’d like to add my two cents - excluding node_modules, .git, and, more generally, any unwanted files or folders is a critical feature for me. Especially since the Excluded files feature does not actually exclude but marks them as hidden.

Have to have this feature.

I’ve made a monorepo for all my projects and notes which are applicable across projects.

When I open Obsidian, it freezes on me(too many projects?).

In order to use Obsidian I planned to be able to open it, view all my code snippets/framework snippets/cli commands across all my projects at the same time so that I can build on them(think project 1.helloworld, 2.intermediate-hiworld 3.pro-hola-world etc).

However being unable to open Obsidian is a buzz kill.

I’ll also add that I preemptively created a new vault, added the “projects” folder to the “files exclude” settings, and then copied my folders over manually 1 by 1. Despite that Obsidian immediately froze when I drag and dropped the projects directory into the vault.

I used

/Projects/
/Projects/*
/Projects/**

Any suggestions?

Obsidian has been really buggy for me for quite a while - I was losing faith! Now I finally know why!

I had assumed that “Excluded Files” was really excluding the folders that I added there, and I rarely reboot (or restart Obsidian) so it wasn’t until just know that I noticed the indexer taking too long.

Please make “Excluded Files” really exclude, or some way to achieve this!