Do you store large collections of pdf attachments in your vault?

Help
1

I have originally asked this question in this older thread. As it differs somehow from the original question and might be a question more people have been (or still are…) dealing with, I move this to a new topic.

Since I have started using obsidian as my main working environment, I have been dealing with the question of how to handle pdf literature attachments and where to store them. I would be interested in some shared experience of others: how do you handle your pdfs? Do you store them inside or outside your obsidian vault?

Currently, I still have a folder external to my vault under which I store all of my pdf attachments. I use Zotero in combination with zotfile, BetterBibtex and mdnotes for organizing my literature and - if required - extracting highlights and exporting them to my vault. The corresponding pdfs are linked in my vault so that I can open them in my external pdf viewer.

In any case, I will keep zotero as my main application for organizing references. Yet, I am still dealing with the question if I should move my attachments folder to my vault. This question became more important again as the new internal pdf viewer of obsidian now allows me to select text and copy and paste it directly to my active note - and there might be some possibilities that obsidian’s pdf viewer might even further develop in the future, allowing me for example to highlight text and extract my highlights directly into my note. Even if this feature won’t come, there is already the pdf extract highlights plugin I can use for that.

Moving my attachments folder to my obsidian vault would have several advantages:

  • pdfs are opened faster
  • Links to pdfs are inserted in a faster and smoother way
  • I don’t have to switch between different applications but can have my working note and my source file side by side
  • The backlink pane shows me all my notes that already include citations of a special source or are somehow related to it

Yet, to date, I am still hesitating. The main reason is:

My attachments folder contains several thousand of pdf files, summing up to an overall size of currently 10 GB - and it is continuously growing. I am afraid that this could massively slow down obsidian (to be honest, I still haven’t completely understood obsidian’s caching process… Does obsidian, for example, only cache file names of other file formats like pdf or also content?).

Have others already experimented with such a large attachment folder? Which solutions have you found?

A kind of compromise could be (maybe as a feature request) that obsidian would allow to “connect” to an external attachments folder, thus allowing to open external pdfs in its integrated viewer. Could this be a meaningful option others of you would like to see? Or doesn’t that make any sense at all?

The overall question came up again today when I moved a couple of pdf files (not my literature files, but some work sheets and information material) to a subfolder of my vault around 40 files maybe, including two videos. Since about half an hour now obsidian is caching (?) in the background and became virtually unusable (at least, reacting extremely slow) for me, continuously eating around 2 GB of RAM…

Summed up, some general questions I would like to discuss with others:

  • Where do you store your attachment files?
  • What pros and cons can you add to the ones already mentioned in my post?
  • What were your experiences with storing large collections inside your obsidian vault? Does it slow down obsidian or is the problem solved once the initial caching process has finished?
  • Would you like a feature I proposed above? Or doesn’t that make any sense to you?

Thanks for any advice/proposition/experience!

16 Likes
2

Hey! this is an interesting situation, it’s definitely tricky to figure out what the best solution is, I got a couple of ideas.

I do see the value in using and having access to all your .pdf’s from Obsidian.

I have 2 proposed solutions:

1. Creating a vault in the location of your pdf’s

Considering you have all your zotero pdf’s in one single folder, (or sub-folders inside one main folder) then you can create a vault right there, and create new .md notes in that folder.

Pros

I think this option is clean and easy.

Cons

It’s tricky to sync across multiple devices considering the size of the vault. The best choice if you need to sync across devices would be something like Dropbox’s paid plan. I currently use the free 2GB Dropbox version, which might not be enough.

Depending on how Obsidian Sync works with large storage vaults that could also be a great option. When first released Obsidian Sync had a limit of 4GB but I don’t see that limit in the sales page anymore, so I’m not sure of the current limit

Option 2 Linked Folders (Symlinks)

There’s a way to create a link between folders, this is called a symlink. I’m not sure how experienced you are with these stuff, so for the sake of clarity I’ll explain it in simple terms.

A symlink is a ‘symbolic link’ , it allows for a file to exist in one location

(such as a zotero folder with all pdf’s)

and to also to symbolically exist in another folder

(such as an obsidian attachments folder)

If you write on one file, the other one (the symlink) is also afected. If you write on the symlink, the original is also affected. That way you can have one same file (or folder) in 2 different places.

If you add something to the symlinked folder (zotero pdfs) the other one will also have that file (obsidian attachments)

This means that the symbolic link won’t take space in the obsidian attachments folder (it might take some space, here’s a forum on this since it’s tricky stuff I’m not fully aware of.)

With this method, you can keep using zotero, and create a symlink that allows you to open an specific .pdf file in obsidian without taking up much more storage.

Cons

If you move folders around, specially if your obsidian vault moves places, things might break. As long as you have your original pdf files you should be able to connect it back, just be willing to have to fix some issues.

You’ll need to do some research to see what’s the best way to do this with your computer.

The tool I use for symlinks (there might be easier alternatives)

Personally I use a terminal app in Linux and Mac called Ranger, there is a really easy way to create relative symlinks for individual files and also folders with some shortcuts. It’s the fasted way I know. but I’m sure there are easier ways to achieve this in Windows and Mac from the file explorer / finder, or with additional tools. I just haven’t tried it with those.

Ranger documentation you can search in that page for ‘symlinks’ again this is a bit more advanced, so I’m sure there are easier ways to do this.

Takeaway

Option 2 is more techy and advanced, so if it sounds overwhelming go for Option 1. However if you decide to go for Option 2 I think it could be a good solution, as long as you are willing to spend some time testing, learning, researching and enduring some frustration. After that it might be worth it.

More on this

I really like the ambitious use-case you present. I will start covering mote topics like this in the Obsidian content I create if you are interested.

I have videos on YouTube in my Obsidian YouTube Playlist and a growing Obsidian Online Course where I’ll be covering more advanced workflows in future updates.

Feel free to message me if you got any questions!

Hope that helps!

6 Likes
3

I’m in a similar situation – lots of pdfs, although I use Bibdesk to autofile and organize them. My library folder is on iCloud so syncs with whatever I have that set up on.

I’ve decided against moving everything into the Obsidian vault. I like my default pdf reader (Skim or Highlights), and it’s not a bother to use it side-by-side Obsidian. Even if the Obsidian pdf viewer develops, I prefer having good dedicated apps for different tasks rather than a mediocre one for all.
I insert a link to the relevant pdf for every literature note I make so that it’s easy to open.

This said, I can definitely see the value of having the pdfs “at hand” when working in Obsidian. I may try the symlink solution, that sounds interesting :).

1 Like
4

Thanks a lot for your interesting replies, @santi and @atiz

@santi: I am working on Linux and I have ranger installed - creating symlinks is a very interesting option I haven’t thought about so far! Just tried it out with a single file - and indeed it shows up in my vault and can be opened by obsidian’s internal pdf viewer.

Yet, I am still not sure how obsidian handles symlinks and also non-md files in general: how does it cache them? In the case of symlinks, does it only cache the file name and then just read it when I open the file in obsidian? Or does it follow the symbolic link?

In general, how does obsidian cache pdfs? Does it cache their content as well? If not, I wouldn’t need symlinks anyway, as in this case there would be no difference compared to the case where I store all my pdfs directly within my vault.

@atiz: yeah, there’s no big deal to have obsidian and my default pdf reader side by side - and this is how I always handled it so far. But there are still some advantages in viewing the pdfs inline like extracting, not having to switch between open windows - and especially the fact that I could see all backlinks that point to the pdf. This would easily allow me to see where else in my several thousand notes I may have already discussed some parts of this paper or book - possibly dating back several years ago…

Anyway, the symlink solution sounds really cool in case obsidian only caches the links and wouldn’t follow them in order to cache their content!?

And @santi, as to your first proposed solution, I am not quite sure: solution for what problem? If I understand you right, it is actually just the option to keep all pdfs in my vault?

1 Like
5

awesome to know you are on Linux too, with ranger it’s great to create symlinks easily.

I’m really not sure how the cache works in obsidian though, it’d be interesting to know if the symlinks alternative would work.

Regarding option 1 I don’t think I was clear enough, it could be the case where you make your pdf folder become the obsidian attachments folder, and you can build your obsidian vault as a parent directory to your pdfs.

As an option 3 there could be a mixture of option 1 and 2 where you symlink the other way around.

your obsidian .md files into your zotero pdf files, just an idea!

You could even just move your whole obsidian vault files as a parent to the location of your pdfs,
this would work, if the location of your original obsidian vault is not important (in my case it is, since I sync it with dropbox, so I would use symlinks)

Keep us updated on what ends up working for you!

6

Yes, these are actually the two possibilities I am thinking about (maybe I haven’t been clear enough in describing it… :wink:):

  1. Either having my pdf attachment folder as a subfolder to my vault - and thus within my vault;
  2. or having it outside and just linking to the pdfs.
1 Like
7

got you, that makes sense, I just realized my option 3 was your option 1 hahaha :wink:

I would go for the symlinks approach if I were you, specially considering how easy it is to do it in ranger.

but there might be things I might not be aware of, like you say the cache, and having the possibility to snyc acrross multiple devices are things to keep in mind.

Good luck with it!

8

Hi. A similar situation here. I have stored my pdf attachments in a subfolder of my vault so far. But I am going to move them outside of my vault. The reason is I would like to keep my workflow linear and separated, that is Obsidian only takes the very simple task of writing.

Yet I found the pdf rendering in .md is a good and convenient feature. So I do not want to give up this feature so easily. I am wondering how you store pdf files and link to them in Obsidian now. Looking forward to hearing from you.

2 Likes
9

How do you create link to pdf? Do you use iCloud link or something else?

10

@zzfjill I just copy the file path from the Finder (I’m on a Mac); this will open the pdf in my default reader, Skim. Alternatively, the pdf reader app Highlights has the convenient feature of creating links to any highlight within the pdf; so I use that too (it looks like highlights://Hoffmann2021m_2#page=54, which then will open p. 54 in the book).

2 Likes
11

@alltagsverstand I understand the issue you are dealing with. I have been using Devonthink to store my pdfs. I have multiple databases in Devonthink already created and continue to add to them each week. I don’t see the need to duplicate this in Obsidian. I can, and do, link relevant articles to Obsidian (using a Devonthink link) if I want to reference a particular article. I can also download highlights and annotations from individual pdfs and copy and paste them to Obsidian. This allows me to summarize content from individual articles in my reading notes.
I hope this helps. At least it provides an alternative.

2 Likes
12

@santi Thanks a ton for this super cool workflow using symlinks. Now, I have links to all my pdfs in Zotero library inside my Obsidian Vault. :pray:

13

Thank you for raising this question. I was in the same exact scenario until I reached your query here. found the solution via symlinks :+1:

14

Sorry to bump an old thread, but I’m wondering if anyone knows if it is possible to use this symlinks trick to work cross-OS while using database managers like devonthink and/or zotero. Here’s what I have in mind:

At home, I have a mac. At work a PC. I want to use devonthink on my mac to manage a large number of documents, and perhaps zotero on a PC for the same purpose. My plan would be to:

On a cloud drive (icloud, google, onedrive):

  • have a large repository folder called, say documents.
  • have somewhere separate my entire obsidian vault
    The cloud drive is synced to both the pc and the mac

on the mac:

  • have devonthink installed with the database in the mac (like they recommend) but indexing documents on the cloud drive
  • have obsidian installed, opening the vault on the cloud drive

on the pc

  • have zotero (devonthink doesn’t work for PC) installed, indexing the documents folder
  • have obsidian installed, also opening the vault on the cloud drive

Finally, on the cloud drive again:

  • using the mac, create symlinks inside the local (synced) attachments folder in the vault, pointing to the local synced version of the documents folder
  • using the pc, do the same, create symlinks in the local synced version of the attachments folder pointing to the documents folder.

At no point do I plan to use the mac and the PC at the same time.

Could this work?