Testing Obsidian's Limits: Performance Advice Needed for 10 Years of Instagram Data

Help
1

Hello everyone,

I am a non-engineer, trying to develop a system with the help of AI to migrate my 10 years of Instagram history (posts, reels, and stories) into a local Obsidian Vault.

Our goal is to build a “Second Brain” where all my past memories, hashtags, mentions, and locations are connected like neural pathways, rather than just keeping a simple data backup.

:loudspeaker: The Scale of the Project & The Background of My “Current SOS”

The scale of this project involves a massive volume of data that is impossible to handle manually:

  • Total Files: Several thousands of markdown notes (For standard posts [IGP], 1,842 posts and 7,487 media files have already been output).
  • Estimated Wiki-Links: Tens of thousands of auto-generated links.

To be honest, this project started with a light heart: “Hey, if I just dump my 10 years of SNS data into Obsidian, I can create a Second Brain instantly!”

But that was the beginning of a muddy struggle with AI. As I battled with the LLM, ideas kept inflating, and the AI started hallucinating and blowing things out of proportion. Before I knew it, I was buried under a mountain of massive overview notes that were completely unmanageable for a beginner. The system design completely blew up and went up in flames.

I managed to desperately clean things up and organize the repository into separate “Project Plans (01_)”, “Designs (02_)”, and “Specifications (03_)”.

As a non-engineer, I have set strict rules for myself, such as “no class structures or common libraries,” so that I can understand every single line of code and maintain it myself in the future. However, the system has grown so large that it is now beyond my ability to track alone. Because of this, the system is NOT yet complete, and my development has completely stalled. I am stuck right now.

(I have open-sourced this repository to seek the wisdom and technical help of developers in the community)
:backhand_index_pointing_right: GitHub Repository: hxxps://github.com/ikifuse/obsidian-ig-migration

:camera_with_flash: Live Demo of the Graph Generated from Actual Data

Since it is difficult to convey the overall image of the system in words alone, I have prepared actual recordings of the current implementation in action. To check the design, specifications, and code consistency, a part of the output has been test-imported into Obsidian.

(Please change “hxxps” to “https” to view the links below)

  • Detail view: hxxps://github.com/ikifuse/obsidian-ig-migration/raw/main/.github/readme-assets/graph-detail.jpg
  • Overview: hxxps://github.com/ikifuse/obsidian-ig-migration/raw/main/.github/readme-assets/graph-overview.jpg
  • Demo Animation of the Graph: hxxps://github.com/user-attachments/assets/40c4c8e6-9325-486e-a54e-ee70ffdaf953
  • Obsidian Card/DM Preview: hxxps://github.com/user-attachments/assets/d39bb8b7-1374-4fbb-a8e7-41d484d28963

:red_question_mark: This is Why I Need Your Help (My Questions)

Right now, I am at the stage of rewriting the specifications (v1.2) based on my separated design documents, and aligning the code with the existing output. However, to prevent further architectural collapse or wasting time on re-work, I desperately need to know the performance limits of Obsidian when mass-importing programmatically generated notes BEFORE I resume development.

Please lend me your wisdom and experience if you manage large-scale Vaults:

  1. Graph View rendering: Has anyone experienced Obsidian freezing or lagging terribly with 10,000+ notes and tens of thousands of active wiki-links?
  2. Search and Backlink indexing: Does background indexing slow down daily note-taking at this massive scale?
  3. Best Practices for Large-Scale Imports: If you have programmatically built a similar massive web-of-notes, how did you optimize the Vault performance? What kind of folder structure or design should I implement to keep Obsidian running smoothly?

I am completely stuck right now and desperately need your technical feedback, insights, or advice.

Since I only speak Japanese, I will be using AI to translate and read your replies, but I would truly appreciate any help you can share!

Thank you so much!

2

It depends on your hardware of course, but no, I wouldn’t expect it to. 10,000 notes is big, but I wouldn’t call it massive by Obsidian standards. With media files you’re up to ~20,000 files, which is for sure on the big side; tho media files are less taxing because they have less to index.

An older version of Obsidian was shown to handle 100,000 notes pretty well (once indexed) aside from the graph, which crashes. Interlude: Obsidian vs. 100,000 - by Alexander Rink

3

Thanks for the reply! It’s reassuring to know that Obsidian should run smoothly with 10k–20k files once they are fully indexed.

Also, I apologize about the GitHub link from last time—it was set to private, so you probably couldn’t access it. I have changed the repository settings to public, so you can check the folder structure and details now:

  • Our Project Repository: github.com/ikifuse/obsidian-ig-migration (Please copy and paste the text in the code block above into your browser, as I cannot post direct links yet due to forum restrictions.)

What we are actually most concerned about is the initial migration phase. Specifically, the behavior when we first copy thousands or tens of thousands of Markdown and media files generated by our tool into the Obsidian Vault, and Obsidian starts indexing everything for the first time.

Could you please give us your thoughts on the following points?

  1. Initial Indexing Load: When importing tens of thousands of files all at once, is there a high risk of Obsidian freezing, crashing, or becoming unresponsive for hours during the initial indexing process?
  2. Safe Migration Practices: What are the recommended practices to avoid issues during this initial import? (For example, should we copy files in smaller batches, or is it better to copy the files while Obsidian is closed and then launch the app?)

If you or anyone has a better idea or code suggestions for our migration tool, we would be extremely grateful if you could create a Pull Request or open an Issue directly on our GitHub repository!

If you have any best practices or advice for a smooth bulk import, we would love to hear them!

4

Just to add some details from our experiments:

  • When importing around 300 notes, the migration process runs safely and successfully.
  • However, once we exceed 700 notes, we start encountering errors.

Because of this, we currently use a workaround where we split the Instagram posts (IGP/IGS) into the first half and second half of each year (6-month batches) to keep each import size under 300 files.

Do you think this “700 notes threshold” is related to a limitation in Obsidian’s initial indexing process, or could it be caused by other environment factors (such as OS file limits)?

Also, is splitting the data into smaller batches like this manually the recommended best practice, or is there a better way to handle it?

Any insights on this would be highly appreciated!

5

I would get everything into a folder and then open the folder as a vault. Obsidian will take some time to index everything, but I don’t know how much. On mobile my vault of ~7,000 files occasionally gets reindexed for some reason, and it takes at most a few minutes.

6

How are you importing? Just saving to the vault folder? I wouldn’t expect crashes, but if you’re doing it with Obsidian open, you could try with it closed instead, and let it index them all afterward.

Another possibility is that rather than the number of files, there may be a specific file or field that Obsidian’s indexer is having trouble with. Unusually big and complex notes (especially ones containing lots of HTML) would be the first candidates to check.