# šŸ obsidiantools: Python package for doing advanced analytics of your vault **URL:** \ **Category:** Workflows & Templates\ **Created:** [September 13, 2021, 8:31pm UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122 "2021-09-13T20:31:02Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![markf](https://forum.obsidian.md/user_avatar/forum.obsidian.md/markf/32/23851_2.png) [@markf](https://forum.obsidian.md/u/markf)\ **Post date:** [September 13, 2021, 8:31pm UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/1 "2021-09-13T20:31:03Z")
# obsidiantools šŸ‘‹ Hi all, Iā€™ve released a Python package called **obsidiantools** , available now through [PyPI](https://pypi.org/project/obsidiantools/). obsidiantools is built upon common libraries in the Python data stack, like Pandas and NetworkX, to enable you to do advanced analytics of your vault and notes. Check out the Github repo: > **[GitHub - mfarragher/obsidiantools: Obsidian tools - a Python package for analysing an...](https://github.com/mfarragher/obsidiantools)** > > Obsidian tools - a Python package for analysing an Obsidian.md vault **I am not a software engineer who does JavaScript or TypeScript - if someone wants to collab on integrating this package into an Obsidian plugin Iā€™d be interested in exploring that!** ## šŸ¦¾ Getting started In your Python environment, run `pip install obsidiantools`. Check out the README for more detail on API usage. Itā€™s incredibly simple. ## šŸ•¶ Demo See the functionality through a virtual Binder environment here: [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/mfarragher/obsidiantools-demo/HEAD?filepath=obsidiantools%20in%2010%20minutes.ipynb) ## šŸ’” Key features This is how **`obsidiantools`** can complement your workflows for note-taking: - **Access a `networkx` graph of your vault** (`vault.graph`) - **Get summary stats about your notes, e.g. number of backlinks and wikilinks, in a Pandas dataframe** - Get the dataframe via `vault.get_note_metadata()` - **Retrieve detail about your notesā€™ links as built-in Python types** - The various types of links: - Wikilinks (incl. header links, links with alt text) - Backlinks - You can access all the links in one place, or you can load them for an individual note: - e.g. `vault.backlinks_index` for all backlinks in the vault - e.g. `vault.get_backlinks()` for the backlinks of an individual note - Check which notes are isolated (`vault.isolated_notes`) - Check which notes do not exist as files yet (`vault.nonexistent_notes`) ## Using obsidiantools in your note-taking workflow Through obsidiantools you can recreate your vaultā€™s graph via NetworkX and Iā€™ve shown a basic graph in the demo. **NetworkX graphs are no replacement for the Obsidian appā€™s graphing capabilities!** For a start, the app is interactive and NetworkX has limited customisation options. **Where NetworkX graphs really complement workflows are through:** - Having a graph structured in Pythonā€™s most popular network analysis graph. - Processing the graph into a summary Pandas dataframe. - Being able to use the advanced capabilities of the library, e.g. doing deep dive analysis on subgraphs of your vault, applying algorithms to your vault (such as PageRank). Especially in large vaults, these capabilities can enable you to narrow down which notes you want to focus on. ## šŸ–¼ Images A basic chart I made with Matplotlib to mirror the Obsidian vault graph (ā€˜nonexistentā€™ notes are greyed out): ![demo-vault-networkx-graph](https://forum.obsidian.md/uploads/default/original/3X/3/8/3815934192aa23d48addd4424314a193c71f3387.png) As you can see from the chart thereā€™s one note that is isolated: ![isolated-notes](https://forum.obsidian.md/uploads/default/original/3X/6/5/656d1b54da1c0f0085d5bdfdc529346b3ad54978.png) dataframe `df` sorted by number of backlinks: ![get-note-metadata-sorted](https://forum.obsidian.md/uploads/default/original/3X/d/8/d87746f855e0646ba3437e9a899916ff7be9b79e.png) PageRank values of notes (via NetworkX graph): ![notes-pagerank](https://forum.obsidian.md/uploads/default/original/3X/3/7/37ae9a67b28e275fb6abadd24bf98ca6d7f78e20.png) ## šŸ— Future development I will use the Github repo for the project development. I wonā€™t have much time for the rest of the year for development but welcome ideas, pull requests etc. I have tested the functionality on my own vaults of up to 100 notes but interested to see in how other peopleā€™s vaults fare with the package. The key things Iā€™d like to do in future are: - More functionality for Zettelkasten vault formats. - e.g. neat way to parse any timestamps from note filenames and integrate in the dataframe. - More metadata columns in dataframe: - e.g. created time, could be useful here. Not supported for Linux though (need to think about design, tests etc.). - metadata on the text e.g. word counts. - Bring the md file readers into the main API. - Iā€™m not sure how the parsing works for non-Latin text currently and whether more config is needed. If the parsing works well e.g. for Chinese then it would be great to bring it in. - Bring markdown link counts into the API. - Expand documentation for network analysis, e.g. code snippets on different centrality measures. Expand to include basic NLP analysis if supporting md files in the main API. ---
**Author:** ![tallguyjenks](https://forum.obsidian.md/user_avatar/forum.obsidian.md/tallguyjenks/32/42858_2.png) [@tallguyjenks](https://forum.obsidian.md/u/tallguyjenks)\ **Post date:** [September 14, 2021, 1:19am UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/2 "2021-09-14T01:19:35Z")
šŸ‘€          ---
**Author:** ![Tomodachi94](https://forum.obsidian.md/user_avatar/forum.obsidian.md/tomodachi94/32/18588_2.png) [@Tomodachi94](https://forum.obsidian.md/u/Tomodachi94)\ **Post date:** [September 14, 2021, 1:45am UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/3 "2021-09-14T01:45:24Z")
This looks interesting. Iā€™ll take a look when Iā€™ve got some time. ---
**Author:** ![markf](https://forum.obsidian.md/user_avatar/forum.obsidian.md/markf/32/23851_2.png) [@markf](https://forum.obsidian.md/u/markf)\ **Post date:** [September 20, 2021, 7:30pm UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/4 "2021-09-20T19:30:37Z")
Already working on next release v0.6 - see [Projects](https://github.com/mfarragher/obsidiantools/projects) page for the progress. Hope to release new version in next few weeks. šŸ™‚ New features are in the `dev` branch. The main addition coded up is support for front matter. What I find interesting about front matter is that people seem to be using it in so many ways (if used at all). I think the package has to keep fairly simple in functionality as a result, and more of the work over time will become about showing recipes that people can do using the ā€˜vault objectā€™ through libraries like Pandas, NetworkX, Matplotlib etc. ---
**Author:** ![markf](https://forum.obsidian.md/user_avatar/forum.obsidian.md/markf/32/23851_2.png) [@markf](https://forum.obsidian.md/u/markf)\ **Post date:** [October 19, 2021, 10:52pm UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/5 "2021-10-19T22:52:11Z")
šŸš€ Hi there, I have done a **new release v0.6** with these extra features: - Support for **front matter** - Support for **embedded files** - More columns in metadata dataframe - get\_md\_links in main API Iā€™ve also updated my [demo repo](https://github.com/mfarragher/obsidiantools-demo) on Github to showcase the new functionality. The more obvious next ideas for v0.7 will involve bringing the plaintext of notes into the API, e.g. ability to extract tags. Iā€™m currently studying and doing all my notes in an Obsidian vault, so that is also a cool personal test case for new features. I have been using embedded images a lot so that was the inspiration for that new functionality and getting this new release done. šŸ•¶ ---
**Author:** ![obsimion](https://forum.obsidian.md/user_avatar/forum.obsidian.md/obsimion/32/17170_2.png) [@obsimion](https://forum.obsidian.md/u/obsimion)\ **Post date:** [February 11, 2022, 11:05pm UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/6 "2022-02-11T23:05:25Z")
ā€¦awesome, thanks! ---
**Author:** ![markf](https://forum.obsidian.md/user_avatar/forum.obsidian.md/markf/32/23851_2.png) [@markf](https://forum.obsidian.md/u/markf)\ **Post date:** [August 7, 2022, 7:31pm UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/7 "2022-08-07T19:31:54Z")
šŸš€ Iā€™ve released **v0.8** today (now v0.8.1 to fix a bug relating to a new dependency). See the [v0.8 release notes](https://github.com/mfarragher/obsidiantools/releases/tag/0.8.0). Summary: - Neat functions to get text in different forms (one with formatting preserved, another with it stripped out) - Markdown extensions included to replicate Obsidian experience - LaTeX equations easily accessible I also didnā€™t add any info on v0.7 in this topic before, but that added in support for extracting tags from notes (and an earlier approach for extracting text). v0.8 enables NLP analysis to be done much easier. Iā€™ve done some prototyping on a concept that auto-generates markdown by analysing text data from a `vault`. Hopefully Iā€™ll have something to share on that later this month! ---
**Author:** ![louis030195](https://forum.obsidian.md/user_avatar/forum.obsidian.md/louis030195/32/16928_2.png) [@louis030195](https://forum.obsidian.md/u/louis030195)\ **Post date:** [August 16, 2022, 5:47pm UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/8 "2022-08-16T17:47:21Z")
Awesome! Iā€™ve used this lib to build an API for semantic search and fine-tune an embedding model on your vault for Obsidian > **[GitHub - louis030195/obsidian-search: Semantic search in Obsidian](https://github.com/louis030195/obsidian-search)** > > Semantic search in Obsidian. Contribute to louis030195/obsidian-search development by creating an account on GitHub. ---
**Author:** ![louis030195](https://forum.obsidian.md/user_avatar/forum.obsidian.md/louis030195/32/16928_2.png) [@louis030195](https://forum.obsidian.md/u/louis030195)\ **Post date:** [August 16, 2022, 7:06pm UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/9 "2022-08-16T19:06:37Z")
Also, Iā€™m trying to play a bit with graph neural network [https://pytorch-geometric.readthedocs.io/en/latest/modules/utils.html#torch\_geometric.utils.from\_networkx](https://pytorch-geometric.readthedocs.io/en/latest/modules/utils.html#torch_geometric.utils.from_networkx) It would be fun to do a node classifier for example ---
**Author:** ![markf](https://forum.obsidian.md/user_avatar/forum.obsidian.md/markf/32/23851_2.png) [@markf](https://forum.obsidian.md/u/markf)\ **Post date:** [August 16, 2022, 8:18pm UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/10 "2022-08-16T20:18:49Z")
@louis030195 itā€™s cool that youā€™re doing work with Transformers and Pytorch via the package! šŸ•¶ Iā€™ve not done much with them so itā€™ll be interesting to see what can be done there. I was thinking about ngrams for more sophisticated search initially but of course those packages can do something more advanced now. For my NLP work, Iā€™ve focused on topic modelling via one semi-supervised algorithm so farā€¦ so Iā€™ve looked at NLP problems from a different angle. Iā€™ve used that algorithm do make proofs-of-concept for these features: - Auto-generated MOCs (see šŸ”— [forum post](https://forum.obsidian.md/t/auto-generation-of-mocs-via-topic-modelling/41795)) - Auto-suggested wikilinks\* They are in this šŸ“ [obsidian-nlp-analytics repo](https://github.com/mfarragher/obsidian-nlp-analytics). \* Iā€™ve updated my repo now with this notebook. ---
**Author:** ![louis030195](https://forum.obsidian.md/user_avatar/forum.obsidian.md/louis030195/32/16928_2.png) [@louis030195](https://forum.obsidian.md/u/louis030195)\ **Post date:** [September 21, 2022, 4:08pm UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/11 "2022-09-21T16:08:30Z")
@markf Awesome, If I find the time Iā€™ll try your approaches I also used your library to fine-tune GPT3 on my vault: [Language model assistance - #5 by louis030195](https://forum.obsidian.md/t/language-model-assistance/27409/5) ---
**Author:** ![markf](https://forum.obsidian.md/user_avatar/forum.obsidian.md/markf/32/23851_2.png) [@markf](https://forum.obsidian.md/u/markf)\ **Post date:** [January 8, 2023, 2:42pm UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/12 "2023-01-08T14:42:56Z")
Hi everyone, Iā€™ve launched **[v0.10](https://github.com/mfarragher/obsidiantools/releases/tag/0.10.0)**, which is an exciting release as Iā€™ve added a lot of new features in the past month and Iā€™d say the capabilities of the `obsidiantools` graph capture the current Obsidian v1 app well. The setup is twice as fast on my largest vault compared to v0.8 of the package. All these files are supported in `obsidiantools`: - Notes (md files) - Media files (images, videos, etc.) - Canvas files ## Canvas files Support for canvas files should reflect the current state of the Obsidian app. For example, you can detail on the backlinks relating to canvas files. You can also recreate the layout of a canvas file in Python: ![canvas-file-graph](https://forum.obsidian.md/uploads/default/original/3X/9/2/921e5d174073e65aa14684c9ea8aa22f331a3a03.png) Iā€™ve added a notebook to my demo repo with info on the support for canvas files and show a comparison with the Obsidian appā€™s graph view. Backlinks for notes that come from canvas files arenā€™t supported, reflecting the current status of the Obsidian app. Though I can see a route to supporting that in future. ## Attachment files in graph There is the option now to include ā€˜attachmentā€™ files in the obsidiantools graph. This is an example of how my test vault is visualised via Pyvis: ![demo-vault_pyvis-graph-includes-attachments](https://forum.obsidian.md/uploads/default/original/3X/1/0/1042df0542528a81a68be7ed92ced8bce8ac76d0.png) Iā€™ve added a notebook to my demo repo with more recipes for graph visualisation and show a comparison with the Obsidian appā€™s graph view. ---
**Author:** ![grepinsight](https://forum.obsidian.md/letter_avatar_proxy/v4/letter/g/d26b3c/32.png) [@grepinsight](https://forum.obsidian.md/u/grepinsight)\ **Post date:** [February 28, 2023, 3:28am UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/13 "2023-02-28T03:28:39Z")
This is awesome. Thereā€™s so much potential in this. Thank you for creating the package! Iā€™m curious if there are any plans to include the ability to mutate Markdown files, such as bulk adding tags to multiple files? ---
**Author:** ![markf](https://forum.obsidian.md/user_avatar/forum.obsidian.md/markf/32/23851_2.png) [@markf](https://forum.obsidian.md/u/markf)\ **Post date:** [March 4, 2023, 7:49pm UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/14 "2023-03-04T19:49:31Z")
There are many great Obsidian plug-ins for editing & reworking notes, so I donā€™t plan to add any note-editing functionality in this package. This package is focused for now on making it easier to analyse the content in vaults. ---
**Author:** ![nealr](https://forum.obsidian.md/user_avatar/forum.obsidian.md/nealr/32/79522_2.png) [@nealr](https://forum.obsidian.md/u/nealr)\ **Post date:** [April 21, 2025, 12:47am UTC](https://forum.obsidian.md/t/obsidiantools-python-package-for-doing-advanced-analytics-of-your-vault/24122/15 "2025-04-21T00:47:52Z")
The last Github update on this was three years ago and the last Issue cleared was 1/2023. Is development still active here? Iā€™m interested in particular in the graph related capabilities.