BibNotes Formatter: New Plugin to Export and Format Annotations from Zotero into Obsidian

With the help of @SkepticMystic, I have put together a plugin called “BibNotes Formatter” which captures my workflow for getting pdf annotations from Zotero into Obsidian (GitHub - stefanopagliari/bibnotes). This plugin generates literaure notes from the source stored in your Zotero library, including both the metadata and the annotations that are stored within Zotero (extracted using the native PDF Reader or the Zotfile plugin). The settings of the plugin provide different tools to customize the format of the literature notes, as well as to perform different transformations to the text of the annotations.

The plugin combines some of the elements that are familiar to those using zotfiles/mdnotes to get your annotations from Zotero into Obsidian with a few significant differences:

  • The plugin reads a json file exported from Zotero (using the Better BibTex plugin) which is updated automatically. As a result, it is possible to automatically keep in synch the entire set of notes exported from Zotero using the “Update Library” command (see GitHub - stefanopagliari/bibnotes)
  • When updating an existing literature note, the plugin does not over-write the changes made to the note previously (e.g. reference blocks, comments added manually on Obsidian). This allows you to keep the literature notes on Obsidian updated when the data is changed in Zotero without the risk of losing the manual edits
  • I have added in the settings a few options to format the highlights. These includes basic formatting (e.g. bold, bullet points, italic, quotation marks), but also more complicated custom transformations (GitHub - stefanopagliari/bibnotes) such as turning the highlights into heading and tasks, merge an highlight with the previous one (to combine a paragraphs running across two pages), copying the comment before the related highlight, adding a word to the list of keywords. I would be happy to introduce additional transformations if there is demand.
  • The plugin works with notes extracted both from the Zotero native pdf reader and via Zotfile.

This is the very first version of the very first adventure in JS/plugin-building. If anybody wants to give it a try and give any feedback on bugs to fix or possible improvement that would be much appreciated.

Importing your Zotero Library into Obsidian

In order to import your references and notes from Zotero, you need to export your library as a “BetterBibTex JSON” format and save this file inside your vualt. To do to follow these steps:

  • install within Zotero the plugin “Better BibTex for Zotero”. For more information on how to install the plugin see the instructions on the website of the plugin.
  • in the main menu of Zotero go to File > Export Library (to export the entire library). It is also possible to to export a specific collection or group of references by selecting these, right-click, and then selecting “export collection” (in the case of a folder) or “export items” (in the case of a collection of references.
  • select the export format “BetterBibTex Json”.
  • select “Export Notes” if you would like to import into Obsidian the annotation.
  • (Optional) select “Keep updated” to automatically update the exported library once an entry is added/deleted/amended
  • save the BetterBibTex JSON file in a folder within your Obsidian Vault
  • in the plugin settings within Obsidian add the relative path within your vault of the library to be imported, as well as the relative path within your vault of the folder where you would like the literature notes to be stored.

Commands

The plugin introduces two commands into Obsidian:

  • Create/Update Literature Note: when you select this command you will be prompted to chose one of references from the library you have imported. If the reference has not been imported yet in the specified folder, a new note will be generated. If a note already exists, its content will be updated without over-writing the existing annotation (e.g. comments added manually from within Obsidian and block-references will not be over-written). The first option (“Entire Library”) can be selected to create/update all the notes from the imported library.

  • Update Library: when you select this command, the plugin will generate/update all the notes that have been modified from Zotero since the last time the same command was selected. If this is the first time that you select this command, then the plugin will create/update literature notes for all the entries in the imported bibliography.

Create Literature Notes

By default the plugin will export both the metadata and the notes stored in Zotero for the selected reference. Both can be deselected in the plugin settings. The main configurations related to the format of the notes are the following:

  • Export Path: in the plugin settings, add the relative folder within your Obsidian vault where the literature notes will be stored. In the field is left empty, the notes will be exported in the main folder.
  • Note Title: In the plugin setting you can specify the format of the note title. Possible values include:
    • {{citeKey}},
    • {{title}},
    • {{author}},
    • {{year}}
  • Template: It is possible to select among two existing templates (one presenting the metadata as a simple list and the other wrapping the information into boxes using the Admonition plugin) or to or provide a custom template (see below).
  • Fields: It is possible to include in your custom template all the fields found in the Better Bibtex json file, as well as additional ones created by the plugin. These include:
    • {{title}}
    • {{citekey}}
    • {{itemType}}
    • {{author}}
    • {{editor}}
    • {{translator}}
    • {{publisher}}
    • {{place}}
    • {{series}}
    • {{seriesNumber}}
    • {{publicationTitle}}
    • {{volume}}
    • {{issue}}
    • {{pages}}
    • {{year}}
    • {{dateAdded}}
    • {{dateModified}}
    • {{DOI}}
    • {{ISBN}}
    • {{ISSN}}
    • {{abstractNote}}
    • {{url}}
    • {{uri}}: link to the entry on the Zotero website
    • {{eprint}}
    • {{file}}: local path of the file attached to the entry
    • {{localLibrary}}: link to the entry on the Zotero app
    • {{keywords}}: tags associated with the entry
    • {{collections}}: collections/folders where the entry is located
    • {{collectionsParent}}: collections/folders where the entry is located, plus the parent folders to these
  • It is also possible to wrap the placeholders into [[ ]] in order to create notes or to preface them with :: in order to create Dataview fields.
  • Missing Fields: Fields that are present in the template but missing in the entry are deleted by default. This can be changed in the settings.

Basic Formatting

In the settings of the plugin, it is possible to select the formatting of the highlights and comments extracted from the text. These include:

  • Double Space
  • Italic
  • Bold
  • Quotation Marks
  • Highlight
  • Bullet Points
  • Blockquote
  • Custom text before or after all highlights
  • Custom text before or after all comments

Additional Highlight Formatting

It is possible to perform additional transformations to designated highlighted sentences. The transformations currently included in the plugin are:

  • Heading: Turn highlighted text into a heading (Level 1 to 6).

  • MergeAbove: Append highlight to the previous one (e.g. to merge paragraph across two pages).

  • Preprend Comment: Place the text of the comment at the beginning of the highlight (rather than at the end as by default).

  • Keyword: Add the highlighted text to the list of keywords listed under the ({{keywords}}) placeholder in the template.

  • Todo: Transform the highlight or comments into a task ("- [ ]").

  • Custom Text: Add custom text before or after a specific highlight

Keywords

Transformations can be triggered by adding a dedicated “keyword” at the beginning of the comment to the specific highlight. This can be a single character (e.g. #) or a single word (e.g. todo). When this character/word is found at the beginning of a comment, the text of the comment or the highlighted text will be reformatted. The keywords can be defined in the settings of the plugin

Highlight Colour

In addition to using dedicated keywords at the beginning of a comment, it is possible to apply specific styling or transformations to highlights based on the colour of the highlight. The plugin recognize the highlight colour extracted by:

  • Zotero native reader (yellow, red, green, blue, purple)
  • Zotfile plugin (black, white, gray, red,orange, yellow, green, cyan, blue, magenta). In order to export the highlight colour you will need to activate this function by going to the main menu of Zoter and selecting Preferences → Advanced → Config Editor. Search for “extensions.zotfile.pdfExtraction.colorAnnotations” and turn the value to “true”. It is also important that the value “extensions.zotfile.pdfExtraction.colorCategories” is restored to the default value.
27 Likes

Good morning @stpag

Wow, that’s a lot of work, this should have been fun and rewarding to code all this :slight_smile:

I think I won’t be alone to ask, in your opinion, what is the main difference between this plugin and the somewhat widely used citations plugin? At a first glance, your plugin seems to be more documented, but I fail to see the obvious differences. Is it the automatic update of annotations?

Thanks for sharing!

Hi @felixchenier,
Yes, the starting point (connecting to Zotero via bib/json file) is the same as the Citation plugin. and to be honest the citation plugin is much better written by people who know what they are doing.

The main reason why I wrote the plugin was to be able to format the annotations in a way that would fit my workflow (e.g. create todos while highlighting by changing the colour, extract keywords, create headers). As far as I know the citation plugin did not allow for this formatting/manipulation of the annotations.
It is possible to do some of these things using Zotfile and customising the settings for the different highlight colour.

5 Likes

Sorry if I am missing something, but where do we put your BibNotes plugin? In the Obsidian plugin folder?

Edit (the forum would not let me edit since it has been too long since my post): I just copied the folder into the plugins folder in my vault. Will it eventually be downloadable from the community plugins tab?

Copying main.is and manifest.json in a folder in the plugin folder should work.

I will submit it to the community plugin.

1 Like

That would be really great. Thanks @stpag.
In new zotero you can select an area and have an image annotations. I wonder does your plugin also export those images into obsidian?
thanks.

One more question about extracted comments and texts in the obsidian: do they have links back to the source file (or preferably their pdf pages) in the zotero?
Thanks again.

I have not looked yet into those two functions but I will. The links back to the pdf are not included at the moment but it is in my todo list.

3 Likes

I try to export the json file from Zotero into the Obsidian folder in iCloud, but Obsidian is greyed out in the downloads window. Anyone knows why? See screenshot.

I’m not sure. In the past I had a similar problem where I was not able to export file directly in the Obsidian folder, but you should be able to export the BetterBibTex Json file somewhere else and copy/paste it inside your vault. Keep in mind, however, that the plugin does not work on mobile.

But does BetterBibTex then register where the file is for auto-update?

Yes. If you select “keep updated” when you export the file the first time, you can then manage that by going on the Zotero main menu → Preferences → Better BibTex → Automatic Export

Thank you, I managed to copy/paste it into the vault, but unfortunately I can’t change the path in preferences → Better Bibtex → Automatic export

I guess I will have to do manual export

I am loving the plugin! I had 2 questions. All my extracted annotations appear in the same bold, italic, and highlighted color. In the settings, I noticed that the settings for colors all show {{highlight}}. How do I change this to carry over the color info from zotfile?

Also, the collections list shows the collection that the items are in and its parent collection. I have a lot of nested collections. Is there a way to make it show more than 1 level of parent collection?

very nice, this looks very promising. i think citation plugin also uses similar json export form BetteBibText, can i use the same file? i think the path for that file is in the settings but it is not located inside my vault

Yes, the Citation plugin accepts either a bib or a json file. I limited this to the latter format because it was considerably faster in parsing a large library.

Unfortunately I could only make it to work when the file was located inside the library.

1 Like

i figured out that part, altho i cant search into the list it offers and have to navigate into it manually. my main problem is that i cant get annotation out and get only metadata. i did the “Add Item Note from Annotations" thing and the note is there. what else i should do? should i export it too? I thought that the part the plugin should do. the annotations that stored in pdf file get imported tho

The plugin is currently able to export only the annotation that has already been exported from the PDF into Zotero, either by using the Zotero plugin Zotfile (right click on the pdf → manage attachment → export citation) or exported from the Zotero native reader.
In the future I would like to cut this intermediate step and integrate in the plugin the same library that zotfile uses to extract the annotation from the PDF, but I’m not there yet

3 Likes

looking forward to see that, the features mentioned in the readme are very exciting. also hope to get available on community plugins as well