Zotero Integration – Import Templates?

Wow, well done and thanks for sharing! I particularly like the list callout thing. Will try it out as soon as I can. So when you say nothing is jumping around, are you referring to the behaviour of ordinary callouts, when you have to click into the callout to be able to edit it but since that reveal’s hidden characters, things “jump around”. Would be great to get rid of that.

Just some quick questions:

This will treat all creators as authors, right? Is it intentional? If not, here is how I managed to at least distinguish between editors and authors:

{%- set editor_only = true -%}
{%- for creator in creators -%}
  {% if creator.creatorType != "editor" %}
    {%- set editor_only = false -%}
  {% endif %}
{%- endfor %}
{% if editor_only -%}
  Edited by {%- for creator in creators -%}
    {%- if creator.firstName and creator.lastName -%}
      [[{{creator.firstName}} {{creator.lastName}}]]
    {%- else -%}
      [[{{creator.name}}]]
    {%- endif -%}
    {%- if not loop.last %}, {% endif %}
  {%- endfor %}
{%- endif -%}
{% if not editor_only -%}
  By {% for creator in creators -%}
    {%- if creator.creatorType != "editor" %}
      {%- if creator.firstName and creator.lastName -%}
        [[{{creator.firstName}} {{creator.lastName}}]]
      {%- else -%}
        [[{{creator.name}}]]
      {%- endif -%}
      {%- if not loop.last %}, {% endif %}
     {%- endif -%}
     {%- endfor %}
{%- endif %}

When you say chronogical it sounds like “in the order the annotations were made”, but I assume you mean “in the sequential order they appear in the text”, right?

I was recently wondering why I ended up using the zotero integration plugin, given that Obsidian-Zotero was so powerful (should take notes on these things). Could this have been the reason? Stability?

What issues have you been experiencing?

My main issue with the zotero integration plugin is that it tends to be slow and sonetimes hangs, which is why I wanted to take a look at the bibnotes formatter plugin, which doesn’t query the Zotero database but uses a JSON file exported by Zotero (which also should work on mobile). Have you tried it?

2 Likes

I’m glad you like it, let me know how the List Callouts work out for you!

Exactly. No hidden markdown formatting appears when you go to edit a list item, which was also one of the reasons I wanted a list rather than callouts. Now that I’ve also gotten rid of the ==highlights==, the editing experience is very fluid.

Regarding the authors, I hadn’t really given it much thought, because most of my sources are journal articles where there only are authors. But I also have items like books, so I’m thankful for your snippet there and will test it out.

Right, that was an imprecision. I was thinking of changing it, but was unsure how. Sequential is exactly right.

Yes, lack of stability / reliability is a likely reason. I detailed why I was going back to ZI from OZ on Discord:

Based on my recent experiences, I would actually caution against using AidenLX’s Obsidian-Zotero plugin right now over MgMeyers’ Zotero Integration, because there are several substantial bugs in the latest version:

It’s just too unreliable and buggy, and unfortunately, the speed of the plugin can’t outweigh that. I hope to return to it soon when these bugs have been resolved, but until then, I will try Zotero Integration again.

All of the above still applies.

Yeah, I have used Bibnotes formatter quite a lot in the past. It has a lot of potential, and it was quite fast. But recently, it’s been unusable for me due to formatting bugs. I cannot get formatting of the various elements like comments and highlights correct due to these bugs. It’s not abandoned, but bug resolution has been prohibitively slow for me.

One of its main appeals to me was the complex transformation rules you could make, like converting certain comments to tasks. But, as my template demonstrates, that is also possible with ZI.

True. I compared ZI and OZ over in my thread with Obsidian-Zotero templates, mainly noting the slowness and also the annoying Zotero picker, which often opens in the background:

Overall, there are some reliability, templating and feature implementation issues when using custom templates with the Obsidian-Zotero plugin, compared to Zotero Integration, but the result is still pretty satisfactory. The main pro is the speed. In my experience, searching for and importing a literature note is probably 10x faster using Obsidian-Zotero compared to Zotero Integration. Search and import speed is a noted issue with Zotero Integration . However, Obsidian-Zotero does offer an API with the latest release which allows other plugins to access the Zotero database through it, so that could be a solution there. Obsidian-Zotero also overcomes the annoyance of the native Zotero citation picker, which most often appears in the background rather than displaying over Obsidian.

Note: This was written before I decided to go back to ZI for the time being, as the aforementioned bugs were later introduced in OZ. I actually find ZI to be much more tolerable now, speed-wise.

1 Like

It is “folded” by default, press the ´>´ and it will fold open and show the rest of the meta data. I generally don’t use it show hide it by default.

If you don’t like that, about line 95 is this line: ´> [!info]- Info - [**Zot etc´ the minus sign right after ´[!info]´ decides start folded, or start unfolded.

BTW, if there is other meta-data missing, that does exist in Zotero, let me know, I have now only hardcoded what I use, but adding it is no problem

Folded:

Unfolded:

2 Likes

@galachus I’ve updated my template based on your feedback. Now, the template matches the default character/color combos from List Callouts. That means everything should work as expected out of the box, with no customization required beyond installing List Callouts. The only exception is if you need magenta highlights, which require a custom List callout.

3 Likes

It’s funny how we evaluate things differently (or at least describe those evaluations differently) depending on the result that we’ve made up our mind about. :sweat_smile:

Thanks for pulling in all the useful quotes from your other posts. I avoid discord as much as I can for asynchronous communication (like discussions around software issues) - it’s just not made for that - so it’s valuable to have it replicated here.

The OZ issues that you mention seem to be rather straightforward to fix (from what I can tell), genuine bugs, not big problems to solve. But, of course, it requires time to do it and the developer has been less active on github recently. Unfortunately, I don’t have the knowledge to contribute fixes.

I have adopted the annotations part of your template for my use case (which I’ll explain below):

## Reading notes
{% macro calloutCharacter(color) -%}
{%- if color == "Orange" -%}
- ~ 
{%- elif  color == "Red" -%}
- & 
{%- elif color == "Purple" -%}
- @ 
{%- elif color == "Blue" -%}
- % 
{%- elif color == "Green" %}
- $ 
{%- elif color == "Gray" -%}
- § 
{%- elif color == "Magenta" -%}
- € 
{%- else -%}
- ! 
{%- endif -%} 
{%- endmacro -%}

{% persist "annotations" %}
{% set annotations = annotations | filterby("date", "dateafter", lastImportDate) -%}
{% if annotations.length > 0 %}
*Imported on {{importDate | format("YYYY-MM-DD HH:mm")}}*
{% for annotation in annotations -%}
{% set citationLink = '[(p. ' ~ annotation.pageLabel ~ ')](zotero://open-pdf/library/items/' ~ annotation.attachment.itemKey ~ '?page=' ~ annotation.pageLabel ~ '&annotation=' ~ annotation.id ~ ')' -%}
{% set annotatedText = annotation.annotatedText | default('') | nl2br %}{% if annotation.colorCategory == "Red" -%}{% set annotatedText = '[[' ~ annotatedText ~ ']]' -%}{% endif -%}
{% set annotationTags = '' -%}{% if annotation.tags.length > 0 -%}{%- for t in annotation.tags -%}{%- set annotationTags = annotationTags ~ '#[[' ~ t.tag | lower ~ ']] ' -%}{% endfor -%}{% endif -%}
{% if annotation.imageRelativePath %}

> [!cite]+ Image {{citationLink}}
> ![[{{annotation.imageRelativePath}}]]{% if annotationTags.length > 0 %}
> {{annotationTags}}{% endif %}{%- if (annotation.comment or []).indexOf("todo ") !== -1 %}
> - [ ] **{{annotation.comment | replace("todo ", "")}}**{% else %}
> **{{annotation.comment}}**{%- endif %}

{%- elif (annotation.comment or []).indexOf("todo ") !== -1 %}
- [ ] **{{annotation.comment | replace("todo ", "")}}**:{% if not annotation.annotatedText %} {{citationLink}}{% else %}
  {{calloutCharacter(annotation.colorCategory)}} {{annotatedText}}  {{citationLink}} {{annotationTags}}{% endif -%}
{% elif annotation.comment -%}
- **{{annotation.comment}}**:{% if not annotation.annotatedText %} {{citationLink}}{% else %}
  {{calloutCharacter(annotation.colorCategory)}} {{annotatedText}}  {{citationLink}} {{annotationTags}}{% endif -%}
{% elif annotation.annotatedText -%} {{calloutCharacter(annotation.colorCategory)}} {{annotatedText}}  {{citationLink}} {{annotationTags}}{% endif %}
{% endfor %}

{% endif %}
{% endpersist %}

What changes did I make?

  • I simplified the code a bit by using variables for repeated code
  • I moved the entire list-items into the macro. i.e. I’m using the macro not just to generate the symbol for each colour but also the dash before it. This was necessary because I use green highlights for headings (i.e. they represent the structure of the annotated text) and I wanted those to have an empty line before them so that the structure of the text is somewhat visualized. Caveat: The empty lines are removed in reading mode. Will need to figure out how to maintain them. I might resort to rendering them as headings.
  • Also the red highlights get special treatment because I use those for keywords in the text. These are rendered as internal links. Having the annotated text as a variable made it easier to put the square brackets around them.
  • Since I am using tags for structural/organizational things only (i.e. for concepts/content), I don’t want the Zotero tags to be rendered as tags but as internal links. To achieve that, I’m using the tags field instead of hashTags field.
  • I have switched to using colorCategory instead of color because I have a lot of pdfs that I annotated outside of Zotero and that are using slightly different shades of red and other colours and it looks like Zotero nevertheless puts them into the correct colour category (despite different hex codes). I have not tested this properly yet, so no guarantees.
  • Finally, I changed the macro from separate if-statements to one with multiple else-if conditions. The main purpose of this was to have a default option in case some undefined colour comes along.

One thing that is driving me nuts with Nunjucks templates is how difficult it is to control the unwanted whitespace they produce. I usually end up removing as many line breaks as possible, just to reduce the possible sources of error but the price for this is that the code becomes much more difficult to read. I can’t seem to wrap my head around how the {%-and -%}` affect the final output and using a lot of if-statements as we do doesn’t exactly make it easier to consider all possible scenarios. (So I expect there to be some whitespace-bugs in the above code. Feel free to correct them :wink: ) If there is any rule of thumb / simple algorithm for when remove whitespace and when not to, I’d be glad to learn about it.

Note that in my template snippet, the additional line before green annotations is produced by not removing the linebreak that is added after each annotation by default ({%- if color == "Green" %} instead of {%- if color == "Green" -%}.

Final thought: It strikes me that it may make more sense to not render comments as list items at all. Any reasons for keeping them as list-items?

1 Like

No problem!

Yeah, but that doesn’t cut it for my current deadlines. I am keeping a keen eye on any updates, though. So I haven’t turned my back on Obsidian-Zotero, but it is a bit of a pain each time you settle in with one system, then have to establish yourself again with another.

Great idea! I hadn’t really experimented with variables in Nunjucks yet, but that looks much cleaner, and more flexible.

Yeah, I’ve helped someone do that in their template before, and was wondering why you wouldn’t render them as headings, if that’s what green represents.

This is very interesting. I’ve been trying to come up with ways to make my highlight colors more useful and actionable, and this is a great candidate.

Could you give an example of your tag structure?

You are going to have issues with this, specifically purple being categorized as blue: Purple annotation is wrongly regarded as Blue · Issue #168 · mgmeyers/obsidian-zotero-integration · GitHub
To avoid this issue, you need to set the hex values explicitly to the colorCategories, like apfelstrudelig did here:

I am wondering why your macro works (it does), when you use color there, but not hex-values. I thought it would need to be {% macro calloutCharacter(colorCategory) -%} {%- if colorCategory == "Orange" -%}.

Yeah, I should do this to be safe, although I only use Zotero Reader at the moment.

Unfortunately no, except for intuition and trial and error.

That’s an interesting suggestion. Related to this, I noted (or perceived there to be) an indentation issue in your template, where commented-upon highlights are only indented by two spaces, rather than the four equal to a tab. Is that intentional? For me, having highlights fully indented under the comment bullet makes it easy to read.

But something like this doesn’t look too shabby, so I’m open to suggestions, if you make it work for you without bullets (just a test). Either way, I would still convert comments to tasks on occasion, but that can be achieved regardless.
image

I am spending way too much time fiddling with this template… Need to leave alone for a while now. I’m no longer sure I prefer the list callout rendering of annotations to the callout version. It at least needs some tweaking.

I noticed that your list items are rendered with a lot more whitespace than mine:

Do you have a specific CSS to achieve this in literature notes? Because when I adjust the List Item Spacing to get a similar amount of whitespace, I that is way too much for ordinary lists in other notes. (I use the minimal theme.)

Two reasons why I’m hesitating to render them as headings:

  1. Green highlights are for headings at any level but if I render them as headings in obsidian, they will be headings at the same specific level. It’s not a big deal, but using the logic of hierarchical headings in markdown even though I’m flattening the original headings hierarchy, is a mismatch to me.
  2. The headings will be rendered in the headings colour of the active theme (which, in my case, is not green, currently). Not a big deal either, but I appreciate that the colours stay the same across contexts (i.e. Zotero and Obsidian), so using headings would break that.

I don’t have a tag structure. My use of tags is work in progress, but a decision I made early on was to use internal links instead of tags for links at the content level. For example, instead of tagging the notes for cat, dog and cow with #mammal, I add [[mammal]] to each note.

Instead I use tags as a kind of meta-language to organize my notes. For example, when I find myself coming to an important insight, I tag it #insight. Then I use the supercharged links plugin to add :bulb: icon to any note that has that tag in the file browser. I also have #diary tag used in a similar way. I also have a #person for rendering links to persons with a person icon, but that tag became redundant when I moved all person notes to a dedicated folder.

So, I yet have to come up with a coherent use for tags. For example, I might use tags for projects and subprojects, but the Projects plugin uses folders for that, so I’ll probably use folders too.

This is off-topic, so I’ll leave it at that.

BTW, one thing I forgot to mention in my last post is this:

It solved the problem of the Zotero picker opening in the background. Slowness is still an issue, but it’s a huge improvement.

My understanding is that you can call the attribute for the macro whatever you like, e.g. macro calloutCharacter(rainbow), you just have to use that term inside the macro to process the value that was passed on to the macro, i.e. {%- if rainbow == "Orange" -%} etc.

Thanks for pointing that out. I was going to wait for this to be fixed by the Zotero devs, but then I modified my template anyway. My solution is to just pass both values to the macro, like this: {{calloutCharacter(annotation.color,annotation.colorCategory)}}, so the macro now looks like this:

{% macro calloutCharacter(color,colorCat) -%}
{%- if color == "#f19837" or colorCat == "Orange" -%}
- ~ 
{%- elif color == "#ff6666" or colorCat == "Red" -%}
- & 
{%- elif color == "#a28ae5" or colorCat == "Purple" -%}
- @ 
{%- elif color == "#2ea8e5" or colorCat == "Blue" -%}
- % 
{%- elif color == "#5fb236" or colorCat == "Green" %}
- $ 
{%- elif color == "#aaaaaa" or colorCat == "Gray" -%}
- § 
{%- elif color == "#e56eee" or colorCat == "Magenta" -%}
- € 
{%- else -%}
- ! 
{%- endif -%} 
{%- endmacro -%}

The difference between two and four spaces seems to be minimal (at least in the minimal theme, :rofl:) but I didn’t realize that the vertical line only shows up when you use four spaces, so I changed it to four spaces. The downside of that, however, is that if the preceding comment is not a list item, then the dash of the indented highlight will not render as a bullet point. Can’t have it all…

What is the markdown for the bottom one? Is the highlight just a quote? (using >), since there is no bullet point.

I have tried to get rid of the bullet point for all list callouts by editing the main.js of the plugin. I managed to get it to work but in preview mode, but the logic for hiding the symbol and replacing it with the icon is totally different in reading mode, so I couldn’t get it to work there. I think it would make sense is the plugin replaced not just the respective character but also the list-item dash.

The idea of adding block-IDs upon import strikes me as ingenious :star_struck:, because this means that we no longer need to preserve previously imported annotations just to make sure that we don’t break block-references. It’s not totally fail-proof, though, because

but in my case, I think, such edits are very rare.

A bigger caveat seems to be that block references are not the only reason we might want to preserve previously imported annotations. For me, the main reason is probably that I sometimes add internal links to the annotation text, and those would obviously be overwritten. Sigh. So I guess, I’ll stick with the current procedure.

I have also tried rendering meta-data for each annotation in smaller font or hiding it, but the problem is that once you introduce HTML (like <small>), markdown inside these blocks are not rendered in preview mode. Also, the HTML tags re-introduce the “jumping around” phenomenon that the list callouts are supposed to eliminate. :roll_eyes:

1 Like

No, I don’t have any special CSS going on. I think there are three factors, which may be at play:

  1. Each highlight has a preceding comment in that screenshot. That makes it more separated.
  2. I think the theme makes a big difference. When I test out my template with different themes, the spacing changes a lot. Minimal and AnuPpuccin seem particularly condensed.
  3. Live-preview has less spacing than Reading mode.

Here’s a comparison between Polka, AnuPpuccin and Minimal, in that order:



You can isolate the CSS change to your Zotero imports by using a cssclass and putting it into the yaml of your Literature note template: cssclass: literature-note (for example).

And in the css file, the css should start with .literature-note .theactualcss {

Could you share your css to add spacing between list items?

Yeah, same here, but thanks for sharing. I see many of the same dilemmas myself.

OMG! I’ve been trying to solve this forever, so thanks a lot for informing me. I was so excited that I rushed to share it in the #academia Discord channel as well, crediting you (forum user tophee) and the github user.

Oh, well, I’m learning a lot here. So it’s just a parameter :slight_smile:

Seems like a nice fix. I assume it matches the first option first, so there won’t be conflicts?

Well, if you decide not to have comments as bullets, then I guess it’s not necessary to indent the highlight anymore?

Yeah, it was just a quick test. Since you’re already modifying the plugin, maybe you could trigger it with > instead :wink:

I think it is not recommended to use two spaces for indentation: Markdown indentation | New Relic Documentation

You could request this as an option on Github.

As far as I can see I have figured out how to have persistent annotations. In the latest version you can import from zotero, edit the annotations, and they will not get overwritten (unless you edit them in zotero, obviously)

Caveat, the template must make the initial checkout from zotero, so you must remove / rename the previous checkout (once).

zotero template gist

1 Like

Sorry for not understanding, but how is this different from basic persistent annotations?
Can you append individual new annotations using your template?

Every annotation is separate, the order is correct, it removes annotations that get removed on zotero
You can add notes to individual annotations.
It is still a work in progress, but seems to work

Okay, sounds quite interesting. But what I don’t understand is how this can work, if it requires that you rename or remove your previous import. How can you persist your notes that way? Or does this just apply if the note was made with another template before?

Does this mean that every annotation has persist fields surrounding it?

I guess the main advantage of your approach is the ability to insert new annotations under the right heading, without duplicating the heading?

My confusion was in your macro (and in my head), not Line Callouts. Your symbols were arbitrary given the headings which threw me (e.g., @ for questions instead of ?, #ff6666 matched to £ instead of !, etc). Which, given what I say below about DV, it’s probably a better idea not to use common symbols.

But, any streamlining of Zotero/Obsidian process is appreciated given the formatting decisions that have to be made in addition to all the fiddling.

Oh, the Line Callout plugin’s icon picker is tiresome, isn’t it?

As @Feralflora suggested for spacing between list items in literature notes, you can put cssclass: literature-note in the YAML of the template. Then add this CSS snippet:

.literature-note {
  --list-spacing: 0.3em; /* default 0.075em; */
}

or you can add the list item spacing across your entire vault with:

:root body {
  --list-spacing: 0.3em; /* default 0.075em; */
}

I use the Minimal theme and these both work for me.

By the way, something to consider if you’re thinking about not using list items and using callouts instead is Dataview usage. In fact, list items, tasks, and headings are the reason why I found @Feralflora’s template so useful.

Using DV, you can:

  • show all list items under a specific heading across research papers (e.g., to examine methods across papers)
  • show all list items with a certain word (e.g., to see all annotations with the same reference)
  • show all list items with a certain tag? Yep.
  • with the line callouts plugin symbols - show all list items of type with a certain word.

All of this without specifying any inline metadata.

Example - show all method-type list items with a phrase:

LIST L.text
FROM "in/papers"
FLATTEN file.lists AS L
WHERE icontains(L.text, "@") and icontains(L.text, "hierarchical cluster analysis")

With all annotations and notes as normal text, you can’t do any of this so easily.

5 Likes

Sorry, bad explanation, you need to remove the old page once, then the template takes over, it only updates annotations, but annotations made with a previous template will not be persistent.

Yes, it every annotation has persistent fields. Combined with static block-ids, it’s possible to really use the annotations in other notes, which is the goal of the template

If I re-import the notes from zotero to obsidian, the previously imported notes will lose the reference (block id) and all other notes created in obsidian will also lose the reference to the notes imported from zotero. Is this solved with your template?
My purpose is not to lose the references to the notes that I import from zotero (highlights and notes) even when I import them again.
I have tried your template and it doesn’t work for that purpose. I know absolutely nothing about coding. However, I have an idea to fix this. I have noticed that when one references a note or a note title, the format ([[name#title]]) is not altered unlike referencing a block ([[name^12345 ]]). So what I would like to do is create a title for each highlight and note that imports from zotero and the title is identified by the first 10 (or any number) letters of the highlight in case it was a text highlight or the first 10 letters of the note of the highlighted image. Obviously, the coding for this must allow generating the title with that pattern automatically. In this way, each note imported from zotero will have an identification that will be preserved even if you delete or import them again, since the references to those notes will be maintained ([[name#abcde]])
image

image
I was wondering if you could help me with the coding of the template the way I’m setting it up.
I don’t speak english so I’m using google translate

i got this. This way I can keep references to zotero highlights being imported into obsidian no matter they are removed and updated:
{% for entry in annotations -%}
{%- if entry.annotatedText -%} # {{entry.annotatedText | truncate(15, true,“”)}}
{{entry.annotatedText}}
{%- if entry.id %} ^{{entry.id | replace(entry.id, entry.annotatedText | replace(" “,”“) | truncate(15, true,”"))}}
{% endif %}
{%- endif %}
{% endfor -%}

It’s not clear who you were replying to, since you didn’t make the comment as a reply to anyone in particular. I assume you were testing @Qwxlea’s template?

Yeah, i was testing @Qwxlea template.

Hey there Frealflora

Thank you so much for a wonderful template, which I like to use. Unfortunately I don’t have much programming expirience and there is one question, that bothers me though, and that I can’t get around with: How do you edit the template, so you can set the order of colors? For example: I would like to have my blue higlights coming first, then my yellow ones etc.etc…

I tried to set that order in the begining, but it doesn’t really work that way :see_no_evil:

Hey @bilderer,
Thanks for the feedback, I am glad you are liking my template! As is, you can’t change the order of the colors.
They will be in order of appearance. As I’ve written under “Planned enhancements”, I am going to incorporate this at some point, but it requires some modification:

Btw, if you tag me with an @FeralFlora, I will get notified, so you might get a quicker response (although this time, I happened to see it quickly).

1 Like