Exporting with citations

What I’m trying to do

I’ve written an obsidian note with {@author_2020} which is rendered correctly within Obsidian as (Author, 2020) by some combination of Pandoc Plugin, Pandoc Reference List, and Zotero Integration.

When I use Pandoc Plugin to turn this into a .docx (or anything else) it exports as “{@author_2020}”.

When I try inserting lines into “Extra Pandoc Arguments” to point it at a .csl file it crashes and burns and won’t create a .docx at all.

Things I have tried

Endless variants on:

–citeproc
–csl “C:\Users\name\Zotero\styles\apa.csl”

and
–citeproc
–csl “C:\Users\path\VaultName.pandoc\apa.csl”

and
–citeproc
–csl .pandoc\apa.csl

(these are the two places that a search of my computer found a file with that name)

The only difference in results is whether I get an error message or not

I’ve even tried replacing
– csl
with
–metadata csl=-

I am pulling my hair out. Surely citing stuff is the #1 thing that academics want to do with Obsidian and Zotero, so how come none of the tutorials are explaining how to do it?

Please help

Further note: this post lists this as the answer:

Here is the general command you can use to convert from Obsidian .md.docx using APA 7 referencing. Change the file paths to your needs, but keep the git URL to use APA 7.

pandoc "path/to/input.md" --citeproc --csl "https://raw.githubusercontent.com/citation-style-language/styles/master/apa.csl" --bibliography "path/to/citations.bib" -o "path/to/output.docx"

So I set “Extra Pandoc Arguments” to this:

–citeproc --csl “https://raw.githubusercontent.com/citation-style-language/styles/master/apa.csl

The result:
image

First of all, to save yourself some trouble, ditch the Pandoc plugin and use the Enhancing export plugin instead. The Pandoc plugin has several unresolved bugs, and it isn’t being updated, and hasn’t been for years.

Since both interface with Pandoc, you’ll be providing the same arguments as extra arguments in both cases. The difference is that Enhancing export has settings for each output format, so make sure you add the extra arguments to the PDF and Word template settings, and not something else.


Now, on to your concrete issue. While you weren’t very specifc about what the issue is (“crashing and burning”), there’s several issues from what I can tell.

  1. The citation syntax is wrong. Yes, despite Pandoc Reference List being able to render the citation (it can ignore the surrounding brackets).

Usually, you don’t want to use curly brackets like { for your citations. These are only necessary for unconventional citekeys that contain otherwise invalid characters.

Depending on whether you use narrative citations or parenthetical citations, you’ll want to use either no brackets or square brackets, respectively. If you do need curly brackets, the @ should go outside the brackets like so: @{author_2020}.

As is, I suspect that Pandoc would consider the citation a narrative citation (without brackets), and leave the curly brackets in place around the narrative citation.

See Pandoc’s citation syntax.

  1. It seems you haven’t provided a bibliography using --bibliography.

The part you took from the example you linked to is only part of what you need. As is, --citeproc is missing the bibliographic information needed to format the citekeys into proper citations. See the linked docs on how to use the --bibliography argument.

  1. Beyond the missing bibliography, I suspect that you may have run into the Pandoc plugin bug that makes it unable to handle spaces in paths.

If any of those paths you mention had spaces in them, then that would throw an error. There are some workarounds for this, but the easiest solution is to ditch the Pandoc plugin in favor of Enhancing export.


Lastly, regarding your frustration with the available guides:

While there are indeed some omissions in some of the available guides, a) you have to make sure you actually follow the guide before criticizing it (in reference to how you only copied part of the linked example), and b) sometimes is better to just reference the original docs, rather than some derived guide.

Specifically, see the section on Citations in the Pandoc user guide. It mentions all the arguments you need, and explains how to use them:

https://pandoc.org/MANUAL.html#citations

3 Likes

Many thanks for your super helpful reply! Sorry I’ve been slow to get back, life has very much intervened :frowning:

Unfortunately, I’m still struggling.

I replaced the pandoc plugin with your suggested Enhanced Export Plugin, which does seem easier to configure, and successfully produces Word docs. So that is a win.

I still can’t work out how to produce Word docs with formatted citations though.

Within Obsidian the Pandoc Reference List plugin renders the following:

  • [@Jones_2024] renders as (Jones, 2024).
  • @Jones_2024 renders as Jones (2024)
  • @[Jones_2024] renders as @[Jones_2024] - not formatted

When I export these to a doc they appear as non-formatted plain text

Clearly, the export plugin needs further configuration to do formatting. Its documentation does link to a walkthrough, but it is behind a paywall at medium.com, so not accessible.

Fortunately, the pandoc manual page you list does suggest what some of this might be. Unfortunately, it’s written at a technical level of detail I’m not equipped to follow. I’m a beginner to all of this; command line arguments and YAML are foreign languages to me. I’m sure they are learnable, but so is French. Nevertheless, I’m sure we can work this out.

The documentation page it links is probably helpful, but assumes a level of technical knowledge I do not have.

It says to trigger citations with:

pandoc --citeproc myinput.txt

I’m trying to use the menu option (“Enhancing Export: Export to…”), so when you invoke that command it already knows the input and output files, and you don’t need to type the “pandoc” command. There is a place in the configuration menu for that plugin for “Extra arguments” and I put --citeproc in there, but that doesn’t seem to do anything on its own.

This is probably because it doesn’t know what format to use. The instructions on the pandoc manual page do further specify:

To use this feature, you will need to have

  • a document containing citations (see Citation syntax);
  • a source of bibliographic data: either an external bibliography file or a list of references in the document’s YAML metadata;
  • optionally, a CSL citation style.

My document has citations in it so I have the first bullet point. I don’t know how I would link an external document, or what external document I should link if so. I understand YAML to be some form of code that you insert at the start of a file, but I don’t know how to write it, and I don’t want to start every single document by pasting in lines of code - that seems repetitively labour-intensive.

The page further says

You can specify an external bibliography using the bibliography metadata field in a YAML metadata section or the --bibliography command line argument. If you want to use multiple bibliography files, you can supply multiple --bibliography arguments or set bibliography metadata field to YAML array.

I tried adding --bibliography to the Extra Arguments filed, but that doesn’t seem to do anything. So again, there clearly has to be some kind of link to some sort of file that is somewhere on my computer, but it isn’t immediately clear what that would be.

I do apologize if I expressed frustration above. I suppose I was expecting this to be a function that worked more out-of-the-box, and not something that requires this much technical knowledge to wire together. Googling how to do it had been throwing out tutorials that made it look easy, but were apparently outdated (i.e., directing to the pandoc plugin).

If the citekeys remain as citekeys, then you haven’t implemented my suggestions. So, although you switched plugin, you’re still facing the same issue of a missing bibliography. --citeproc can’t do anything without a bibliography.

These plugins are both just scaffolding for Pandoc. What you need to do doesn’t change (much) because of the scaffolding, because it’s still Pandoc behind the scenes. Of course, it might vary how you specify the arguments.

The “Extra arguments” field is just a convenient field the that both Pandoc plugins provide where you can add additional arguments into the main command used to run Pandoc. You could in fact also place these arguments there in the field above Extra arguments.

Exactly, it doesn’t do anything on its own. How could it know what @Jones_2024 is referring to without the associated metadata?

This is indeed the sticking point. The type of file you need is explained here: Pandoc User Guide → Citations → Specifying Bibliographic Data.

Basically, you specify it like so:
--bibliography="C:/path/to/your/bibliography.bib"

Where bibliography.bib can a bibtex (.bib), biblatex (.bib), CSL JSON (.json), CSL YAML (.yml), or a RIS (.ris) bibliography file exported from a reference manager like Zotero.

After this, you’ll have requirements 1. (citeproc) and 2. (bibliography) down. Then you just need to specify the citation style using --csl. Here, you provide a file path or URL to a .csl file, the same way you do for --bibliography.


Live citations in Word

Since you seem focused on Word exports, you should be aware that using --citeproc might not be the best option in this case, because it will give you static citations that you can’t dynamically edit in Word (change citation style, update bibliography list). For other exports like PDF, stick with --citeproc and its associated arguments.

But if you want editable citations in Word, like the ones you get when using the Zotero addon for Word, there is another option.

This option is to use a filter made be retorquere (the Better Bibtex creator). Find the relevant guide here:

  1. Download the zotero.lua filter from the link above.
  2. Find your Pandoc folder using the pandoc --version command in a OS’s terminal / command prompt.
    • In the Pandoc folder, create a “filters” subfolder, and put the downloaded zotero.lua file into this folder.
  3. In the extra arguments of your Word export template in Enhancing Export’s settings, put --lua-filter zotero.lua.
    • Make sure to remove --citeproc, --bibliography, --csl. The zotero.lua will handle all this instead.
  4. If necessary, specify arguments for Zotero.lua.
    • This can be done in the note’s properties, like so:
---
# all the regular stuff you have here
zotero:
  library: <group name> # omitted to use your personal library
  scannable-cite: false # only relevant when you're compiling to scannable-cite .odt
  client: <zotero or jurism> # defaults to zotero
  author-in-text: false # when true, enabled fake author-name-only cites by replacing it with the text of the last names of the authors
  csl-style: apa # pre-fill the style
  sorted: true # sort clustered citations by author.
---

or using arguments placed in Extra arguments like so:
--metadata=zotero_csl-style: apa

  1. Export using Enhancing Export: Export to Word.
4 Likes

Just to clarify the above, adding something like --bibliography="${vaultDir}/Metadata/Zotero/library.bib" --citeproc --csl="${vaultDir}/Metadata/ieee.csl" to the “Extra arguments” field of the Enhancing Export plugin settings and then going through the export steps for that plugin on the same template type should result in formatted inline citations?

I’ve tried this a few times now, changing around absolute and relative paths. All build successfully but they do not produce the expected rendering. I get the same @something strings in the final document and no references list.

Enhancing Export: v1.10.3
Pandoc: v3.1.11.1
Obsidian App: v1.5.3

Edit: I did also try reloading Obsidian after adding the plugin. This cleared out any changes I made in the “Extra arguments” field. I have also tried putting the above arguments directly into the “Arguments” field with no change in behavior.

Yes, that should work.

Sorry for not getting back to you sooner. Is it still not working for you? Which format are you exporting to in this scenario, where it’s not working? Do you get any errors in the console? Can you share a screenshot or copy paste of your arguments?

That’s a real bummer, and perhaps why it’s not working? I experienced a similar issue before, but that was fixed. Perhaps the same thing is happening again? You should report it on the Github repo :slight_smile:

Sorry, I’m afraid I’m still not getting it to work :frowning:

I have tried the following:

Changing extra arguments to read (without line breaks):

--citeproc 
--csl="https://raw.githubusercontent.com/citation-style-language/styles/master/apa.csl" 
--bibliography="C:\Users\name\OneDrive - institution\path\My Zotero Library.bib" 

The resulting Word file is still not formatted (it still reads [@name_year]).

You are correct that I’m fixated on Word, as it is the lingua franca for much of academia (for worse or worse). I therefore tried following your instructions for the lua filter.

My powershell says the version is 3.1.11.1, and the user data directory is:
C:\Users\name\AppData\Roaming\pandoc

File explorer did not find a ‘pandoc’ directory within the ‘Roaming’ folder, so I created one with that name. Windows did not complain about this duplicating any existing directory names, so I assume (incorrectly?) that I am not missing a folder that is simply hidden.

I then created a \filters folder, and within this a file called pandoc.lua into which I pasted the text from the download that you so helpfully linked.

I changed the Extra Argumnets to read --lua-filter zotero.lua, deleting all the citeproc elements listed above.

When I export the document, there is still no formatting. I added the following to the start of the document:

---
zotero: 
	csl-style: "apa"
	sorted: "true"
---

This also seems to have no effect.

I’m afraid I’m feeling a little defeated. Am I missing something obvious? Am I being a numpty?

For reference, pandoc itself is working: citations are rendered correctly within Obsidian.

Do you get any errors in the console? I would expect there to be some error(s), but you haven’t made it clear whether you’ve checked the console yet. So do check the console (instructions here), and try using double backslashes in the path \\, or forward slashes.

There was a mistake made in the process, namely naming the filter pandoc.lua instead of zotero.lua:

This won’t work, as you are calling a filter named zotero.lua, which doesn’t exist:

So the file names of the filter don’t match as is. You should rename pandoc.lua to zotero.lua. Or alternatively, use the argument --lua-filter pandoc.lua, so long as the filenames match! (I would go with option number one, though).


You were right to create the user data directory yourself. For some reason, it’s not always present by default.

There was something you missed, so cheer up, there is hope yet :wink:

It’s not Pandoc doing that, though. Despite the name, Pandoc Reference List only uses Pandoc to convert bibliography files from .bib to .json, as required. I believe the name refers to the fact that it picks up on Pandoc citations and creates a reference list for those. I believe that the citation parsing is a custom implementation made by mgmeyers, and the citation rendering is citeproc.js, not Pandoc’s citeproc.

Ok, well that’s encouraging.

I switched the name of the file to pandoc.lua and updated that to be the same in Obsidian, but that still is not working.

Thanks for pointing me towards the console.

When I first load the document there is a red error which reads:

TypeError: Cannot read properties of undefined (reading 'editor')
    at t.get (app.js:1:2091356)
    at getEditorFromState (plugin:obsidian-outliner:88:13)
    at VerticalLinesPluginValue.waitForEditor (plugin:obsidian-outliner:2552:28)
    at new VerticalLinesPluginValue (plugin:obsidian-outliner:2602:14)
    at e.eval [as create] (plugin:obsidian-outliner:2795:82)
    at e.update (app.js:1:367539)
    at e.setState (app.js:1:468006)
    at t.clear (app.js:1:1378062)
    at t.clear (app.js:1:2086260)
    at t [as constructor] (app.js:1:1374961)

I don’t know really what that means.

When I try exporting to docx it throws a yellow warning:

plugin:obsidian-tasks-plugin:149 [2024-02-29-14:09:09.738]
[warn][tasks.Cache]
indexFile: WARNING: indexing non-markdown file: folder_path/file_name.docx 

Again, not quite sure how to interpret this.

I guess I really am a numpty!

If you could be more specific than “still is not working” that would be helpful. So you changed the filename (why, wasn’t it already pandoc.lua? Previously, you said: “I then created a \filters folder, and within this a file called pandoc.lua”) and you changed the filter to be --lua-filter pandoc.lua (correct?) - and then what happened? Did Enhancing export give a pop-up notification like this:

image

Or did it pop up an error message? Or did none of those things happen?

As for the console error and warning you mention, the first one doesn’t seem related, and the second one doesn’t seem to interfere. But it’s worth trying to disable both of those before running an export, just to see if there could be any interference.

By the way, when you say that the citekeys are still showing, does it look like this? (including the “Do Zotero Refresh”):

If so, then everything is actually working now, because this is the expected result. The next step is to open the Zotero tab in Word, then open the document preferences and press okay. This converts citations into proper Zotero citations that you can edit:

image

Lastly, if the above is not the case, did you try specifying the bibliography and CSL file like this as I suggested:

I apologise for my lack of specificity.

Currently, I’m trying the lua filter version so there are no backslashes to make double. My configuration of the Enhanced Export plugin looks like this:

(there is no space in pandoc.lua, no matter what that appears to be)

My file in pandoc/filters is this:

image

Unless you mean its link to the Pandoc path, which looks like this:

image

What I mean by “not working” is that I start with text in Obsidian looking like this:

Note: the same citation is there in multiple formats for testing purposes. The raw markup text for the three is, in order:

@[cathelat_2019]
[@cathelat_2019]
@cathelat_2019

A Word docx is indeed produced, but it renders like this:

Unless I am mistaken, these citations appear to be in raw unformatted markup.

Does that help at all?

You’re right! Sorry.

No, I was just confused by the fact that you first said:

And then:

Which would indicate that the name wasn’t already pandoc.lua, so it’s a contradiction.

Sidenote, the following is an invalid citation:


It appears so.

You still didn’t answer this:

Nor did you tell me if you tested this:

So it’s a bit tedious to only get half the answers each time.

Lengthy back-and-forth threads are an inherently confusing format.
The process for creating the file was:

  1. Open ‘c:\Users\name\AppData\Roaming’, and create new folders such that the path becomes ‘c:\Users\name\AppData\Roaming\pandoc\filters’
  2. Create a new text file, which appears with the default name: ‘New Text Document.txt’
  3. open this text file and paste in lua code (as retrieved from the web)
  4. rename text file as ‘zotero.lua’
  5. get told in this forum that it should have been named ‘pandoc.lua’
  6. rename file as ‘pandoc.lua’.

Hopefully that either clears up the confusion, reveals where I’ve buggered it up, or both.

I’d missed it becuase it flashes by so quickly and is immediately covered by the opening word doc, but a message does pop up from Enhancing Export saying:

image

Sorry I missed that. I don’t know how to disable an error message in the console. My understanding is that the console is just a print-out log rather than active code, so I’m not sure what there is to disable. I experimented with right-clicking the red message but this just gives a context menu saying:

image

None of those look like they would disable anything.

Experimenting with it a little, the red message (“TypeError: Cannot read properties of undefined…”) appears as soon as Obsidian is opened, regardless of which note it opens to. There’s something somewhere in the vault or setup that it doesn’t like. Disabling the Enhance Export plugin does not stop it from appearing on startup.

A new copy of the yellow warning appears whenever I try to export one of these Word docs, and I think is just saying that it can’t read the docx file it’s created (which is fine).

I do appreciate your patience with this frustrating situation, and continued willingness to help… but am I just borked now?

That’s not what happened. You told me you had created a file called pandoc.lua (see for yourself below):

And then you told me you were still calling the filter as zotero.lua:

So I pointed out that this was a mistake, given that if the file is named pandoc.lua, you can’t call it with --lua-filter zotero.lua.

So I said, either rename the file to zotero.lua, or call the filter by its current name, which you had said was pandoc.lua. Either option would make the names match:

I never told you to rename the file to pandoc.lua, because that’s what you told me it was already named.

I recommended option one (as you can see above), namely renaming the file from pandoc.lua to zotero.lua, as this is the name given to the filter by its maker, but you went with option two, which was keeping the file named pandoc.lua, and calling it as --lua-filter pandoc.lua. Whatever, as long as they match.

I meant momentarily disabling the plugins (Outliner and Tasks), of course. But I our takes on the error messages are aligned, in that they aren’t related to the failure to export citations correctly.

I don’t know, there must be something we are missing. Can you try redownloading the zotero.lua filter by going to this page and just clicking link here, which should download the file?

image

I am a bit baffled by why you decided to copy-paste the contents of the downloaded zotero.lua file into a new file called pandoc.lua, rather than just moving the original zotero.lua file into the filters folder in the Pandoc user data directory. Could it be that something went wrong during this copy-paste?

First of all, massive thanks to you @Feralflora ! I’ve been looking for this feature since a long time (as you can see here From Obsidian to Zotero - Zotero Forums)

I’ve followed your teaching and I will share it here, maybe it can help @AlexG.
Just to specified, I’m on Windows, working with pandoc 3.11.1 (to combine it with pandoc-crossref), obsidian V1.5.8, enhancing export plugin V1.10.4 and word 2022. Like @AlexG I use pandoc reference list
I’ve downloaded zotero.lua (without copy and pasting) and I’ve put it in the right folder

In Enhancing export plugin, I selected “Word” export and I put as extra-argument the following :
–resource-path=“C:\Users\mypath to zotero softwar”
–resource-path=“C:\Users\mypath to my bibtex library” (I think not mandatory for this one)
–resource-path=“C:\Users\my path to my vault attachment”
–filter pandoc-crossref (not mandatory, I use it to numbered my captions. Delete it if you don’t have pandoc-crossref installed in the same version of pandoc)
–number-sections
–lua-filter zotero.lua
–metadata=zotero_csl-style:“apa”

In the note I want to export, I don’t even need specific YAML !
Just make sure that the citation with which you want to have the “magic link” in zotero are formatted like this in obsidian : [@citekey]

Then, like @Feralflora said, you will just have to go in zotero tab in the word document that will open itself and select your way of citation in preference “et voilà” you will have your citation in word zotero format.
It will allow you to do your bibliography in word AFTER having completed the exportation from obsidian

I don’t know if it helps, but this topic definitely helped me ! Thanks guys !

1 Like

VICTORY!

I’ve got a version to work!

Only with the most basic lua command, and only with limited formatting options, but it works! YAY!

What was wrong?

It turns out the problem was here:

I had this set to the ‘Markdown’ template.

Why did I do a silly thing like that?

Because a) I didn’t notice ‘Word’ buried in the long list of options, and b) you are asked to select what format you want to convert it to each time you export a file:

Why would it ask you to select your format each time you export a file if that answer is already predetermined, by being locked in from the settings menu? That doesn’t make any sense, so I assumed the template setting was for some other mysterious purpose.

What arguments have I got to work?

At present, only this:
--lua-filter zotero.lua

(yes, I changed the file name back to ‘zotero.lua’ once I learned that it was arbitrary and only had to match in the two places)

What don’t I have working?

If I add even just –metadata=zotero_csl-style:“apa” to my additional arguments field, I get a pop-up error notice. It lists out the (very very long) command it submitted to pandoc, and then adds: “Unknown option -m. Unknown option -e. Try pandoc.exe --help for more information”.

For the record, when I try using the --citeproc command (with the --bibliography tag, etc) I get this same verbose error window, except now the last line reads: “Citeproc: citation cathelat_2019 not found”. I’m not going to bother troubleshooting that, but it is presumably solvable.

What this formatting can and can’t do

YES: “(Jones, 2024)”
YES: “(Jones, 2024; Conquerer, 1066)”
NO: “People like Jones (2024) have argued…”
NO: “(see, for example, Jones, 2024)”

To be fully usable for academic writing it would need to cover all of those options, but that is a battle for another day. Presumably, adding these will require a more sophisticated version of the lua code.

Tidy up answers to @Feralflora on prior miscommunications

All due apologies for the zotero.lua/pandoc.lua naming confusion. I’m sure I wrote the wrong thing in one line, and that set off a cascade of misunderstanding.

When I click the link to the lua filter it takes me to a page that shows the lua code in plain text. I don’t get any options to download the file directly by right-clicking the link, so my only option was to open the page, ‘select all’, and then paste it into an empty text file.

And, to be fair, that did seem to work in the end :slight_smile: