Do you get any errors in the console? I would expect there to be some error(s), but you haven’t made it clear whether you’ve checked the console yet. So do check the console (instructions here), and try using double backslashes in the path \\, or forward slashes.
There was a mistake made in the process, namely naming the filter pandoc.lua instead of zotero.lua:
This won’t work, as you are calling a filter named zotero.lua, which doesn’t exist:
So the file names of the filter don’t match as is. You should rename pandoc.lua to zotero.lua. Or alternatively, use the argument --lua-filter pandoc.lua, so long as the filenames match! (I would go with option number one, though).
You were right to create the user data directory yourself. For some reason, it’s not always present by default.
There was something you missed, so cheer up, there is hope yet
It’s not Pandoc doing that, though. Despite the name, Pandoc Reference List only uses Pandoc to convert bibliography files from .bib to .json, as required. I believe the name refers to the fact that it picks up on Pandoc citations and creates a reference list for those. I believe that the citation parsing is a custom implementation made by mgmeyers, and the citation rendering is citeproc.js, not Pandoc’s citeproc.
I switched the name of the file to pandoc.lua and updated that to be the same in Obsidian, but that still is not working.
Thanks for pointing me towards the console.
When I first load the document there is a red error which reads:
TypeError: Cannot read properties of undefined (reading 'editor')
at t.get (app.js:1:2091356)
at getEditorFromState (plugin:obsidian-outliner:88:13)
at VerticalLinesPluginValue.waitForEditor (plugin:obsidian-outliner:2552:28)
at new VerticalLinesPluginValue (plugin:obsidian-outliner:2602:14)
at e.eval [as create] (plugin:obsidian-outliner:2795:82)
at e.update (app.js:1:367539)
at e.setState (app.js:1:468006)
at t.clear (app.js:1:1378062)
at t.clear (app.js:1:2086260)
at t [as constructor] (app.js:1:1374961)
I don’t know really what that means.
When I try exporting to docx it throws a yellow warning:
If you could be more specific than “still is not working” that would be helpful. So you changed the filename (why, wasn’t it already pandoc.lua? Previously, you said: “I then created a \filters folder, and within this a file called pandoc.lua”) and you changed the filter to be --lua-filter pandoc.lua (correct?) - and then what happened? Did Enhancing export give a pop-up notification like this:
Or did it pop up an error message? Or did none of those things happen?
As for the console error and warning you mention, the first one doesn’t seem related, and the second one doesn’t seem to interfere. But it’s worth trying to disable both of those before running an export, just to see if there could be any interference.
By the way, when you say that the citekeys are still showing, does it look like this? (including the “Do Zotero Refresh”):
If so, then everything is actually working now, because this is the expected result. The next step is to open the Zotero tab in Word, then open the document preferences and press okay. This converts citations into proper Zotero citations that you can edit:
Lastly, if the above is not the case, did you try specifying the bibliography and CSL file like this as I suggested:
Currently, I’m trying the lua filter version so there are no backslashes to make double. My configuration of the Enhanced Export plugin looks like this:
Lengthy back-and-forth threads are an inherently confusing format.
The process for creating the file was:
Open ‘c:\Users\name\AppData\Roaming’, and create new folders such that the path becomes ‘c:\Users\name\AppData\Roaming\pandoc\filters’
Create a new text file, which appears with the default name: ‘New Text Document.txt’
open this text file and paste in lua code (as retrieved from the web)
rename text file as ‘zotero.lua’
get told in this forum that it should have been named ‘pandoc.lua’
rename file as ‘pandoc.lua’.
Hopefully that either clears up the confusion, reveals where I’ve buggered it up, or both.
I’d missed it becuase it flashes by so quickly and is immediately covered by the opening word doc, but a message does pop up from Enhancing Export saying:
Sorry I missed that. I don’t know how to disable an error message in the console. My understanding is that the console is just a print-out log rather than active code, so I’m not sure what there is to disable. I experimented with right-clicking the red message but this just gives a context menu saying:
None of those look like they would disable anything.
Experimenting with it a little, the red message (“TypeError: Cannot read properties of undefined…”) appears as soon as Obsidian is opened, regardless of which note it opens to. There’s something somewhere in the vault or setup that it doesn’t like. Disabling the Enhance Export plugin does not stop it from appearing on startup.
A new copy of the yellow warning appears whenever I try to export one of these Word docs, and I think is just saying that it can’t read the docx file it’s created (which is fine).
I do appreciate your patience with this frustrating situation, and continued willingness to help… but am I just borked now?
That’s not what happened. You told me you had created a file called pandoc.lua (see for yourself below):
And then you told me you were still calling the filter as zotero.lua:
So I pointed out that this was a mistake, given that if the file is named pandoc.lua, you can’t call it with --lua-filter zotero.lua.
So I said, either rename the file to zotero.lua, or call the filter by its current name, which you had said was pandoc.lua. Either option would make the names match:
I never told you to rename the file to pandoc.lua, because that’s what you told me it was already named.
I recommended option one (as you can see above), namely renaming the file from pandoc.lua to zotero.lua, as this is the name given to the filter by its maker, but you went with option two, which was keeping the file named pandoc.lua, and calling it as --lua-filter pandoc.lua. Whatever, as long as they match.
I meant momentarily disabling the plugins (Outliner and Tasks), of course. But I our takes on the error messages are aligned, in that they aren’t related to the failure to export citations correctly.
I don’t know, there must be something we are missing. Can you try redownloading the zotero.lua filter by going to this page and just clicking link here, which should download the file?
I am a bit baffled by why you decided to copy-paste the contents of the downloaded zotero.lua file into a new file called pandoc.lua, rather than just moving the original zotero.lua file into the filters folder in the Pandoc user data directory. Could it be that something went wrong during this copy-paste?
I’ve followed your teaching and I will share it here, maybe it can help @AlexG.
Just to specified, I’m on Windows, working with pandoc 3.11.1 (to combine it with pandoc-crossref), obsidian V1.5.8, enhancing export plugin V1.10.4 and word 2022. Like @AlexG I use pandoc reference list
I’ve downloaded zotero.lua (without copy and pasting) and I’ve put it in the right folder
In Enhancing export plugin, I selected “Word” export and I put as extra-argument the following :
–resource-path=“C:\Users\mypath to zotero softwar”
–resource-path=“C:\Users\mypath to my bibtex library” (I think not mandatory for this one)
–resource-path=“C:\Users\my path to my vault attachment”
–filter pandoc-crossref (not mandatory, I use it to numbered my captions. Delete it if you don’t have pandoc-crossref installed in the same version of pandoc)
–number-sections
–lua-filter zotero.lua
–metadata=zotero_csl-style:“apa”
In the note I want to export, I don’t even need specific YAML !
Just make sure that the citation with which you want to have the “magic link” in zotero are formatted like this in obsidian : [@citekey]
Then, like @Feralflora said, you will just have to go in zotero tab in the word document that will open itself and select your way of citation in preference “et voilà” you will have your citation in word zotero format.
It will allow you to do your bibliography in word AFTER having completed the exportation from obsidian
I don’t know if it helps, but this topic definitely helped me ! Thanks guys !
Because a) I didn’t notice ‘Word’ buried in the long list of options, and b) you are asked to select what format you want to convert it to each time you export a file:
Why would it ask you to select your format each time you export a file if that answer is already predetermined, by being locked in from the settings menu? That doesn’t make any sense, so I assumed the template setting was for some other mysterious purpose.
What arguments have I got to work?
At present, only this: --lua-filter zotero.lua
(yes, I changed the file name back to ‘zotero.lua’ once I learned that it was arbitrary and only had to match in the two places)
What don’t I have working?
If I add even just –metadata=zotero_csl-style:“apa” to my additional arguments field, I get a pop-up error notice. It lists out the (very very long) command it submitted to pandoc, and then adds: “Unknown option -m. Unknown option -e. Try pandoc.exe --help for more information”.
For the record, when I try using the --citeproc command (with the --bibliography tag, etc) I get this same verbose error window, except now the last line reads: “Citeproc: citation cathelat_2019 not found”. I’m not going to bother troubleshooting that, but it is presumably solvable.
What this formatting can and can’t do
YES: “(Jones, 2024)”
YES: “(Jones, 2024; Conquerer, 1066)”
NO: “People like Jones (2024) have argued…”
NO: “(see, for example, Jones, 2024)”
To be fully usable for academic writing it would need to cover all of those options, but that is a battle for another day. Presumably, adding these will require a more sophisticated version of the lua code.
Tidy up answers to @Feralflora on prior miscommunications
All due apologies for the zotero.lua/pandoc.lua naming confusion. I’m sure I wrote the wrong thing in one line, and that set off a cascade of misunderstanding.
When I click the link to the lua filter it takes me to a page that shows the lua code in plain text. I don’t get any options to download the file directly by right-clicking the link, so my only option was to open the page, ‘select all’, and then paste it into an empty text file.
But man, @AlexG, that’s literally the first pitfall I told you to look out for when switching to Enhancing export:
and again here:
Now it makes sense how nothing was having any effect.
I don’t say this to put you down, but this (and the number of other misunderstandings in this thread) tells me you would really benefit from reading a bit more carefully. Because I really tried to foresee these pitfallls and steer you around them, as you can see above.
See above
As for your point b / your question here:
Because it is not predetermined, that’s not what the dropdown is for. It doesn’t determine which format you export to by default, it just lets you switch between export command templates as you configure them.
Like I said in my first message in this thread, each file format you can output to has its own command template. This enables you to customize the command for that specific format. For example, by adding the zotero.lua filter to the Word command, and only to the Word command, because that’s the only place it is relevant.
It seems that you only have one dash at the beginning. You need to have two, as I showed you here:
I suspect you copied it from @Mathunter’s message above, which a) is missing a code fence and b) probably had some form of (iOS) autocorrect that changed -- into an em dash.
Either the bibliography path is incorrectly specified, or the bibliography file doesn’t contain the citekey / item in question (more likely).
I tested your citation examples using the corresponding Pandoc syntax like so:
---
zotero:
author-in-text: true
csl-style: apa
sorted: false
---
## Test conditions
YES: “(Jones, 2024)”
YES: “(Jones, 2024; Conquerer, 1066)”
NO: “People like Jones (2024) have argued…”
NO: “(see, for example, Jones, 2024)”
## Pandoc syntax test
Parenthetical citation follows [@Oreskes.Conway2010DefeatingMerchantsDoubt].
Dual citation follows: [@Oreskes.Conway2010DefeatingMerchantsDoubt; @Brulle2014InstitutionalizingDelayFoundation]
People like @Oreskes.Conway2010DefeatingMerchantsDoubt have argued...
(see, for example, @Oreskes.Conway2010DefeatingMerchantsDoubt)
And got this result in Word (after refreshing the citations):
So seems like number 3 works, and arguably 4 too, as I don’t see why the year shouldn’t be in parentheses in that case. Can you expand on that? And how were you citing, since you got a different result?
I see, must be a browser difference then, because in my case, it automatically downloads when I click that. But in any case, you should be able to right-click and choose “Save link as”.
Sorry for side tracking, but this is so true for many of the threads in this forum. It’s so demotivating to give an answer and see they don’t even read the answer, and instead repeats a question on something you’ve already answered.
So kudos to you @Feralflora , and others for being persistent and sticking with it through it all.
Actually, I didn’t use the right Pandoc citation syntax in example number 4. The following is the right way to add a prefix to the citation to get the result you were after.
So, to get this result: “(see, for example, Jones, 2024)”
you cite it like this: [see, for example, @jones2024]
Which then becomes this in word:
In conclusion, all the examples are fully supported using the right syntax.
Yeah, it really is @holroy. It feels pretty disrespectful to everyone’s time and effort. Of course, something can be misread or simply missed, but sometimes it just seems like people going tl;dr, even though some topics just require longer answers. Especially as you dig out from being knee-deep in misunderstandings.
You are right, the other formatting options do become possible when that front-code is added (it’s YAML, right?). I didn’t want to have to paste this code at the start of every file:
[quote=“Feralflora, post:19, topic:75428”]
---
zotero:
author-in-text: true
csl-style: apa
sorted: false
---
So I did a lot of trial and error with the “Extra Arguments” box, and worked out how to get the same command in there:
Fantastic! You are a whizz. I would never have figured that out on my own.
That is exactly where I got it from. I’d been confused about the em dashes in place of hyphens (I’d been experimenting with trying each in turn)… but of course, it must have been an auto-format thing. That makes so much sense!
Where things went wrong earlier
This is actually something that I can speak to as I’ve literally taught an entire university class on it.
Feel free to skip if this isn’t up your alley. I’m not here to force feed anyone lectures; There is no quiz
Psychologists have a name for this problem: “The curse of expert knowledge”. It’s hard for experts to explain things to beginners, because they see intuitively obvious layers of meaning, context, and nuance in the words that they say. It is very difficult for them to see how their words don’t carry all this meaning for beginners and they can be very confused as to why the beginners struggle, or misconstrue their words in such obvious ways.
For this reason, companies that build commercial products (as opposed to gold-hearted volunteers who help the world for free) never let their engineers write their instruction manuals. The engineers will tell people to “adjust the upper flanges until they achieve a 3mm tolerance to the outer manifold”, and then will be utterly baffled when beginners don’t follow along with (what they see as) such clear and precise instructions. Companies with budgets employ technical writers to translate this kind of expert-speak into normal English. That might not sound like a necessary job, or even a hard one, but, unfortunately, it very much is.
Take your explanation above:
That paragraph is easy to parse into a clear meaning, but only if you start with a clear picture in your head of what a command template is, what a lua file is, and presumably also what pandoc does, and how those three things interact to translate documents. That context is immediately obvious in your mind, making the paragraph a clear read. But I’m still not quite sure what it means despite reading it three of four times. This isn’t because I’m dumb or being disrespectful, and it isn’t because you are bad at explaining. It is because it’s legitimately hard for humans to communicate across such different levels of baseline understanding. It’s a well-known phenomenon.
If the Enhanced Export plugin was built by a company with big enough resources (instead of being somebody’s labour of love), they would have spent a bunch of money hiring UX experts (User eXperience) to do dozens of hours of user testing: The UX workers would watch consumers try to interact with a series of prototypes, and then tweak the designs until they finessed away any points of confusion. It’s slow, skilled, and time-consuming work but it means you end up with far fewer people like me flailing through patient attempts to solve their problems. To be doubly clear, this is not a realistic ask of smart hackers who are putting tools together on their own, and then letting people use them for free! I realize this!
Please don’t take ANY of this as veiled criticism, or a demand that you do more - not unless someone starts handing you a budget to hire skilled helpers.
I just saw your frustration and thought it might provide a tiny bit of solace to know that we’re not trying to be disrespectful when we struggle with seemingly basic instructions. It’s to your enormous credit that you keep on rolling through these inevitable frustrations and find ways to help people anyway. It’s extra work for you (and the people you are helping), and kudos to you for that.
You don’t have to trial-and-error your way to success
Thanks! But this serves as an example of what I’m talking about:
You don’t have to figure this out yourself! You got the solutions already in the first response.
See, you could have just referred to the Pandoc citation syntax documentation, which I linked for you in my first message, since you were citing using the wrong formatting:
Now, I don’t know if you’ll consider this jargon as well, but here’s what those Pandoc docs say:
Citation items may optionally include a prefix, a locator, and a suffix. In
Blah blah [see @doe99, pp. 33-35 and *passim*; @smith04, chap. 1].
the first item (doe99) has prefix see, locator pp. 33-35, and suffix and *passim*. The second item (smith04) has locator chap. 1 and no prefix or suffix.
This is what I referred to when correcting my example for you. So, when you say you would never figure it out yourself, despite having been pointed to the same resource, how can I help but to think that you aren’t reading the necessary resources?
Hidden layers and responsibility
While I acknowledge what are saying about the layers of meaning that we take for granted, I think you also have to take responsibility to a degree where I can assume that you — as a budding Pandoc user — are attempting to familiarize yourself with the fundamental basics of Pandoc.
As for your specific example, first I want to emphasize this sentence comes at the conclusion of a lengthy thread. Any clarifications should have been sought much earlier.
Second, Pandoc is a command-line tool, so talking about commands shouldn’t be out of place, and if it is, Pandoc has a “Getting started” page that caters to you specifically:
This document is for people who are unfamiliar with command line tools.
Perhaps it’s the curse at play, but I don’t see what else in that sentence can be considered jargon, besides filters. File formats? Output?
Hope this breakdown helps:
Pandoc converts text files from one file format to another.
The output is the results of such a conversion.
You use commands composed of so-called arguments to interact with Pandoc.
Enhancing export conveniently stores and executes these commands for you.
Enhancing export comes with default command templates with the most common arguments needed to convert from markdown to various other file formats.
For academic purposes (citing and bibliographies) we need to add some additional arguments.
Pandoc filters are scripts that change the default conversion behavior of Pandoc, giving you a different output.
The zotero.lua filter we are using here adds code that enables Word to recognize the citations, as if they were inserted by the Zotero addon for Word.
There’s a limit to how far back we can go with these clarifications.
Returning to responsibility, I want to emphasize that at no point did you ask:
“What is a lua filter?”
“Can you clarify what an export template / command template is?”
“When you say the ‘word template, and not something else’ can you clarify what you mean?”
“What is the dropdown of file formats in Enhancing export’s settings for?”
Framing: From expert jargon to motivated reasoning
Embracing the commentary on communication, I think we also have to consider the phenomenon of motivated reasoning, in particular identity-protective reasoning. Emphasizing the “curse of knowledge” frame is arguably identity-protective and alleviates feelings of cognitive dissonance after having done something “silly”, to quote you. It’s a well known phenomenon.
And the lack of clarification, à la the bullet list of definitions above, is as much a consequence of the time-constraints involved in debugging issues for others during one’s downtime (e.g. while on the toilet) — coupled with an expectation that the other party also does their part with the given resources — as it is a consequence of jargon blindness. Or maybe this is just my motivated reasoning at play…
Nevertheless, I do of course agree that there is plenty of room for improvement in the UX and user guide (or lack thereof) for Enhancing export. This is also due to a language barrier (note the plugin name).
With that said, I think with some self-reflection beyond the frame of “the curse of expert knowledge”, you’ll find that you’ll be more likely to get help in the future without someone as stubbornly persistent as me coming to your aid.
When I tried to pick up basic PHP to customize a website I knew I was in for weeks of YouTube tutorials, web manuals, and StackOverflow threads, and I was happy to do that. It was fun. It’s obviously an easier and quicker challenge to learn pandoc’s command line interface and configuration settings, but it’s also a less widely useful skill, and probably not a challenge that I would have signed up for knowingly.
When I picked up Obsidian I knew it was more customizable than all the slick commercial software that is fully UX tested for raw beginners, so I was certainly expecting a learning curve; Googling recipes and tweaking to implement them, etc. It was only when I stated setting it up for academic writing that I started getting sucked deeper into the pandoc rabbit hole. I was kind of hoping I could just look up a standard command or two, instead of having to learn an entire architecture with its own syntax (even if not a deeply complicated one).
So I never set out to become, “a budding Pandoc user”, or really wanted to be one… If I knew what the learning curve looked like faster I’d probably have cut my losses and just used something else for academic writing.
But here we are now on the other side.
So, all that said, your description of pandoc above is much MUCH more beginner friendly than the standard web manuals. Have you thought about posting it as a resource for beginners (other than a long way down the bottom of a discussion thread that few will want to wade through)?
I’m glad we made it through! Thanks for clarifying your perspective.
And I agree that there is a need for a beginner-friendly Pandoc guide, tailored to Obsidian users. I actually started working on this a while ago, so this exchange was helpful in shedding more light on what needs clarification. If there are some specific explanations you think this guide should contain, then it would be helpful for me if you could share your thoughts on that.
Heck, I’d be happy to help you edit it. I don’t know much about pandoc (but that might actually a useful perspective), but I do have some writing skills and I know a bit about UX.
If I were writing it my rough first draft would start with an outline something like this:
Setting
This guide is about writing documents that include academic citations - be that for journal articles, books, your own notes, etc. Making this work withing Obsidian is the easy part. This guide also covers how to export documents into Word (or other formats) that include correctly formatted citations. This is the more difficult part.
Other guides you will find online with “academic workflows” are about how to automatically create neatly formatted notes with Obsidian that contain information drawn from Zotero entries. This is not covered here.
What needs to be done
The basic problem here is getting at least four different things to all work together:
Zotero has a system for tracking and formatting citations
Obsidian writes text in markdown, which allows for some basic formatting (that we assume you are fmailiar with). It doesn’t have any native capacity for citations
Word (or other format, like PDF) has its whole own syntax for handling formatting and doesn’t natively handle citations either.
pandoc is a program that specializes in translating documents from one format to another. On its own t doesn’t know anything about Zotero either, but there are ways of telling it where to find all the required information
That’s a lot of moving parts and it takes a little bit of plumbing to get them all working smoothly together.
Warning: It will probably take at least an hour to work through all of the steps in this recipe. Possibly longer if anything goes wrong or you need more unusual tweaks
Step 1: Getting the supporting cast into place
Some of those programs need a bit of extra help to give them the tools to deal with each other.
Pandoc Instructions on:
how to install it
How to find the path
Zotero
Install better bibtex. Why? It gives more useful internal names to the documents it tracks, that make it easier for other programs to talk to it.
Instructions on creating a library file
Instructions on where to find the citation format file (for APA7 or whatever)
Obsidian Install:
the enhanced export plugin to send citations anywhere else, and
the Pandoc Reference List plugin to show formatted ciatitions within Obsidian . We’ll tell you how to set these up below
Step 2 (optional): Setting up Better Bibtex
Various people have posted instructions for renaming keys into an academic shape (e.g., Jones_1998 whatever). This isn’t required but does make them more intuitive to work with, so is recommended
Step 3: Setting up Obsidian Pandoc reference list plugin
This formats references when they are presented within Obsidian.
NOTE: This has nothing to do with them being formatted when you export them to any other format. You can skip this step if you don’t mind the references looking a bit funky when you are working within Obsidian
I can’t remember where I found the recipe for htis, but it was relatively straightforward. You have to look up the path for where pandoc is, and past it into the right box IIRC
Step 4: Setting up Obsidian Enhanced Export
This is the fiddly bit.
Pandoc is the program that does all the heavy lifting here. Working with it directly takes some skill, so the Export plugin was created to give it a (somewhat) more user friendly interface within Obsidian.
The problem is that even with this more user friendly interface it still takes some doing to get it to coordinate between Obisidian, Zotero, and Word. Doing so still requires finding the right set of commands to feed it. The good news is that we will help you find those commands, and show you where to put them into the plugin so that it remembers you for them, and automates it every time you want to export a new note.
pictures of which tick boxes need to be ticked how
It’s best to leave the boxes full of detault text alone unless you really know what you are doing.
That “Extra Arguments” box is where we are going to tell it the commands necessary to manage citations.
The rules for these commands (“arguments” in tech-speak) are pretty simple.
You can enter as many commands into that box as you like
Each new command is separated by a space
if you want to write yours out in a text editor it will be a lot more human-readable if you separate them with line breaks - those will turn into spaces when you paste them into the Extra Arguments box
Each command starts with two hypthens, and then can’t contain a space within it (unless it is text like a file address that is encolosed within “quotation” marks)
It doesn’t matter what order you enter your commands (right?)
you will have quite a few commands. Don’t panic about this.
Option 1: The basic way
The goal here is to tell Word (or whichever) what a fully formatted output should look like. You need to tell pandoc where to find a full description of your output format, and then give it separate commands to create the in-text citations, and another one to create the bibliography at the end.
List of required commands, common options, and a demo of what the paths look like in Windows and Linux
Option 2: The fancier way (that let Word and Zotero work together to reformat documents)
MS Word has its own plugin for Zotero that lets you put special placehoulders for citations into your document. If you do this you can ask Zotero to translate the entire thing into whatever format a given journal requires, and you can repeat that translation every time you submit it to a new journal with a different format. That’s pretty handy.
You can ask Pandoc to create a Word doc with these placeholder values. It’s just slightly more complicated.
The bad news: the commands are so complicated that they require dozens of lines of commands sending to pandoc
The good news: Someone has complied most of those commands into a document that you can place in a file called a “lua filter”. All you need to do is point pandoc at this filter file, and it will do the hard work for you.
Instructions on downloading the lua filter, where to place it,
required and optional formatting commands
Step 5: Citing with this new setup
Instructions on entering the various permutations into the Obsidian document [see @Jones_2024], etc
If you use the option above with a lua filter, then when you have exported into Word it will look like X, at first, but then you click this button and everythig is tidied up.
Ok, that was a lot more involved than I intended to start out writing. A lot more involved.
If you do want to cowrite / edit this thing then is there a way you can send a private message and we can work offline? I’m not sure this forum is the most useful place for long-form editing work.
