Writing academic papers - suggestions

[TMR123]

I am trying to use Obsidian (as a novice user) to prepare an academic paper, rather than using a word processor like MS Word. It is nice in that I can keep the ideas separate (facilitating reuse) and follow a top down approach. However, what I can’t see is how to combine a note with its linked notes into a text file, or MS Word, to share with colleagues. I have tried:

• Pandoc plugin which does produce a single file but doesn’t include the linked pages.
• Webpage HTML Export that can export all the notes from a given folder, but:
  1. The article and related notes need to be in the same folder, limiting reuse
  2. What is produced is an HTML structure and not a single document that a colleague could easily work on.

Any suggestions by anyone else using Obsidian in this way.

Thnx

[jeetsukumaran]

There is the longform plugin:

GitHub - kevboh/longform: A plugin for Obsidian that helps you write and edit novels, screenplays, and other long projects.

Not tried it myself, but it seems like a really nice way to stitch together docs.

Unfortunately, it does not seem to do embeds.

I’m sad to say that of the two manuscripts I’ve started trying to put together in directly in obsidian, in both cases I gave up and just wrote it out in a LaTeX document from scratch.

One challenge I’ve faced when trying to move from vault to final manuscript is the handling of internal links. I think pandoc ships with a filter that strips the link markup from mediawiki style links, but that then leaves the [[ugly-and-probably-meaningless]] file paths in the text. I’ve forced myself to alias every internal link with a phrase that works even if the link is stripped away to support this outflow approach. (Paradoxically, my attempt at making my information more robust made it especially vulnerable/fragile to the 1.2.2 pipe bug wreaked – so much havoc with my vault! )

But that solves only part of the outflow challenge. The other one is citations. My vault notes maintain references to source notes (e.g., [[@darwin1856-origin]]), but pandoc needs actual citations. Fortunately, the file naming convention the Citations plugin suggests works really well when stripping the external [[ ]] brackets … BUT then the wikilink processor breaks this. I finally hacked in a mostly-working Pandoc filter that preserved the source references as pandoc citations.

The other issue was that eventually I felt I needed more control over layout, math expressions, and (especially) image embedding, layout, sizing etc. than any pure Markdown format flow could give me.

Really interested in seeing/hearing about other folk’s workflows that might work better .

[CawlinTeffid]

It sounds like you are keeping the parts of your paper spread throughout the vault and mixed with other things. If so, that seems too complicated and fragile to me. I would copy-paste the things I need into my draft and work from there. Some people break down their drafts over several files, but as far as I know they usually keep the parts together, which enables the use of things like Longform or the HTML export plugin you tried. This isn’t to say your approach is invalid, but I believe it’s unusual and that you may be making things harder than they need to be.

If you want to stick to your current approach, it would help to share more details about it.

[jeetsukumaran]

@CawlinTeffid

If so, that seems too complicated and fragile to me. I would copy-paste the things I need into my draft and work from there.

Yep – that’s what I ended up doing. And as far as putting together the concept paper behind the result, it’s great.

The only reason I switched (back) to LaTeX for the final document is that it got to be quite frustrating getting pandoc to do things just so with layouts etc., especially in resizing and gridding/squishing together images on slides. So since I was copy text over anyway into a new document, might as well put it into LaTeX which gives me full control over these things.

I’ll just note that that the frustrations I had with markdown for the final product really has not much to do with Obsidian but rather the Markdown format and support for publishing from there, in conjunction with my unfamiliarity with Pandoc filters. Also, there’s probably cultural prejudice here, because I find the LaTeX so much easier to structurally parse in my head than the corresponding pandoc:

\begin{frame}{Title}
  \begin{columns}
    \begin{column}
        \item The quick brown fox
        \item Best of times
    \end{column}
    \begin{column}
        \item Foo Bar
        \item Hello World
    \end{column}
  \end{columns}
\end{frame}

than:

::: {.columns}
:::{.column}
- The quick brown fox
- Best of times
:::
:::{.column}
- Foo Bar
- Hello World
:::
:::{.columns}

And once I start to add in things like more complex layouts or (shudder) tables, then the Markdown visual noise might exceed even that of LaTeX! I’m still hoping to give it a try for simpler presentations or manuscripts.

[TMR123]

My workflow is somewhat different and considerably more primitive. As an academic of many years I find that my “fly by the seat of my pants” approach to publishing, and indeed research in general, has become less effective as time goes by. Perhaps it is just old age, but I forget half the things I wanted to write about, and can’t find the other half. Hastily scribbled ideas and notes on conference proceedings, articles and beer mats, seem to magically disappear right when I need them and only appear when they are of no use.

Hence, my interest in Obsidian. I have tried other things before arriving here, like local text notes, Google Docs, and even Notion, but none really worked for different reasons, I won’t bore you with. I like the promise of working in linked text files and having a collection point for all my ideas, knowledge, etc., and then being able to search, combine, and reuse it in a flexible way. Also, there are lots of videos on YouTube of people demonstrating how they use Obsidian, which is a great way to learn a new tool!

I originally used to write academic texts in Emacs, using TeX/LaTeX to format it (my thesis for example ). These days, I pretty much use MS Word for everything. Journals and conferences always provide templates; most of my colleagues use it; it is free at my university; most of my work is collaborative. So, I end up working in Word, passing different versions back and forth. I can’t see that changing anytime soon.

As well as using Obsidian for building a PKB, I am also road-testing it as part of the bootstrapping process of the current article I am working on. When the text is sufficiently mature to pass over to my colleague to work on, then I’ll have to dump it in Word format. So far, so good. The text is coming together quite nicely and consists of a main document, that will eventually become the article and a lot of links to concepts and literature summaries that are relevant. This feels comfortable and is very flexible, with new notes appearing as I go along. However, I get stuck in the last part of this process, that of exporting the notes:

• Pandoc is OK but only dumps the top-level file into Word. Links to other notes can be included, but this doesn’t provide anything that can be shared with non-geeks.
• Longform seems to be intended for a writer who starts off knowing the set of different subfiles there will be (such as scenes in a literary work) that can be compiled into a central file, but this doesn’t let you add new (or reuse old) notes as you go along and then have them all compiled together in a recursive manner. I did RTFM but couldn’t figure out how to tag a current file as one that should be included in a given Longform project to be compiled with the rest.
• Webpage HTML Export enables, amongst other things, a given folder of notes to be dumped as HTML files, which are easily reviewed in a browser, but this doesn’t include notes in other places, and also, produces something that is read only!

So, I’m not sure what I’ll do in the end, or whether I’ll repeat this process for the next article. It depends on the relative benefits of having my academic knowledge structured and reusable in Obsidian vs. the pain that exporting a largish set of notes.

Regards

[Dor]

I have a number of methods, but all involve using other programs. I can put notes together, I can make pandoc work, or HTML, but all these are high friction and require shifting my mind to making it work rather than the content. For a number of reasons, I don’t use Longform.

Since most of what I do needs to end up in docx before being passed on, I might use Word. Review and comment features are good and collaboration easy. Usually, I will use the Word plugin Writage to read/save markdown. I’ll switch completely to the docx version because it’s otherwise hard to know which is the canonical copy.

More commonly, especially in earlier stages of preparation, I will use WriteMonkey3. I have it set up to sync with the markdown notes wherever they are; this means I can do the writing or editing in any markdown editor I want. Within WM3, it’s easy to shuffle sections and notes around and then export to a new version. Has many features and it’s my favourite program for writing. But I wouldn’t recommend it because the UI isn’t newbie friendly. But always, in the end, it is imported into docx.

Ulysses (Mac/iOS) and Inspire Writer (Windows) are also good for writing support restructuring note sequences. They can work with markdown notes in the file system, but most functionality is reserved when the notes are inducted into the database. I won’t generally be prepared to do that.

[TMR123]

Thanks Dor, a very nice set of tools to explore…

Mind you, I realize that I don’t actually need to work in markup outside Obsidian, all I want is an easy way to bring all the linked content back into the original file. So, since we are working on text files, a few lines python is enough to do that, something like:

import re

filename = 'R.md'  # Replace with the name of your Markdown file
output = []

with open(filename, 'r') as file:
    for line in file:
        links = re.findall('\[\[.*?\]\]', line)

        if links:
            for link in links:
                link = link[2:len(link)-2] + ".md"
                print ("link is",link)
                if filename != link:
                    with open(link, 'r') as linked_file:
                        output.append("\n---\n")
                        output.append(linked_file.read())
                        output.append("\n---\n")
                        output.append("**Coming from link:**\n")

        output.append(line) 

with open(filename, 'w') as file:
    file.writelines(output)

This is very, very rough around the edges, since it only took me 5 minutes to write, and is not even recursive, and should definitely NOT be run on any files in your vault, but proves the point. The resulting file can added back to Obsidian and then put through Pandoc to turn it into Word.

Regards