Import from TiddlyWiki 5 to Obsidian

I migrated around 2,600 entries from TiddlyWiki 5 to Obsidian.

My method was based on this really helpful blog post. These instructions are for MacOS. I’m not much of a programmer so a lot of this stuff is kinda hacky and someone can probably come up with a cleaner version of this. I am happy with the end result though.

Here’s an outline of my steps:

1. First export all the Tiddlers to HTML files

Run this command in your TiddlyWiki directory

tiddlywiki --render [!is[system]] [encodeuricomponent[]addsuffix[.html]] text/plain $:/core/templates/custom-template

A couple notes about that command: 1. I am using the encode URI flag because I have a lot of entries with punctuation and special characters which I wanted to change manually. You may not need it. 2. I created a custom template for myself that has the tags in plain text listed at the top. Here is my template:

title: $:/core/templates/custom-template

`created: `<$view field="created" format="date" template="YYYY-0MM-0DD"/>
<$fields exclude='text bag title tags type created modified' template='$name$: $value$
'></$fields>

<$list filter="[all[current]tags[]!is[system]sort[title]]">#<$view field="title"/> </$list>`
`
<$view field="text" format="text" />

2. Rename the files

I used Hazel on Mac for this but you could use other apps

  • Do a bit of cleanup to remove the URL encoding on the file name. I do this so that I can control exactly how the characters get replaced in the file name.
  • Change creation date and modification date

3. Convert to Markdown with Pandoc

Run this command on the folder that contains all your HTMLified tiddlers.

for f in *.html; do pandoc "$f" -f html-native_divs-native_spans -t commonmark --wrap=preserve -o "-md/${f%.html}.md"; done
  • iterates on all files in a folder
  • puts them in a “/-md” subfolder

4. Cleanup with Sublime Text

Using the batch find/replace on the whole folder I ran a lot of cleanup on the entries.

I couldn’t figure out how to stop Pandoc from escaping all kinds of characters so I ran this (in normal find/replace mode)

Find: `\` Replace: nothing

Also needed to clean up internal links. Pandoc spits internal links out as [Note Name](#Note%20Name) Use regex mode (click the * icon in the find/replace UI) to find internal links and replace them with [[ ]] notation

Find:  `\[([^\[\]]*)\]\(\#(.*?)\)`
Replace:  `[[$1]]`

That’s about it! Did some other minor cleanup with Sublime. LMK if this helps. I think there is definitely room to simplify the process.

9 Likes

I was also using Stroll (based on TW5) and had put aside migrating over folders until I could find an easy method, so glad to find this!

Except I’m so code-illiterate that even though I’ve read through your, and the original link’s step by step instructions, I can’t get past the first step haha (for example, after I open an empty TW, how do I allow markdown? - this was step 1 of the article)

if someone would be so kind to volunteer their time and literally write every single step so someone like me could import their TW files to markdown, I’d super appreciate it <3

totally get if no one does cuz it would be so tedious, but still wanted to comment that this was a very helpful post! (probably since most here seem to know basic coding)
*I am working to learn! just a very slow progress :’(

I don’t think you need to allow markdown, you can just run that command in the folder that contains your TiddlyWiki installation (the parent of your “tiddlers” folder)

1 Like

thanks for the reply!
When you say to “run the command”, is it by using a program? Or is this something that’s done by making an edit within a file?

1 Like

I had the same questions as you initially and am working on writing down a step-by-step workflow. Which OS are you on?

1 Like

How to Export from TiddlyWiki to Obsidian

Note: I am on Mac, so all the steps are specially for that OS. I followed the steps by @kepano and just tweaked/clarified the things that didn’t work for me initially. So this is very much his work. @pikacho, I hope this is helpful for you and others.

Step 1: Set up TiddlyWiki in Node.js

  1. Download and install Node.js
  2. Open “Terminal”, run npm install -g tiddlywiki
    • If an error occurs, run sudo chown -R $USER /usr/local/lib/node_modules (that gives you admin privileges for the specific folder
    • Run npm install -g tiddlywiki again
  3. Check if it works by running tiddlywiki --version. If everything worked it should show you the version of TiddlyWiki
  4. Run tiddlywiki myfirstwiki --init server
  5. Run tiddlywiki myfirstwiki --listen to start the server

Step 2: Import Notes into new TiddlyWiki

  1. Open http://127.0.0.1:8080/ in your browser. It should load an empty TiddlyWiki.
  2. Drag the file that contains the TiddlyWiki you want to export (for me it’s “empty.html”) in the top of the browser window
  3. Press “import”. This imports all of your Tiddlers into the empty wiki so we can export them later. You might need to reload the wiki.
  4. In Terminal, press Control + C to exit the server

Step 3: Export Notes as .html

  1. Navigate to the folder “myfirstwiki” in Finder.
    • For me that’s ~/myfirstwiki/. In Finder Shift + Command + G opens a window in which you can copy my path.
  2. Right click → Services → New Terminal on the “myfirstwiki” folder.
  3. In that Terminal window run tiddlywiki --render [!is[system]] [encodeuricomponent[]addprefix[tiddlers/]addsuffix[.html]]
  4. You can find the exported Tiddlers in the “output” folder in the “myfirstwiki” folder.

Step 4: Change .html to .md notes using Pandoc

  1. Install Pandoc

  2. Navigate to the “output”-folder containing the .html Tiddlers

    1. Open Terminal
    2. Show directory by typing and entering pwd
    3. Move to directory
      • cd makes you enter a folder, pwd shows you the directory you’re in
      • This is how I entered the folder, yours might be slightly different
        • cd myfirstwiki
        • cd tiddlers
        • cd output
        • pwd to confirm you’re in the right directory
  3. Once you’re in the right folder run this command in Terminal: for f in *.html; do pandoc "$f" -f html-native_divs-native_spans -t commonmark --wrap=preserve -o "${f%.html}.md"; done

  4. In Finder, I then copied and pasted the Markdown files into a separate folder.

A note on the original code by @kepano
  • The original code (for f in *.html; do pandoc "$f" -f html-native_divs-native_spans -t commonmark --wrap=preserve -o "-md/${f%.html}.md"; done) that puts all of the files into a seperate folder didn’t work for me. All of the tiddlers that are transferred to Markdown (.md) files are now in the same folder as the .html files.

Step 5: Clean up using Atom

  1. I use Atom, other editors will work just as fine. I loaded the folder with all of the Markdown Tiddlers into Atom and then opened “Find in Project.”
  2. Clean up links.
    • Use regrex mode (click the * icon in the find/replace interface)
    • Find: \[([^\[\]]*)\]\(\#(.*?)\) Replace: [[$1]]
    • This cleans up the links from [Note Name](#Note%20Name) to the [[]] notation.
  3. After that, I didn’t need to do any more cleaning up. Run more find/replace searches if necessary.

Step 6: Import into Obsidian

  1. Navigate to the Obsidian Vault in Finder. Create a new folder “TiddlyWiki Import”.
  2. Copy and paste the Markdown Tiddlers in the folder.
    • Note: A lot of my notes were just empty pages that got created by linking to them. The way Obsidian handles backlinks makes them redundant. In Finder I sorted the converted Tiddlers by size and only imported the ones that were bigger than 1 byte.
  3. Open (if it’s already open then reload with Cmd + R) Obsidian and voilà! Your notes are right in the TiddlyWiki Import folder!

Some final notes
I excluded the use of a custom template in step 3 to minimize friction. I can see how that would be very handy to add though. Also somehow my journaling notes didn’t get exported properly so I will have to manually transfer those.

I am not a coder, so my knowledge comes just from trial and error. If more experienced forum members have things to add/simplify please let me know. All of these steps worked well for me but if you run into issues, let me know and I will do my best to help :slight_smile:

6 Likes

Hi folks!
I wanted to publish my notes on a website, so do you guys know how to go from obsidian to Stroll/tiddlywiki?

@Joschua is that for windows?

I’m using MacOS, so no. It might work as well I just haven’t tested it.

Thank you for taking the time to write the instructions. I followed it and it worked for me.

There is a slight error in your instruction in step 3, number 3 - tiddlywiki is missing at the start. So, it should read:

tiddlywiki --render [!is[system]] [encodeuricomponent[]addprefix[tiddlers/]addsuffix[.html]]

Hey @fusionmuck! Glad to hear that it worked for you, even with that error. Thank you for letting me know, I fixed it now!

You’re welcome - the instructions was very well written!

1 Like

These steps by @kepano and @Joschua proved really helpful for me in my migration. I would like to add a few pointers to help anyone who may have faced some of the challenges I did.

In Step 5 of @Joschua’s solution, I still had html anchor tags in my markdown files. So I used the following regex to fix them. Find: <a[\s]+([^>]+)>((?:.(?!\<\/a\>))*.)<\/a> Replace: [[$2]]

Also, after I was done, I had %20 in my file names because of the spaces I used in TiddlyWiki. This solution: https://unix.stackexchange.com/a/174131 helped me resolve that.

Cheers.

1 Like

There is a post on reddit for that. I had it working for me. But the problem arose when I began using [[ | ]] notation for alias. In tiddlywiki5 its the opposite of Obsidian. If you like to use the basic markdown notation. you will have everything working!!

Let me know if you need the link

Hey there! I’m trying to do something very similar. Can you explain a little more how you used the “custom template” to generate the tags in plain text listed at the top?

Did you create a tiddler entitled $:/core/templates/custom-template before exporting?

Thanks @kepano and @Joschua for the instructions! I’ll add what I had to change for Windows, plus some adjustments/details if anyone else wants to play with it :slight_smile:

The first thing to look out for on Windows is, you need to open your PowerShell in the folder where you want to create your tiddlywiki server folder in - if you just leave it on the admin PowerShell, it’ll create it in system32 and you’ll have permission problems when you try to do anything with it.

about exporting the tiddlers to html files:
@sweenejp I don’t know if you still need to know this, but for you or anyone else looking to do this, to use the custom template, you need to leave the tiddlywiki server running in a different terminal window, then it’ll take the template flag. You do need to create that tiddler and have it in the same tiddlywiki you’re exporting from - in Windows, I just created a tiddler called custom-template, because having $ in the title was giving PowerShell problems. I don’t think it matters what you call it, as long as you use its full name in the template flag.

I also ended up exporting the metadata into a separate file, like the original linked instructions suggested, mostly because the note transclusion wasn’t working for me the way I wanted when I made the text a part of the template. So my full code for the render was the following, with the tiddlywiki server running:

tiddlywiki --render [!is[system]] [encodeuricomponent[]addprefix[tiddlers/]addsuffix[.html]] --render [!is[system]] [encodeuricomponent[]addprefix[tiddlers/]addsuffix[.meta.html]] text/plain custom-template

My custom-template - very much @kepano 's code, with slight changes - looked like this:

created: <$view field="created" format="date" template="YYYY-0MM-0DD"/>
title: "<$view field="title" format="text"/>"
<$fields exclude='text title bag tags type created modified' template='$name$: $value$
'>
</$fields>

tags:<$list filter="[all[current]tags[]!is[system]sort[title]]">
- <$view field="title"/></$list>
This outputted an html file that seems to have the best results when put through pandoc, and a metadata file like so:
created: 2015-07-27
title: "Adventure making checklist"
creator: Vero
modifier: Vero
tags:
- ttrpg
- how-to

After that, cd output\tiddlers (or just cd output, if you didn’t use the addprefix[tiddlers/] version), and here’s the actual difference for Windows.

This needs to run in PowerShell specifically, with the actual path being the path to your tiddlywiki server:

$files = Get-ChildItem -Path C:\Users\username\Documents\tiddlywikiserver\output\tiddlers -Recurse -ErrorAction SilentlyContinue -Filter *.html |
  Where-Object { $_.Extension -eq '.html' }
foreach ($f in $files) {
$newName = $f.Name + ".md"
pandoc "$f" -f html-native_divs-native_spans -t commonmark --wrap=preserve -o "$newName"
}

The markdown files created this way don’t need to be search+replaced for escapes, and seem to look fine, though I’m still going through mine. Still, this output seems to need the least work while keeping the most semantic tags from the html versions, which works for me :slight_smile:

I then cleaned up the filenames using Bulk Rename Utility, and created a Templater template that lets me add the meta into a yaml frontmatter from the meta file into the content file. You need to have both the files in the same folder in Obsidian and their names, if you used the --render I did, are the same plus .meta for the metadata. Open the content file, leave your cursor on the beginning of the file and apply the template:

---
<% tp.file.include(tp.file.find_tfile(tp.file.title + ".meta")) %>
---


Thank you to everyone that has shared the steps they’ve taken to migrate but when I tried migrating through HTML export I found too many issues with the output. I wrote a somewhat opinionated Java command-line tool to handle a direct migration from tiddlers to Obsidian markdown and wanted to share it here for anyone facing the same issues.

Basic usage is:

java -jar tw2md.jar ~/Documents/tiddlywiki/myWiki ~/Documents/obsidian/tiddlyWiki

Feel free to give it a shot, I’m happy to merge PR’s for bug fixes and improvements.

1 Like