Dear community, I have been using WikidPad for many years, but finally migrated to obsidian, for the modern functionality it offers.
I have written a Python script that did the migration for me. It is crude and not well documented, but if there is interest I could put up a smoother version.
2 Likes
Cool! Wikidpad was the first text-file–based notes app I used, years ago. I still have some notes in my vault that I originally wrote in Wikidpad, demonstrating the value of plain text files. (Between then and now I’ve used Zim Wiki and a variety of text editors.)
Thanks a lot for this!
Me too, but I miss some Wikidpad-features, especially the link view tree (Link tree view in the left sidepanel) which is clearer than the graph view. I migrated because the development of Wikidpad stopped, and I’m afraid that I can’t use it any more at one time or another.
I have my old Wikidpad database with about 5000 notes still in use, and migrate manually, when I update an item.
Your script seems great. I’m not a programmer- I really would appreciate a smoother and better documented version.
If there isn’t already a plugin that does that, you could post a description in Feature Requests or Plugin Ideas (unless one already exists, in which case you could add your voice to it).
I posted it already (see link above), but there isn’t too much interest in it unfortunately.
Oops, sorry — I assumed that link was to an external explanation or picture of the feature.
Kia ora Bjorn, I am also interested in converting a large WikidPad wiki to Obsidian, thank you for the Python code.
It looks like your WikidPad wiki text files already have the extension .md
Does that mean they were already in Markdown rather than WikidPad native syntax? I can see you convert some WikidPad native constructs so were your files in some hybrid form? And did you run another conversion step from WikidPad native syntax to Markdown?
Hi, No I changed the extension before to be able to visualize as markdown. The formats are sufficiently close to be useful. All the conversion was done in the script as far as I remember.
Hi, I’m new to programming, and when I get an error message, I don’t know what to do. The error message is as follows:
Traceback (most recent call last):
File “”, line 1, in
NameError: name ‘wikidpad’ is not defined
What can I do?
Hi, when do you get this error message? Try to post more context.
Hi, I created the file wikidpad-to-obsidian.py by copy and paste code from Github and put it in Wikidpad directory. Then run it either from cmd or Powershell into the same folder. Something I had to missed (sorry for my english).
I have Python 3.12.10 on Windows 10
The most important thing is to make a copy of your files. The first hurdle is probably that the script expects .md files. I changed this in my wikidpad prior to the conversion.
The script was never polished to be run in one go. I ran each chunk interactively and checked the results. It is probably important to understand what each part of the script does. Can you post more context from the command line?
BTW, the directory needs to be the one with the database files.
Do you mean rename all the .wiki files into .md files?
Yes! copy all your files to a new folder, rename them and process the renamed copies.
Hi!
Thanks for the script. I adapted it for converting my own Wikidpad stuff.
That’s still imperfect but I get a pretty good conversion.
from pathlib import Path
import urllib
import re, os
from urllib.parse import urlparse
from string import punctuation
from tqdm import tqdm
# I used this script in 2023 to convert a wikidpad wiki into a Obsidian.md wiki
# The wikidpad wiki has to be in the "original sqlite" format ie a collection of text files
# The script was run and tweaked and was not run or tested on more than one wiki.
# If you use this, be sure to run this script on a copy of the files as the files are modified
# This code renames the wiki files:
def rename_to_md():
files=Path('.').glob("*.wiki")
print(f'Renaming files')
for file in files:
new_name = file.with_suffix(".md")
file.rename(new_name)
# print(f"Renamed: {file.name} -> {new_name.name}")
# Sometimes wp creates pages with ~ (tilde). Rename these
def replace_tilde(pages):
tildepages = [p for p in pages if "~" in str(p)]
print(f'Replacing tilde in {len(tildepages)} pages')
for tildepage in tildepages:
a, b = str(tildepage).split("~", maxsplit=1)
np = Path(f"{a}.md")
if not np.exists():
tildepage.rename(np)
# Some wp files have unicode and some punctuation quoted.
# subpages are renamed with a pipe | character
def rename_files(pages):
print('Converting')
for page in pages:
# print(page)
uq = urllib.parse.unquote(str(page), encoding='cp1252')
if not str(page) == uq:
nm = uq.replace("/", "|")
page.rename(nm)
print(f"Renamed: {page} -> {nm}")
def read_txt(page):
return page.read_text(encoding='utf-8').lstrip("\ufeff")
# This is not neccessary, but I wanted to remove the title from the page as Obsidian shows the filename by default.
# The code below strips page name from first line and removes the line if only punctuation remains
def strip_title(txt,title):
lines=txt.splitlines()
if len(lines)>0:
firstline, *rest = lines
newfirstline = firstline.strip("+ ").replace(" ","").replace(title, "")
if not set(newfirstline) - set(punctuation):
newfirstline = ""
if firstline != newfirstline:
txt = newfirstline + "\n" + "\n".join(rest)
return txt
# remove [alias:...] and replace with obsidian alias:
# https://help.obsidian.md/Linking+notes+and+files/Aliases
def remove_aliases(txt):
regxal = re.compile(r"(?:\[)alias:(.+)(?:\])")
matchobj = re.search(regxal, txt)
if matchobj:
aliases = matchobj.group(1).strip().split("; ")
nb = f"---\naliases: {', '.join(aliases)}\n---\n\n"
return nb + txt[:matchobj.start()]+txt[matchobj.end():]
return txt
# Replace wikidpad absolute file links with obsidian md style links:
# for example
# file:/home/bjorn/Desktop/mecwiki/yeast-colony-pcr.md
# [file:/home/bjorn/Desktop/mecwiki/yeast-colony-pcr.md]
# Is replaced by
# [yeast-colony-pcr.md](<file:/home/bjorn/Desktop/mecwiki/yeast-colony-pcr.md>)
# This could be improved to handle images by prepending a ! for some links
def replace_links(txt):
regxfl = re.compile(r"([^\]])(?:\[?)(file:/[^\s\]]+)(?:\]|\s)")
def rep(m):
prev= m.group(1)
url = m.group(2)
fn = os.path.basename(url)
return f"{prev}[{fn}](<{url}>)"
return re.sub(regxfl, rep, txt)
def replace_images(txt):
regx = re.compile(r"rel://files/(\S+)")
return re.sub(regx, r"![[\1]]", txt)
def replace_lines(txt,regx,func):
result=''
for line in txt.splitlines():
line= re.sub(regx,func,line)
result+=line+'\n'
return result
def replace_headers(txt):
regx = re.compile(r"^([\s\t]*)(\++)")
def rep(m):
if len(m.groups())<2:
return ''
hcode='#'*len(m.group(2))
return f'{m.group(1)}{hcode} '
return replace_lines(txt,regx,rep)
def replace_itemize(txt):
regx = re.compile(r"^([\s\t]+)(\*+)")
def rep(m):
if len(m.groups())<2:
return ''
level=len(m.group(1))+len(m.group(2))-1
icode=' '*max(0,level-1)
return f'{icode}- '
return replace_lines(txt,regx,rep)
def replace_enumerate(txt):
regx = re.compile(r"^([\s\t]+)([0-9]+)")
def rep(m):
if len(m.groups())<2:
return ''
level=len(m.group(1))
icode=' '*max(0,level-1)
return f'{icode}{m.group(2)}'
return replace_lines(txt,regx,rep)
def replace_bold(txt):
regx=re.compile(r"\*(.+?)\*")
txt,n= re.subn(regx,r"**\1**",txt)
return txt
def replace_eqn(txt):
regx=re.compile(r"\[:eqn:\s*\"((?s:.)*?)\"\]")
txt= re.sub(regx,r"$$\1$$",txt)
return txt
def replace_plugin(txt):
txt= re.sub(r"\[:([A-Za-z]+):\s*\/\/",r"```\1\n",txt)
txt= re.sub(r"\/\/\]",r"\n```",txt)
return txt
def replace_table(txt):
def rep(m):
if len(m.groups())<1:
return ''
lines=m.group(1).splitlines()
lines=[l.rstrip('|').lstrip('|') for l in lines]
lines=['|'+l+'|' for l in lines]
ncol=max([l.count('|') for l in lines])
lines.insert(1,'| - '*(ncol-1)+'|')
return '\n'+'\n'.join(lines)
txt= re.sub(r"<<\|\s*(.*?)>>",rep,txt,flags=re.DOTALL)
return txt
# The code below collects all file names in the wiki and searches each file for these file names.
# it then replaces all *defined* CamelCase and [wikiwords] links in the page by [[CamelCase]] or [[wikiwords]]
# This part of the script might take some time to run and could be tweaked to be more efficient.
# It ended up doing too many links on the pages for my taste, a simple filter for link length or a blacklist might
# help.
def replace_pages(txt):
for preg in pregs:
txt = re.sub(preg, r"[[\1]]\2" , txt)
return txt
def translate_wiki(txt,title):
txt= strip_title(txt,title)
txt= remove_aliases(txt)
txt =replace_pages(txt)
txt= replace_links(txt)
txt=replace_images(txt)
txt= replace_headers(txt)
txt= replace_itemize(txt)
txt= replace_bold(txt)
txt= replace_eqn(txt)
txt= replace_plugin(txt)
txt=replace_table(txt)
return txt
def translate(page):
"""Translate wikidpad file syntax to markdown"""
try:
txt= read_txt(page)
outfile=page.with_suffix(".md")
outfile.write_text(translate_wiki(txt,page.stem),encoding='utf-8')
except Exception as err:
print('Failed to process',page,':',err)
def translate_all():
print('Translating syntax from wikidpad to markdown')
pages = sorted(Path('.').glob('*.wiki'))
for page in tqdm(pages):
translate(page)
pages = sorted(Path('.').glob('*.wiki'))
replace_tilde(pages)
rename_files(pages)
pages = sorted(Path('.').glob('*.wiki'))
pregs = [re.compile(f"\[?({p.stem.strip('[]')})\]?(\W)") for p in pages]
translate_all()
2 Likes
that looks great! Maybe this should be marketed to the WikidPad community.
1 Like
Hi, I’ve tried the two scripts and bot sent me the same error message:
T:\Tecnozona\to_obs.py:204: SyntaxWarning: invalid escape sequence ‘[’
pregs = [re.compile(f"[?({p.stem.strip(‘’)})]?(\W)“) for p in pages]
T:\Tecnozona\to_obs.py:204: SyntaxWarning: invalid escape sequence ‘]’
pregs = [re.compile(f”[?({p.stem.strip(‘’)})]?(\W)") for p in pages]
Traceback (most recent call last):
File “T:\Tecnozona\to_obs.py”, line 6, in
from tqdm import tqdm
ModuleNotFoundError: No module named ‘tqdm’
1 Like
Missing tqdm
install tqdm with “pip install tqdm”
1 Like
Thanks a lot BjornFJohansson and GMT for the script! I have extended the version posted by GMT with a function to replace pre tags like this:
<<pre
preformatted text
>>
from pathlib import Path
import urllib
import re, os
from urllib.parse import urlparse
from string import punctuation
from tqdm import tqdm
# I used this script in 2023 to convert a wikidpad wiki into a Obsidian.md wiki
# The wikidpad wiki has to be in the "original sqlite" format ie a collection of text files
# The script was run and tweaked and was not run or tested on more than one wiki.
# If you use this, be sure to run this script on a copy of the files as the files are modified
# This code renames the wiki files:
def rename_to_md():
files = Path('.').glob("*.wiki")
print(f'Renaming files')
for file in files:
new_name = file.with_suffix(".md")
file.rename(new_name)
# print(f"Renamed: {file.name} -> {new_name.name}")
# Sometimes wp creates pages with ~ (tilde). Rename these
def replace_tilde(pages):
tildepages = [p for p in pages if "~" in str(p)]
print(f'Replacing tilde in {len(tildepages)} pages')
for tildepage in tildepages:
a, b = str(tildepage).split("~", maxsplit=1)
np = Path(f"{a}.md")
if not np.exists():
tildepage.rename(np)
# Some wp files have unicode and some punctuation quoted.
# subpages are renamed with a pipe | character
def rename_files(pages):
print('Converting')
for page in pages:
# print(page)
uq = urllib.parse.unquote(str(page), encoding='cp1252')
if not str(page) == uq:
nm = uq.replace("/", "|")
page.rename(nm)
print(f"Renamed: {page} -> {nm}")
def read_txt(page):
return page.read_text(encoding='utf-8').lstrip("\ufeff")
# This is not neccessary, but I wanted to remove the title from the page as Obsidian shows the filename by default.
# The code below strips page name from first line and removes the line if only punctuation remains
def strip_title(txt, title):
lines = txt.splitlines()
if len(lines) > 0:
firstline, *rest = lines
newfirstline = firstline.strip("+ ").replace(" ", "").replace(title, "")
if not set(newfirstline) - set(punctuation):
newfirstline = ""
if firstline != newfirstline:
txt = newfirstline + "\n" + "\n".join(rest)
return txt
# remove [alias:...] and replace with obsidian alias:
# https://help.obsidian.md/Linking+notes+and+files/Aliases
def remove_aliases(txt):
regxal = re.compile(r"(?:\[)alias:(.+)(?:\])")
matchobj = re.search(regxal, txt)
if matchobj:
aliases = matchobj.group(1).strip().split("; ")
nb = f"---\naliases: {', '.join(aliases)}\n---\n\n"
return nb + txt[:matchobj.start()] + txt[matchobj.end():]
return txt
# Replace wikidpad absolute file links with obsidian md style links:
# for example
# file:/home/bjorn/Desktop/mecwiki/yeast-colony-pcr.md
# [file:/home/bjorn/Desktop/mecwiki/yeast-colony-pcr.md]
# Is replaced by
# [yeast-colony-pcr.md](<file:/home/bjorn/Desktop/mecwiki/yeast-colony-pcr.md>)
# This could be improved to handle images by prepending a ! for some links
def replace_links(txt):
regxfl = re.compile(r"([^\]])(?:\[?)(file:/[^\s\]]+)(?:\]|\s)")
def rep(m):
prev = m.group(1)
url = m.group(2)
fn = os.path.basename(url)
return f"{prev}[{fn}](<{url}>)"
return re.sub(regxfl, rep, txt)
def replace_images(txt):
regx = re.compile(r"rel://files/(\S+)")
return re.sub(regx, r"![[\1]]", txt)
def replace_lines(txt, regx, func):
result = ''
for line in txt.splitlines():
line = re.sub(regx, func, line)
result += line + '\n'
return result
def replace_headers(txt):
regx = re.compile(r"^([\s\t]*)(\++)")
def rep(m):
if len(m.groups()) < 2:
return ''
hcode = '#' * len(m.group(2))
return f'{m.group(1)}{hcode} '
return replace_lines(txt, regx, rep)
def replace_itemize(txt):
regx = re.compile(r"^([\s\t]+)(\*+)")
def rep(m):
if len(m.groups()) < 2:
return ''
level = len(m.group(1)) + len(m.group(2)) - 1
icode = ' ' * max(0, level - 1)
return f'{icode}- '
return replace_lines(txt, regx, rep)
def replace_enumerate(txt):
regx = re.compile(r"^([\s\t]+)([0-9]+)")
def rep(m):
if len(m.groups()) < 2:
return ''
level = len(m.group(1))
icode = ' ' * max(0, level - 1)
return f'{icode}{m.group(2)}'
return replace_lines(txt, regx, rep)
def replace_bold(txt):
regx = re.compile(r"\*(.+?)\*")
txt, n = re.subn(regx, r"**\1**", txt)
return txt
def replace_eqn(txt):
regx = re.compile(r"\[:eqn:\s*\"((?s:.)*?)\"\]")
txt = re.sub(regx, r"$$\1$$", txt)
return txt
def replace_plugin(txt):
txt = re.sub(r"\[:([A-Za-z]+):\s*\/\/", r"```\1\n", txt)
txt = re.sub(r"\/\/\]", r"\n```", txt)
return txt
def replace_pre(txt):
# replaces pre formatted text tags
# must occur at the beginning of a line, with nothing else after it except whitespaces
# m modifier: multi line. Causes ^ and $ to match the begin/end of each line (not only begin/end of string)
txt = re.sub(r"^<<pre\s*$", r"```\n", txt, flags=re.M)
txt = re.sub(r"^>>\s*$", r"\n```", txt, flags=re.M)
return txt
def replace_table(txt):
def rep(m):
if len(m.groups()) < 1:
return ''
lines = m.group(1).splitlines()
lines = [l.rstrip('|').lstrip('|') for l in lines]
lines = ['|' + l + '|' for l in lines]
ncol = max([l.count('|') for l in lines])
lines.insert(1, '| - ' * (ncol - 1) + '|')
return '\n' + '\n'.join(lines)
txt = re.sub(r"<<\|\s*(.*?)>>", rep, txt, flags=re.DOTALL)
return txt
# The code below collects all file names in the wiki and searches each file for these file names.
# it then replaces all *defined* CamelCase and [wikiwords] links in the page by [[CamelCase]] or [[wikiwords]]
# This part of the script might take some time to run and could be tweaked to be more efficient.
# It ended up doing too many links on the pages for my taste, a simple filter for link length or a blacklist might
# help.
def replace_pages(txt):
for preg in pregs:
txt = re.sub(preg, r"[[\1]]\2", txt)
return txt
def translate_wiki(txt, title):
txt = strip_title(txt, title)
txt = remove_aliases(txt)
txt = replace_pages(txt)
txt = replace_links(txt)
txt = replace_images(txt)
txt = replace_headers(txt)
txt = replace_itemize(txt)
txt = replace_bold(txt)
txt = replace_eqn(txt)
txt = replace_plugin(txt)
txt = replace_table(txt)
txt = replace_pre(txt)
return txt
def translate(page):
"""Translate wikidpad file syntax to markdown"""
try:
txt = read_txt(page)
outfile = page.with_suffix(".md")
outfile.write_text(translate_wiki(txt, page.stem), encoding='utf-8')
except Exception as err:
print('Failed to process', page, ':', err)
def translate_all():
print('Translating syntax from wikidpad to markdown')
pages = sorted(Path('.').glob('*.wiki'))
for page in tqdm(pages):
translate(page)
pages = sorted(Path('.').glob('*.wiki'))
replace_tilde(pages)
rename_files(pages)
pages = sorted(Path('.').glob('*.wiki'))
pregs = [re.compile(f"\[?({p.stem.strip('[]')})\]?(\W)") for p in pages]
translate_all()
1 Like