Self-hosted LiveSync (ex:Obsidian-livesync) released

Share & showcase
,
1

Hi, everyone. I released Self-hosted LiveSync.
It available for PC, Mac, Android and iOS (and should be Linux).
It must be useful for the people who have limitations to using the cloud (e.g. Researcher, Creator, or Developer) to keep NDA or some agreement.
Or writing in many devices and switching frequently and need conflict detection and merging.

obsidian_live_sync_demo

This plugin uses CouchDB as the LiveSync’s backend.
So you can use your own CouchDB or IBM Cloudant instance.

IBM Cloudant setup procedure is written in the repo:

Self-hosted LiveSync:Github Repositiory

Just aimed for two purposes.

  1. Use a self-hosted and full-controlled server.
    This is required to keep some agreement like NDA in some companies and organizations, even it’s encrypted end to end.
  2. Real-time synchronization and conflict detection like OneNote.
    I wanted to reflect modifications to other devices immediately. And near real-time bi-directional editing is also. And in check the diff and merge in Obsidian. Not only per 10 seconds.

So, I omitted some features to realize purposes.

  1. End to End encryption.
  2. Saving File History.
  3. No synchronization of theme, settings, and anything non-vault items.

The official Sync and LiveSync look similar on the surface. but I think it’s not conflicted for core purposes.
But, Let me thank you again for merging this plugin with community plugins. I have pretty big respect for their generous and community-respectful attitudes.

Note: To avoid data corruption, do not use this plugin with any other synchronize solutions. If you want to synchronize to another backend, please sync one by one.

21 Likes
2

Hi there,

Thanks for your work on this plugin.
I am deeply invested in Markdown note-taking (currently using Typora on macOS/Windows, One Markdown on iOS and Syncthing to sync between all platforms), but have been wanting to switch to Obsidian for a more feature-rich experience while still being able to self host. This plugin gives me exactly what i’ve been looking for!

I’ve briefly managed to test it between macOS, iOS and Windows using a Cloudant and a self-hosted CloudDB instance but have encountered a few problems:

  • Sometimes folders don’t seem to sync and I receive the error: “folder name changed:(this operation is not supported)”. The changes are sometimes reflected on other devices, and other times aren’t.
  • Sometimes notes won’t sync if being accessed on multiple devices, as in, if I have a note open on both macOS and iOS but only editing on the iOS device, the changes will not show on the macOS device ever, or vice-versa.
  • I know that you’d mentioned this on the readme already, “Folder deletion handling is not completed.”, but not being able to entirely manage files/folders and have the changes reflect on all devices really impedes on my ability to be able to use this plugin effectively. I eagerly await this feature to be implemented.
  • Sometimes note changes are never synced. I’m aware that a Reddit user was experiencing a similar problem, but rather than only for significant changes, I’m finding that this occurs for all types of changes but only on certain documents. This is probably the most concerning problem for me at the moment, as I cannot guarantee that any changes I make to a file will be retained across my devices at all.
  • There is some misguidance on which URL to use when using Cloudant if following the instructions on the readme. You suggest to copy the “External Endpoint (preferred)” address, but in fact, you should copy the “External Endpoint” address and add the name of the database to the end. E.g., https://xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-bluemix.cloudant.com/nameofdatabasehere/. This should be clarified on the readme.

I appreciate that this plugin is currently in the early stages of development, but I am extremely interested in it and more than willing to help it reach a level of stability.

Unfortunately, I’m a medical student with non-existant programming experience so wouldn’t be able to assist directly with development, but if there’s anything you’d like for me to test, etc, please do let me know. I’m contactable by email, Discord or Element.

3 Likes
3

Thank you for using this plugin!
I’m really happy with Obsidian too, and I want to use in it my office, I made this.
I’m very glad to hear your report!

I’ll answer one by one.

Folder problems:
Since 0.1.9, it should be improved. but it’s not perfect.
Renaming the folder affects all contained files. so the plugin has to sweep files that delete files that are in previous locations and add new files that are in new locations.
BTW, It could be more improved. I’ve noticed an idea by writing this.

Won’t sync:
It may be caused by conflict detection and auto-resolving.
Could you copy the log and send it to me?

Never synced:
I’m sorry so much. But it’s mysterious. May I ask the path of the file? (from vault root)
If you have any non-ASCII characters in filenames, perhaps it has been fixed in 0.1.9.

Misguidance:
Yes, It should be needed.
At almost the bottom of the readme, written without any note.
It will be improved! I will add some notes where you pointed.

No, you assisted directly now. Reporting is the source of improvement. Let me thank you once again.

Feel free to contact me in any tool. Simply making an issue on GitHub is the easiest way. Twitter’s DM is also responsive, but I’m tweeting in Japanese, maybe bored.
But even in any tool, anything is welcome.

4

Thank you for so quickly taking on the feedback and making adjustments to the plugin. I’ve given the 0.1.14 release some testing today. Below are the problems I have found:

  • Everything was working fine until I decided to sync a large MP3 attachment of 12.5mb. The file wasn’t able to sync. The first problem that occurred following the attempt to sync was that the live syncing aspect of the plugin stopped working. Manually replicating the database worked initially for a few moments, but that eventually also stopped working. Then, if I attempted to sync the database at all, I received a “Replication error” message. Unfortunately, I couldn’t seem to copy from the Sync Status log to send it to you, but I did see an endless “->replication paused, ->Replication activated” loop. The only fix to this problem was to delete the Cloudant database and create a new one. After creating a new Cloudant database, I decided to sync a smaller MP3 file instead but this led me to the same replication loop problem. I find this quite odd as I’d synced several MP3 attachments prior to trying the large 12.5mb file with no problems, so I’m not sure what triggered the error.
  • Sometimes if a new file is created on one device, syncing on another device presents a “could not write to local (newfile:plain) new file.md” error message, despite the file syncing fine with no problems.
  • If a new folder is created on one device, the change does not appear on other devices unless a file is inserted within the new folder. This means it is impossible to create empty folders that sync between devices.

As for the problems in my previous comment, these seem to have been resolved for the most part as of my recent testing which is great to see. Good work!

I was also wondering if it might be possible for you to add instructions on setting up a self-hosted CouchDB server, accessible from the internet for use with the plugin, to the readme?
A lot of the documentation regarding this found on Stack Overflow, etc, refers to much older releases of CouchDB that isn’t very relevant.
That would be great for those of us trying to avoid the 1GB storage cap on the Cloudant Lite tier or wanting to keeping our files stored on our own devices!

1 Like
5

I’m sorry that the update notification is late.

Attachment problems:
Large file treatment has been improved, and I’ve tested in 40MB of mp4 or 300MB PDF. perhaps I’d overlooked something. I’ll investigate it more deeper.

Error message on synchronization:
“(newfile:plain-…” is caused by the plugin failed to write a note.
Self-hosted LiveSync synchronizes the file by redoing minor changes.
But if there are missing pieces in intermediate changesets, the message is shown temporarily. (Intermediate changeset often shrank for faster synchronization when you leave obsidian open.)
If subsequent redo is applied successfully, it doesn’t matter (as you see), but while applying changes, we can’t peek what will subsequent.

But, I forgot to log the errors. this would be improved in the next release.

Added: This happened by message bug too, thay you said! Really thank you! Fixed in 0.1.15!

Empty Folder:
Oh, I completely forgot to write notes of empty folder treatment.
Yes, Self-hosted LiveSync syncs only files, not to folder.
So, As you said, an empty folder would not be synced.
And when the folder becomes empty, the folder will be deleted.
It’s a large limitation, I have to write it into readme. Thank you for pointing it out.

totally:
I’m glad to hear that other issues have been solved! Thank you for your patient and gentleness.

About CouchDB’s docs:
Yes, you are right. And using Self-hosted CouchDB is preferred.
And, as you saw, everybody forgets CouchDB, even though this is a great database.
Pasted image 20211111100750

I think too, the setup docs of CouchDB could be very useful.
So, I will write it without touching about web-server.

I think topics are almost in below:

  • Self-hosting limitations.
  • Setup CouchDB ( bare? / dockerimage? )
  • Preferred configurations
  • Expose to the internet by ngrok for testing.

And, I’d like to write the topic about below:

  • Expose to the internet by own domain and the setup the reverse proxy with SSL.

Not so hard technically. But, as Security Specialist, it’s so heavy topic.
So, I begin with the first four topics.

6

Documentation on the five topics you’d mentioned would be perfect!

Regarding the empty folder syncing, is this a limitation that can be overcome?

I look forward to testing your next release.

7

This sounds very promising! Thank you.

Will this still sync to “flat” local .md files on a desktop OS client?
And will edits made to those files outside of Obsidian be picked up? Of course, given that the Obsidian app is running?

A docker container would allow to easily set it up on a Synology NAS for SMBs, freelancers (with sensitive data or under NDAs), or home users.

Thanks a lot.

9

I released a new version of the plugin before encouraging guys to host their own servers.
End-to-end encryption is implemented to prevent effects by misconfigurations and/or vulnerability of the server software.
(Of course, including many fixes.)
It’s still unstable so I eating myself now. But the feature is released.
So I’m ready to write documentation.

Added: it’s able to disable deleting empty folder. It will just be ignored. I’ll make a new setting.

10

Yes, the plugin syncs every file that obsidian detected in the vault as is.
I didn’t check editing by other software while running obsidian, but if your obsidian reloads the file, it will be synced.
Honestly, I’m not getting to be true friends with Docker yet, I used that to my test server first last month.
But actually, easy for use impressively. I have to learn.

11

it’s amazing!
really need this !

1 Like
12

thank you for your great work !

1 Like
13

On testing 0.1.16 with E2EE enabled, I’m finding that the plugin still gets stuck in a replication loop. This leads to very high CPU usage (20-30% on a Ryzen 5 3600) at idle with fluctuating memory usage.

Here is the log:
11/17/2021, 7:02:02 PM->loading plugin
11/17/2021, 7:02:02 PM->Open Database…
11/17/2021, 7:02:02 PM->Database Info
11/17/2021, 7:02:03 PM->Open Database…
11/17/2021, 7:02:03 PM->Database Info
11/17/2021, 7:02:03 PM->Initialize and checking database files
11/17/2021, 7:02:03 PM->Updating database by new files
11/17/2021, 7:02:03 PM->UPDATE DATABASE
11/17/2021, 7:02:03 PM->UPDATE STORAGE
11/17/2021, 7:02:03 PM->CHECK FILE STATUS
11/17/2021, 7:02:03 PM->Initialized,NOW TRACKING!
11/17/2021, 7:02:03 PM->Pull before replicate.
11/17/2021, 7:02:04 PM->Replication pull completed.
11/17/2021, 7:02:04 PM->Replication activated
11/17/2021, 7:02:05 PM->Replication activated
11/17/2021, 7:02:06 PM->Replication activated
11/17/2021, 7:02:07 PM->Replication activated
11/17/2021, 7:02:09 PM->Replication activated
11/17/2021, 7:02:12 PM->Replication activated
11/17/2021, 7:02:13 PM->Replication activated
11/17/2021, 7:02:15 PM->Replication activated

Disabling E2EE led to the plugin becoming stuck in “NOW TRACKING!”.

14

Sorry for late, thank you for testing!
Could you enable verbose log?
It’s little mysterious that only ‘replication activated’ shown so many times.
(if really activated, should ‘paused’ is shown too.)
Could you make an issue on GitHub?

15

I’ve tried to replicate the problem with a new database, but everything seems to be working fine with 0.1.17.
I’ll keep on testing and if I run into any other issues, I’ll make sure to retain verbose logs and create issues on Git.

1 Like
16

I’m very happy and relieved to hear that!
Thank you for your patience and waiting for improvements.

Yes, please! and of cource, and if yor are not sure that is the bug, please feel free to contact me.

17

At the moment the plugin relies on CouchDB to sync files. Would it be easier taking an approach similar to Syncthing, where the plugin is able to sync raw .md files and folders to a remote computer directly?
Taking this approach would mean the vault can remain intact without modifications during transport.
Syncthing also uses UPnP which would make setup for those wanting to host on their own hardware for use over the internet much easier than setting up a CouchDB server.
This would prevent the file size limitations and other troubles that come with using databases.

2 Likes
18

Very long.

It may lose some features

  1. LIveSync
  2. Conflict detection and resolution.
    When using CouchDB, We can handle the tree of changeset in the replication, and resolving even in off the line.
  3. Dedupe chunks and keep low transfer amount.

And, of course, we have the limitation of JavaScript. It’s the plugin.

I tried WebRTC for peer-to-peer synchronization but some network denied.

HTTP is the simple and most penetrating protocol.

And I tried to make a plugin that as you said. As the complementary of Official “Sync” and Self-hosted LiveSync. Cloud Backup.
It would be very useful. I thought too.

But there are a few limitations, I realized it’s nonrealistic or may not be simple so much.

OneDrive, GoogleDrive, or other any drive may not accept the connections from obsidian by CORS limitations. If you want to access these hosts, need a proxy server to modify the request. To be secured, require
users to host the server.

A self-hosted WebDAV server can use this without request-modifying-proxies. But still, self-hosting is required.

And, currently, mobile Obsidian has the limitations that it can’t access the non-secure endpoint or self-signed endpoint. So self hosted server should be exposed globally. It’s should not be done in so easy and optimistic attitudes.

So, If I write the server software? It will be work well to synchronize everything.
But very hard to prove there’s nothing vulnerablities.

And, it may be harder than launching docker images of CouchDB and traefik(for reverse proxy).I writing this docs.

1 Like
19

I see, thank you for your thorough answer! I hadn’t realised there were so many limitations to Obsidian plugins themselves.

It seems like you made the right choice to use CouchDB.

1 Like
20

Hi vorotamoroz

Thanks for your creative plugin, I do need this for hosting notes on my own server.

I managed to setup the couchdb server with HTTPS, setup the local plugin and forward all notes from laptop to the server, I then setup the mobile app from scratch, the mobile app could download notes from the server.

But the rare thing is I cannot manage them to sync with each other, the notes I modified on laptop or mobile never forward to another peer.

I’m not quite familiar with couchdb, is there a way to check the latest changes on the server? So I may know if the changes are properly pushed to the server or not. I disable the P2P encryption and enable the verbose log on each peer, but there is not much information shown in the log. Or is there a way to check the local database on the laptop or mobile?

Thanks in advance.

2 Likes
21

I cant type the URI, username, password, etc under Android version.

1 Like