Thank you for using this plugin!
I’m really happy with Obsidian too, and I want to use in it my office, I made this.
I’m very glad to hear your report!

I’ll answer one by one.

Folder problems:
Since 0.1.9, it should be improved. but it’s not perfect.
Renaming the folder affects all contained files. so the plugin has to sweep files that delete files that are in previous locations and add new files that are in new locations.
BTW, It could be more improved. I’ve noticed an idea by writing this.

Won’t sync:
It may be caused by conflict detection and auto-resolving.
Could you copy the log and send it to me?

Never synced:
I’m sorry so much. But it’s mysterious. May I ask the path of the file? (from vault root)
If you have any non-ASCII characters in filenames, perhaps it has been fixed in 0.1.9.

Misguidance:
Yes, It should be needed.
At almost the bottom of the readme, written without any note.
It will be improved! I will add some notes where you pointed.

No, you assisted directly now. Reporting is the source of improvement. Let me thank you once again.

Feel free to contact me in any tool. Simply making an issue on GitHub is the easiest way. Twitter’s DM is also responsive, but I’m tweeting in Japanese, maybe bored.
But even in any tool, anything is welcome.

Thank you for so quickly taking on the feedback and making adjustments to the plugin. I’ve given the 0.1.14 release some testing today. Below are the problems I have found:

  • Everything was working fine until I decided to sync a large MP3 attachment of 12.5mb. The file wasn’t able to sync. The first problem that occurred following the attempt to sync was that the live syncing aspect of the plugin stopped working. Manually replicating the database worked initially for a few moments, but that eventually also stopped working. Then, if I attempted to sync the database at all, I received a “Replication error” message. Unfortunately, I couldn’t seem to copy from the Sync Status log to send it to you, but I did see an endless “->replication paused, ->Replication activated” loop. The only fix to this problem was to delete the Cloudant database and create a new one. After creating a new Cloudant database, I decided to sync a smaller MP3 file instead but this led me to the same replication loop problem. I find this quite odd as I’d synced several MP3 attachments prior to trying the large 12.5mb file with no problems, so I’m not sure what triggered the error.
  • Sometimes if a new file is created on one device, syncing on another device presents a “could not write to local (newfile:plain) new file.md” error message, despite the file syncing fine with no problems.
  • If a new folder is created on one device, the change does not appear on other devices unless a file is inserted within the new folder. This means it is impossible to create empty folders that sync between devices.

As for the problems in my previous comment, these seem to have been resolved for the most part as of my recent testing which is great to see. Good work!

I was also wondering if it might be possible for you to add instructions on setting up a self-hosted CouchDB server, accessible from the internet for use with the plugin, to the readme?
A lot of the documentation regarding this found on Stack Overflow, etc, refers to much older releases of CouchDB that isn’t very relevant.
That would be great for those of us trying to avoid the 1GB storage cap on the Cloudant Lite tier or wanting to keeping our files stored on our own devices!

1 Like

I’m sorry that the update notification is late.

Attachment problems:
Large file treatment has been improved, and I’ve tested in 40MB of mp4 or 300MB PDF. perhaps I’d overlooked something. I’ll investigate it more deeper.

Error message on synchronization:
“(newfile:plain-…” is caused by the plugin failed to write a note.
Self-hosted LiveSync synchronizes the file by redoing minor changes.
But if there are missing pieces in intermediate changesets, the message is shown temporarily. (Intermediate changeset often shrank for faster synchronization when you leave obsidian open.)
If subsequent redo is applied successfully, it doesn’t matter (as you see), but while applying changes, we can’t peek what will subsequent.

But, I forgot to log the errors. this would be improved in the next release.

Added: This happened by message bug too, thay you said! Really thank you! Fixed in 0.1.15!

Empty Folder:
Oh, I completely forgot to write notes of empty folder treatment.
Yes, Self-hosted LiveSync syncs only files, not to folder.
So, As you said, an empty folder would not be synced.
And when the folder becomes empty, the folder will be deleted.
It’s a large limitation, I have to write it into readme. Thank you for pointing it out.

totally:
I’m glad to hear that other issues have been solved! Thank you for your patient and gentleness.

About CouchDB’s docs:
Yes, you are right. And using Self-hosted CouchDB is preferred.
And, as you saw, everybody forgets CouchDB, even though this is a great database.
Pasted image 20211111100750

I think too, the setup docs of CouchDB could be very useful.
So, I will write it without touching about web-server.

I think topics are almost in below:

  • Self-hosting limitations.
  • Setup CouchDB ( bare? / dockerimage? )
  • Preferred configurations
  • Expose to the internet by ngrok for testing.

And, I’d like to write the topic about below:

  • Expose to the internet by own domain and the setup the reverse proxy with SSL.

Not so hard technically. But, as Security Specialist, it’s so heavy topic.
So, I begin with the first four topics.

Documentation on the five topics you’d mentioned would be perfect!

Regarding the empty folder syncing, is this a limitation that can be overcome?

I look forward to testing your next release.

This sounds very promising! Thank you.

Will this still sync to “flat” local .md files on a desktop OS client?
And will edits made to those files outside of Obsidian be picked up? Of course, given that the Obsidian app is running?

A docker container would allow to easily set it up on a Synology NAS for SMBs, freelancers (with sensitive data or under NDAs), or home users.

Thanks a lot.

I released a new version of the plugin before encouraging guys to host their own servers.
End-to-end encryption is implemented to prevent effects by misconfigurations and/or vulnerability of the server software.
(Of course, including many fixes.)
It’s still unstable so I eating myself now. But the feature is released.
So I’m ready to write documentation.

Added: it’s able to disable deleting empty folder. It will just be ignored. I’ll make a new setting.

Yes, the plugin syncs every file that obsidian detected in the vault as is.
I didn’t check editing by other software while running obsidian, but if your obsidian reloads the file, it will be synced.
Honestly, I’m not getting to be true friends with Docker yet, I used that to my test server first last month.
But actually, easy for use impressively. I have to learn.

it’s amazing!
really need this !

1 Like

thank you for your great work !

1 Like

On testing 0.1.16 with E2EE enabled, I’m finding that the plugin still gets stuck in a replication loop. This leads to very high CPU usage (20-30% on a Ryzen 5 3600) at idle with fluctuating memory usage.

Here is the log:
11/17/2021, 7:02:02 PM->loading plugin
11/17/2021, 7:02:02 PM->Open Database…
11/17/2021, 7:02:02 PM->Database Info
11/17/2021, 7:02:03 PM->Open Database…
11/17/2021, 7:02:03 PM->Database Info
11/17/2021, 7:02:03 PM->Initialize and checking database files
11/17/2021, 7:02:03 PM->Updating database by new files
11/17/2021, 7:02:03 PM->UPDATE DATABASE
11/17/2021, 7:02:03 PM->UPDATE STORAGE
11/17/2021, 7:02:03 PM->CHECK FILE STATUS
11/17/2021, 7:02:03 PM->Initialized,NOW TRACKING!
11/17/2021, 7:02:03 PM->Pull before replicate.
11/17/2021, 7:02:04 PM->Replication pull completed.
11/17/2021, 7:02:04 PM->Replication activated
11/17/2021, 7:02:05 PM->Replication activated
11/17/2021, 7:02:06 PM->Replication activated
11/17/2021, 7:02:07 PM->Replication activated
11/17/2021, 7:02:09 PM->Replication activated
11/17/2021, 7:02:12 PM->Replication activated
11/17/2021, 7:02:13 PM->Replication activated
11/17/2021, 7:02:15 PM->Replication activated

Disabling E2EE led to the plugin becoming stuck in “NOW TRACKING!”.

Sorry for late, thank you for testing!
Could you enable verbose log?
It’s little mysterious that only ‘replication activated’ shown so many times.
(if really activated, should ‘paused’ is shown too.)
Could you make an issue on GitHub?

I’ve tried to replicate the problem with a new database, but everything seems to be working fine with 0.1.17.
I’ll keep on testing and if I run into any other issues, I’ll make sure to retain verbose logs and create issues on Git.

1 Like

I’m very happy and relieved to hear that!
Thank you for your patience and waiting for improvements.

Yes, please! and of cource, and if yor are not sure that is the bug, please feel free to contact me.

At the moment the plugin relies on CouchDB to sync files. Would it be easier taking an approach similar to Syncthing, where the plugin is able to sync raw .md files and folders to a remote computer directly?
Taking this approach would mean the vault can remain intact without modifications during transport.
Syncthing also uses UPnP which would make setup for those wanting to host on their own hardware for use over the internet much easier than setting up a CouchDB server.
This would prevent the file size limitations and other troubles that come with using databases.

1 Like

Very long.

It may lose some features

  1. LIveSync
  2. Conflict detection and resolution.
    When using CouchDB, We can handle the tree of changeset in the replication, and resolving even in off the line.
  3. Dedupe chunks and keep low transfer amount.

And, of course, we have the limitation of JavaScript. It’s the plugin.

I tried WebRTC for peer-to-peer synchronization but some network denied.

HTTP is the simple and most penetrating protocol.

And I tried to make a plugin that as you said. As the complementary of Official “Sync” and Self-hosted LiveSync. Cloud Backup.
It would be very useful. I thought too.

But there are a few limitations, I realized it’s nonrealistic or may not be simple so much.

OneDrive, GoogleDrive, or other any drive may not accept the connections from obsidian by CORS limitations. If you want to access these hosts, need a proxy server to modify the request. To be secured, require
users to host the server.

A self-hosted WebDAV server can use this without request-modifying-proxies. But still, self-hosting is required.

And, currently, mobile Obsidian has the limitations that it can’t access the non-secure endpoint or self-signed endpoint. So self hosted server should be exposed globally. It’s should not be done in so easy and optimistic attitudes.

So, If I write the server software? It will be work well to synchronize everything.
But very hard to prove there’s nothing vulnerablities.

And, it may be harder than launching docker images of CouchDB and traefik(for reverse proxy).I writing this docs.

1 Like

I see, thank you for your thorough answer! I hadn’t realised there were so many limitations to Obsidian plugins themselves.

It seems like you made the right choice to use CouchDB.

1 Like

Hi vorotamoroz

Thanks for your creative plugin, I do need this for hosting notes on my own server.

I managed to setup the couchdb server with HTTPS, setup the local plugin and forward all notes from laptop to the server, I then setup the mobile app from scratch, the mobile app could download notes from the server.

But the rare thing is I cannot manage them to sync with each other, the notes I modified on laptop or mobile never forward to another peer.

I’m not quite familiar with couchdb, is there a way to check the latest changes on the server? So I may know if the changes are properly pushed to the server or not. I disable the P2P encryption and enable the verbose log on each peer, but there is not much information shown in the log. Or is there a way to check the local database on the laptop or mobile?

Thanks in advance.

1 Like

I cant type the URI, username, password, etc under Android version.

1 Like

Thank you for using this plugin!
But it is something mysterious.
Your vaults have been synced once, but no more sync again…

Please check the remote URI and databases are exactly the same once again?

The latest change sequence no is shown in the sync log, if you enable verbose log. update_seq is so.

Have you enabled any sync?

If any sync is enabled, it could not be able to
change.