Add API support for reading and writing binary by chunks or streaming

Developers: Plugin & API
1

Currently readBinary function is supported but it’s inefficient to read large file, especially on mobile.

Is it possible for Obsidian devs to add api to support reading and writing files by chunks (“streaming” api)? So that it’s easier for the plugin to deal with large files.

Thanks!

1 Like
2

Is it possible to have the file disassembled and reconstructed as a short term fix until the support for reading by chunks is added?

3

don’t think it’s a valid solution, because the plugin has to call readbinary to load the whole file into memory firstly, which still cannot avoid crashing obsidian especially on mobile

4

may i ask are there any obsidian devs considering this feature? thanks!

5

Hello, Obsidian team! :waving_hand:

I am developing a plugin focused on optimizing work with large files, and I would like to draw your attention to a technical limitation on mobile platforms that significantly complicates the implementation of some useful features. The main problem is that the Obsidian API does not provide functions for streaming files — currently, only the vault.read() and vault.modify() methods are available, which load the entire file contents into memory at once, leading to critical performance issues or even OOM.

On the desktop, plugins have direct access to the Node.js API, such as fs, which allows reading and writing files in chunks without loading the entire content into memory: fs.createReadStream. The mobile version, on the other hand, is based on Capacitor, which does not have access to Node.js, significantly limiting the ability to work with large files, caches, or media where it is most needed.

But since the mobile versions of Obsidian are based on Capacitor, it is technically possible to implement such an API, as Capacitor already has such capabilities through:

Integrating such a mechanism would be logical and harmonious with the current architecture of the mobile platform. This can be implemented as additional API methods that do not change the existing functionality — simply extending the current capabilities without the need to rewrite existing code or break backward compatibility.

6

In my plugin I have to write downloaded files into the vault. Those files can be small or really big.

In the begining I worked with

writeBinary(normalizedPath: string, data: ArrayBuffer, options?: DataWriteOptions): Promise<void>;

This works well for small files but as the data parameter suggets this is not going to work well with large files. Using this function it will load the complete file into memory.

The good thing is that files written like that are immediately available afterwards for use and I don’t have to take care what happens if the write process is interrupted.

Use case or problem

Downloading S3 files such as images, audio and videos from S3 buckets and use them in markdown files.

Proposed solution

  • I suggest introducing a function that writes binary files as streams, closely mirroring the function already in place.

  • If this approach isn’t viable, a deeper dive into Obsidian’s reload process might be required. For some unknown reason, reloading Obsidian doesn’t interrupt active streams.

  • It would be beneficial if there was an event to monitor Obsidian reloads. This would enable plugin developers to appropriately manage and close any open streams. Thankfully, stream closure is already operational when the unload event gets triggered.

Current workaround

Currently, I’ve crafted my own function to write files using streams:

return new Promise((resolve, reject) => {
    const writeStream = createWriteStream(objectPath);
    const cache = this;

    this.addOpenStream(writeStream);

    writeStream.on("finish", function () {
        cache.removeOpenStream(writeStream);
        resolve();
    });
    writeStream.on("error", function () {
        cache.removeOpenStream(writeStream);
        reject();
    });
    stream.on("error", function () {
        cache.removeOpenStream(writeStream);
        reject();
    });
    stream.pipe(writeStream);
});

This effectively writes a readable stream to the specified path. But it has its limitations:

  1. Immediate Availability: Unlike the earlier method, the file isn’t instantly accessible. To mitigate this, I’ve implemented a retry mechanism:
async function getAbstractFileWithRetry(
    path: string,
    retries = 10,
    interval = 100
): Promise<TFile | null> {
    for (let i = 0; i < retries; i++) {
        const file = app.vault.getAbstractFileByPath(path);

        if (file) {
            return file as TFile;
        }

        await new Promise((res) => setTimeout(res, interval));
    }

    return null;
}
  1. Stream Management: Presently, there’s no clear method to manage open streams if Obsidian undergoes a reload. While stream management is functional when the plugin is deactivated and the event onunload is fired and when Obsidian is fully shut down, there’s a loophole. If users reload Obsidian while a stream is active, it results in files being locked in a peculiar state, rendering them inaccessible by Obsidian. These files seem to get protected, requiring administrator permissions to delete. However, fully closing Obsidian restores the files’ accessibility.

By addressing these issues, we can achieve a more seamless and reliable process for writing and accessing files in the plugin.

Link to my plugin in its current state - obsidian-plugin-s3-link/src/cache.ts at master · RagedUnicorn/obsidian-plugin-s3-link · GitHub

1 Like
7

Moved to developers and api.