WebAssembly build of LevelDB

[puzrin]

It would be nice to have webassembly build of leveldown. There are many cases when platform independent build is more preferable than top performance.

Example: wrapped Google's woff2 https://github.com/fontello/wawoff2. ~ 2x slower than native, but no need to recompile, download and so on. Convenient.

[vweevers]

FWIW you might be better off with one of the many compatible implementations, following the same interface as leveldown. As for compiling: leveldown has prebuilt binaries for linux, windows, osx and electron. If your platform isn't supported, please do tell us.

Of course a WebAssembly build would be fun, don't let me stop you!

[mafintosh]

how would a webassembly build deal with all the file io happening in leveldb?

[vweevers]

Emscripten can simulate a file system using IndexedDB (source)

🤣

[vweevers]

Revisiting this thread and reading my comment above, I now realize it may seem I was dismissing the idea, apologies.

Given that:

1. Chrome is gonna be removing snapshots from IndexedDB in favor of blocking readwrite-transactions (which negatively affects the performance of level-js, concurrent streams in particular)
2. This is how other engines already work (Chrome is the only engine that snapshots)
3. Emscripten does have a file system based on File and Blob (only in workers though)

I think it's worth our time to (at the very least) explore a WebAssembly build of LevelDB.

[vweevers]

Emscripten does have a file system based on File and Blob

Scratch that:

This file system provides read-only access to File and Blob objects.

So in browsers, IndexedDB is still the only option. Ugh.

[puzrin]

When i created this issue, i forgot about fs IO. Intial idea was to have portable lib with almost zero cost. Compiling to WASM does not seems as easy as expected. Probably this can be closed until far future...

[vweevers]

@puzrin In case you opened the issue with Node.js in mind, Emscripten has NODEFS, which:

lets a program in node map directories (via a mount operation) on the host filesystem to directories in Emscripten’s virtual filesystem. It uses node’s synchronous FS API to immediately persist any data written to the Emscripten file system to your local disk.

It being synchronous is problematic.

As for browsers, I think we need to push browser makers a little bit. IndexedDB is going in the wrong direction IMO and no alternative is in the works AFAIK (except for kv-storage which is a Web API built on top of IndexedDB, inheriting its flaws).

[puzrin]

I don't know if port to browser needed at all. Even if needed, IMHO performance is not required there and any FS backend could be ok. My intent was to get fast enougth universal alternative for node.js only (slowdown 2x may be acceptable price).

But as i said, idea was to get all this sweets with almost zero effort (as was done with woff2). Now i understand, things are not so easy for this package. Seems not all of my ideas are clever :)

[vweevers]

I don't know if port to browser needed at all. Even if needed, IMHO performance is not required there and any FS backend could be ok.

The problem with IDB is not the performance of an individual read or write. Besides feature bloat and hand-holding, the basic semantics of transaction lifetime are broken. This affects overall performance of an application, esp. one with long-running concurrent reads and writes. It's strange that backpressure is not a first-class citizen (in a runtime environment that has a UI thread!), or even recognized by spec and engine authors as a real need. I wouldn't mind if IDB stripped 90% of its features, in favor of a focused core with solid primitives. Lower level is better.

My intent was to get fast enougth universal alternative for node.js only (slowdown 2x may be acceptable price). [..] Now i understand, things are not so easy for this package.

Roger. I think that settles the node-related discussion, for now at least. Thanks for opening this issue!

The browser-related discussion is out of scope for leveldown, so I'll move this elsewhere.

[tracker1]

I know it's a little old... was literally thinking that this would be really cool combined with node, in order to support "any" platform from a single build... but that the storage primitives would need to be loaded somehow in a compatible/secure way. I don't think this is close to standardized yet though.

[vweevers]

IMO we've come pretty far already with level@5, which ships prebuilt binaries for linux, mac, windows, arm, android, and it works in browsers. It means most of the time you don't need to build.

I know it's a little old

I'd say it's young ;)