feat(db): allow external sqlite blobs

[alexdachin]

Currently Trilium stores the contents of a File note type inside the SQLite database. While this is extremely nice for simplicity and portability, it can become problematic once you start uploading many reference files (for example some large quality images, large PDFs, videos etc). While in theory SQLite supports huge databases, storing many files in a single database:

• makes the database file harder to backup
• increases database lock times
• increases the database corruption risk slightly

I propose storing big blobs in the filesystem and storing a reference to them in the database. The files are stored in the data directory to begin with for simplicity, but it also opens the gate for storing them in other places in the future (like s3 compatible storage systems). This keeps the SQLite database small and performant, enables faster incremental backups (like rsync), reduces the risk of database corruption and still allows Trilium to manage things like note hierarchy with cloning.

In order to not disturb the workflow of other people, this feature is disabled by default (so even large blobs are still stored internally in the database). To enable it you need to set TRILIUM_EXTERNAL_BLOB_STORAGE environment variable.

The threshold of storing notes externally vs internally is 100kb by default but can be changed with TRILIUM_EXTERNAL_BLOB_THRESHOLD environment variable. This way small blobs are read from the database more efficiently compared to storing all blobs in the filesystem.

#6546

[alexdachin]

I tried inlining some of these AbstractBeccaEntity in the 0233__migrate_geo_map_to_collection.ts migration, but some things are event based and the subscribers are using AbstractBeccaEntity as well.

The alternative would be to add a warning that people have to upgrade to the the previous Trilium version first and wait for migrations to complete, but I wanted to avoid that

[perfectra1n]

Can you explain more as to what it’s doing, how it works, and what it’d be useful for? Looks cool at first glance! :)

[alexdachin]

Can you explain more as to what it’s doing, how it works, and what it’d be useful for? Looks cool at first glance! :)

Thanks, I updated the PR description.

There is some more context in the issue I linked in the description as well, but I tried to summarize everything here as well.

[eliandoran]

As a start, the implementation seems pretty good.

There is a bug when importing large files as code. Try importing this zip with external storage enabled and it will result in a 9 MB JSON file, but when accessing it it's empty.

Trace-20250708T195034.json.zip

[adoriandoran]

I have noticed an increased risk of BLOB key collisions when the server is running on Windows. The BLOB key is case-sensitive, whereas the underlying file system is case-insensitive.

[adoriandoran]

Once the data directory is relocated, the external BLOB paths break, since they are relative to Trilium’s directory. I suggest storing these paths in the database as relative to the “external-blobs” directory instead.

[alexdachin]

I see, thank you all for the feedback! I'm away for a few days, but I'll have a look as soon as I get back

[alexdachin]

Thank you again for checking this out!

I found out what the issue was with the large json file and fixed it.

Also, good points regarding case insensitive file systems, I switched to a random uuid instead and stored the relative path.

Do you mind checking it one more time? 🙏