Usage of the VFS shim as a standalone VFS extension

[georgi-neykov-hub]

@utelle, myself and my colleague have been digging through the project and evaluating it as a possible alternative to SQLCipher and the SQLite's authors' SEE extension. We are searching for a standalone loadable encryption-enabled VFS extension that can be used for both a Swift-based iOS project and a Kotlin/C/C++ Android project.
There are a few questions that we couldn't answer ourselves as we are very far from SQLite programming experts:

• If we forget about compatibility and interoperability between different SQLite encryption libraries, are there any stoppers to omit all the patches applied in script/patchsqlite3.sh amalgamation file and compile the library's sources against an official untouched sqlite.c file?
• Client processes on iOS would already have an initialized SQLite instance, probably loaded as a dynamic library. Can sqlite3mc_initialize() be called manually ?
• What modifications are required to be able to build the VFS as a standalone dynamically linked extension for SQLite?

Thank you

[utelle]

IMHO we would be better off to convert this issue into a discussion.

myself and my colleague have been digging through the project and evaluating it as a possible alternative to SQLCipher and the SQLite's authors' SEE extension.

This project, SQLite3 Multiple Ciphers, is fully compatible with SQLCipher encrypted database files, but supports only a subset of the SQLCipher SQL API.

The SQLite's authors' SEE extension is a closed-source extension. Therefore this project can't provide compatibility with SEE.

We are searching for a standalone loadable encryption-enabled VFS extension that can be used for both a Swift-based iOS project and a Kotlin/C/C++ Android project.

I'm not aware of any loadable encryption-enabled VFS extension implementation. SQLCipher modifies several SQLite source files (thereby reinstating the HAS_CODEC API that was dropped from the original SQLite implementation in early 2020), while the SEE source code is a drop-in replacement for the SQLite source amalgamation.

The author of sqleet claimed in December 2020 that a standalone encryption-enabled VFS implementation would be possible. However, up to now he did not publish any code.

There are a few questions that we couldn't answer ourselves as we are very far from SQLite programming experts:

• If we forget about compatibility and interoperability between different SQLite encryption libraries, are there any stoppers to omit all the patches applied in script/patchsqlite3.sh amalgamation file and compile the library's sources against an official untouched sqlite.c file?

Well, compatibility and interoperability with existing encryption implementations was and is a main goal of SQLite3 Multiple Ciphers.

Even if compatibility and interoperability is not an issue, simply omitting the patches will not work. At least some of the pragma handling code will have to be moved to the VFS implementation. Handling pragmas only in the VFS implementation is problematic, because pragma commands will be passed to the VFS only if a database actually uses the VFS, otherwise VFS-related pragma commands are silently ignored. Therefore it will be difficult to configure the VFS at runtime.

• Client processes on iOS would already have an initialized SQLite instance, probably loaded as a dynamic library. Can sqlite3mc_initialize() be called manually ?

Yes.

• What modifications are required to be able to build the VFS as a standalone dynamically linked extension for SQLite?

Well, the current implementation makes use of several internal SQLite functions and accesses some internal data structures. A standalone extension can access neither internal SQLite functions nor internal data structures. Removing these dependencies will not be an easy task, although it might be possible.

[georgi-neykov-hub]

@utelle thank you for the elaborate answer. Your last argument about the internal models usage made us to completely abandon the idea of a standalone VFS extension. We will be compiling the complete sources plus the patched amalgamation. You can close the issue.

[utelle]

Your last argument about the internal models usage made us to completely abandon the idea of a standalone VFS extension.

As with other encryption solutions (like SEE or SQLCipher) it is currently necessary to replace the SQLite system library.

Of course, a loadable, stand-alone VFS extension with encryption support would be nice. However, that would impose several restrictions - especially regarding compatibility and interoperability.

When I started to work on SQLite3 Multiple Ciphers, I had hoped for a solution that would work with an unmodified version of SQLite. However, to be able to support interoperability patching the SQLite source code was unavoidable, unfortunately.

You can close the issue.

I will convert the issue into a discussion, because the topic may be of interest to others.