I've done this in the past so I know its possible (on windows anyway). It would be nice to be able to load the library from an embedded resource so all libraries can be distributed with a single assembly.
Something along the lines of exporting the library to a specific app/version name into a temporary location, and then load the library from there. As part of the app domain unloading you can remove the library (or leave it).