Problem/Motivation
Especially for files it would be great to have a stream based encryption supported.
Reasons why you want to have it
In order to serve encrypted files you need to decrypt them. Stream based encryption has two advantages:
- You don't need to keep the original data 100% in memory, so you save PHP memory
- You never have the decrypted file 100% in memory, which is more secure
Details on the implementation
Add a flag to encryption methods to indicate that they support streaming
For example file_encrypt
potentially deals with big pieces of data, so we want to suggest streaming based approaches by default. Therefore
we need a flag on the encryption methods to tell people: You are using a less secure encryption method.
Encrypting/decrypting data
Both halite and defuse support streaming:
Halite:
interface StreamInterface
{
public function getPos(): int;
public function getSize(): int;
public function readBytes(int $num, bool $skipTests = false): string;
public function remainingBytes(): int;
public function writeBytes(string $buf, int $num = null): int;
}
Defuse:
public static function encryptResource($inputHandle, $outputHandle, Key $key)
public static function decryptResource($inputHandle, $outputHandle, Key $key)
Given the quite different API, we maybe need to basically merge them together and create a custom interface including both?
Exposing streams from encryption method plugins
IMHO the encryption method plugins should not implement the streaming interface directly but rather have a method which returns a StreamInterface
object, that handles the actual streaming. File_entity could then use that object to decrypt/encrypt its file data.
prospect of the future
Given that each non streaming encryption method can be wrapped in a stream, which just encrypts/decrypts the complete data, file_entity could be implemented using always a streaming implementation.
Remaining tasks
User interface changes
API changes
Data model changes
Comment | File | Size | Author |
---|---|---|---|
#8 | interdiff.txt | 1.69 KB | dawehner |
#8 | 2738647-8.patch | 7.24 KB | dawehner |
#7 | 2738647-6.patch | 5.56 KB | dawehner |
#2 | 2738647-2.patch | 5.64 KB | dawehner |
Comments
Comment #2
dawehnerThis is really just experimentation so far. Let's first discuss things actually.
@todo Update the issue summary so it makes more sense actually
Comment #3
nerdsteinOne other key point here is a test case. We likely need to put out a new version of the Defuse library, since the latest version supports streaming. This is the backend of the Real AES project: https://www.drupal.org/project/real_aes
I would evaluate what the latest Defuse version is as motivation for this new streaming interface.
Can you elaborate more on 'Idea: leverage the stream inside file_entity'? I would be curious more about how we envision this will be used. Would this be a separate encryption profile? Or, we create a completely different API outside of the standard Encrypt Profiles for Streaming Profiles that can be selected where/when appropriate?
Comment #4
dawehnerI'll working on summarizing what we discussed in IRC.
Comment #5
dawehnerComment #6
dawehnerAfter looking into defuse for a while, I think their way is the way to go
Let's come up with the interface which could make sense here:
The simplest implementation for this interface is to actually read in the entire stream, decrypt/encrypt it, and write it back. This is provided in
NoStreaming
and is proven to work, at least on the file_encrypt tests.Integration with external libraries (defuse / halite)
In order for them to integrate into that architecture we would need some way for encryption methods to provide a special streaming class.
A suggestion to do so (not yet in the patch):
stream_class
which can be used to create a streaming implementationNoStreaming
which should always workPotential halite implementation
ParagonIE\Halite\File::encrypt
andParagonIE\Halite\File::decrypt
both provide a way to pass in a resource, so the stream implementation would just hae to be able to extract the actual halite key output of the encryption profile, which should be doable.Potential defuse implementation
Defuse\Crypt\File::encryptResource
andDefuse\Crypt\File:decryptResource
has a similar signature to halite, so again, the only trick we need is to extract the right key object.File_encrypt Implementation
More on https://github.com/d8-contrib-modules/file_encrypt/issues/5
Comment #7
dawehnerHere is some form of simple patch, not yet with the swappability included.
Comment #8
dawehnerHere is a method on the encryption method which allows you to retrieve the stream implementation. This seems to work basically how you expect it to work like.
I guess the next step is to implement it actually for halite or defuse to prove its what we need. Another step would be to encrypt a gigantic file and check how much memory is used in memory for that. Currently the fact that we use a memory stream in
\Drupal\file_encrypt\EncryptStreamWrapper::stream_close
seems to make that bit impossible. Let's see how things turn out in the future ... but this seems to be totally a file_encrypt problem now ... hopefully ...Comment #9
gregglesIs this issue a Major priority?
Thanks for your work on it so far!