Problem/Motivation

Especially for files it would be great to have a stream based encryption supported.

Reasons why you want to have it

In order to serve encrypted files you need to decrypt them. Stream based encryption has two advantages:

  1. You don't need to keep the original data 100% in memory, so you save PHP memory
  2. You never have the decrypted file 100% in memory, which is more secure

Details on the implementation

Add a flag to encryption methods to indicate that they support streaming

For example file_encrypt potentially deals with big pieces of data, so we want to suggest streaming based approaches by default. Therefore
we need a flag on the encryption methods to tell people: You are using a less secure encryption method.

Encrypting/decrypting data

Both halite and defuse support streaming:

Halite:

interface StreamInterface
{
    public function getPos(): int;
    public function getSize(): int;
    public function readBytes(int $num, bool $skipTests = false): string;
    public function remainingBytes(): int;
    public function writeBytes(string $buf, int $num = null): int;
}

Defuse:

public static function encryptResource($inputHandle, $outputHandle, Key $key)
public static function decryptResource($inputHandle, $outputHandle, Key $key)

Given the quite different API, we maybe need to basically merge them together and create a custom interface including both?

Exposing streams from encryption method plugins

IMHO the encryption method plugins should not implement the streaming interface directly but rather have a method which returns a StreamInterface object, that handles the actual streaming. File_entity could then use that object to decrypt/encrypt its file data.

prospect of the future

Given that each non streaming encryption method can be wrapped in a stream, which just encrypts/decrypts the complete data, file_entity could be implemented using always a streaming implementation.

Remaining tasks

User interface changes

API changes

Data model changes

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

dawehner created an issue. See original summary.

dawehner’s picture

FileSize
5.64 KB

This is really just experimentation so far. Let's first discuss things actually.

@todo Update the issue summary so it makes more sense actually

nerdstein’s picture

One other key point here is a test case. We likely need to put out a new version of the Defuse library, since the latest version supports streaming. This is the backend of the Real AES project: https://www.drupal.org/project/real_aes

I would evaluate what the latest Defuse version is as motivation for this new streaming interface.

Can you elaborate more on 'Idea: leverage the stream inside file_entity'? I would be curious more about how we envision this will be used. Would this be a separate encryption profile? Or, we create a completely different API outside of the standard Encrypt Profiles for Streaming Profiles that can be selected where/when appropriate?

dawehner’s picture

I'll working on summarizing what we discussed in IRC.

dawehner’s picture

Issue summary: View changes
dawehner’s picture

After looking into defuse for a while, I think their way is the way to go

  • Its as simple as you can get
  • It supports any kind of resource, so both file and simply memory could be used
  • Even remote resources, aka. any kind of stream wrapper is supported
  • By using resources the actual encryption library (defuse) can choose to do things in an actual streaming way

Let's come up with the interface which could make sense here:


interface EncryptStreamInterface {
  public function encrypt($input_handler, $output_handle, EncryptionProfileInterface $encryption_profile);
  public function decrypt($input_handler, $output_handle, EncryptionProfileInterface $encryption_profile);
}

The simplest implementation for this interface is to actually read in the entire stream, decrypt/encrypt it, and write it back. This is provided in NoStreaming and is proven to work, at least on the file_encrypt tests.

Integration with external libraries (defuse / halite)

In order for them to integrate into that architecture we would need some way for encryption methods to provide a special streaming class.
A suggestion to do so (not yet in the patch):

  • Provide a stream_class which can be used to create a streaming implementation
  • When the encryption method doesn't provide one, fall back to NoStreaming which should always work

Potential halite implementation

ParagonIE\Halite\File::encrypt and ParagonIE\Halite\File::decrypt both provide a way to pass in a resource, so the stream implementation would just hae to be able to extract the actual halite key output of the encryption profile, which should be doable.

Potential defuse implementation

Defuse\Crypt\File::encryptResource and Defuse\Crypt\File:decryptResource has a similar signature to halite, so again, the only trick we need is to extract the right key object.

File_encrypt Implementation

More on https://github.com/d8-contrib-modules/file_encrypt/issues/5

dawehner’s picture

FileSize
5.56 KB

Here is some form of simple patch, not yet with the swappability included.

dawehner’s picture

FileSize
7.24 KB
1.69 KB

Here is a method on the encryption method which allows you to retrieve the stream implementation. This seems to work basically how you expect it to work like.

I guess the next step is to implement it actually for halite or defuse to prove its what we need. Another step would be to encrypt a gigantic file and check how much memory is used in memory for that. Currently the fact that we use a memory stream in \Drupal\file_encrypt\EncryptStreamWrapper::stream_close seems to make that bit impossible. Let's see how things turn out in the future ... but this seems to be totally a file_encrypt problem now ... hopefully ...

greggles’s picture

Is this issue a Major priority?

Thanks for your work on it so far!