Add ability to configure ephemeral storage size (#71)

* Add ability to configure ephemeral storage * Update tests * Update docs * Bump minimum AWS SDK version
aarondfrancis · May 11, 2022 · a630b39 · a630b39
1 parent ccd625f
commit a630b39
Show file tree

Hide file tree

Showing 13 changed files with 130 additions and 22 deletions.
diff --git a/composer.json b/composer.json
@@ -16,7 +16,7 @@
         "illuminate/console": "^7.0|^8.0|^9.0",
         "maennchen/zipstream-php": "^2.1",
         "guzzlehttp/guzzle": "^6.3|^7.2",
-        "aws/aws-sdk-php": "^3.20.0"
+        "aws/aws-sdk-php": "^3.216.1"
     },
     "require-dev": {
         "orchestra/testbench": "^5|^6|^7",

diff --git a/config/sidecar.php b/config/sidecar.php
@@ -30,6 +30,12 @@
      */
     'memory' => env('SIDECAR_MEMORY', 512),
 
+    /*
+     * The default ephemeral storage for your functions, in megabytes.
+     * This can be overridden per function.
+     */
+    'storage' => env('SIDECAR_STORAGE', 512),
+
     /*
      * The default architecture your function runs on.
      * Available options are: x86_64, arm64

diff --git a/docs/configuration.md b/docs/configuration.md
@@ -1,7 +1,7 @@
 
 # Configuring Sidecar
 
-After running `php artisan sidecar:install`, you should have a `sidecar.php` file in your `config` folder. 
+After running `php artisan sidecar:install`, you should have a `sidecar.php` file in your `config` folder.
 
 There are several configuration options in that file, which we'll cover in this section.
 
@@ -17,11 +17,11 @@ To get started, run the following command:
 php artisan sidecar:configure
 ```
 
-The first thing it will do is guide you through creating a new AWS user in the web interface, which it will then use to create everything else it needs. 
+The first thing it will do is guide you through creating a new AWS user in the web interface, which it will then use to create everything else it needs.
 
 Note that this won't start any services, it just creates some policies in IAM.
 
-This is the same general method that Laravel Vapor uses: you provide it with Admin Access and then it configures itself. Sidecar takes it a step further and provides you the option to self-destruct the admin keys once it has configured itself. 
+This is the same general method that Laravel Vapor uses: you provide it with Admin Access and then it configures itself. Sidecar takes it a step further and provides you the option to self-destruct the admin keys once it has configured itself.
 
 If you'd like to manually set everything up, take a look at the command to see exactly what it's doing, and you can recreate it in the IAM portal.
 
@@ -62,15 +62,29 @@ return [
 ];
 ```
 
+## Function Storage
+
+The ephemeral storage can also be customized, and again, if it isn't, the default from your `sidecar.php` file will be used.
+
+```php
+return [
+    /*
+     * The default ephemeral storage for your functions, in megabytes.
+     * This can be overridden per function.
+     */
+    'storage' => env('SIDECAR_STORAGE', 512),
+];
+```
+
 ## Package Base Path
 
-By default, all of your Lambda resources are going to be relative to the `base_path()` of your application. That means when you're defining your code packages, you'll use the root of your application as the starting point. 
+By default, all of your Lambda resources are going to be relative to the `base_path()` of your application. That means when you're defining your code packages, you'll use the root of your application as the starting point.
 
 If all of your Lambda code lives in e.g. `resources/lambda`, then you can update your `package_base_path` to reflect that.
 
 config/sidecar.php {.filename}
 ```php
-return [ 
+return [
     /*
      * The base path for your package files. If you e.g. keep
      * all your Lambda package files in your resource path,
@@ -91,14 +105,14 @@ By default, the environment name that Sidecar uses is your `APP_ENV` from your `
 If you'd like to use something other than the `APP_ENV`, you can do so by providing a `SIDECAR_ENV` environment variable.
 
 ```php
-return [ 
-    /*  
+return [
+    /*
      * Sidecar separates functions by environment. If you'd like to change
      * your Sidecar environment without changing your entire application
      * environment, you may do so here.
      */
     'env' => env('SIDECAR_ENV', env('APP_ENV')),
 ];
 ```
-    
-To learn much more about environments and how to use them, see the [Environments](/environments) section.
+
+To learn much more about environments and how to use them, see the [Environments](/environments) section.
diff --git a/docs/functions/customization.md b/docs/functions/customization.md
@@ -78,6 +78,25 @@ class ExampleFunction extends LambdaFunction
 }
 ```
 
+## Storage
+
+Lambda functions can configure the amount of ephemeral storage available to them in the `/tmp` directory. This storage is shared between function instances, which means it persists across _invocations_ of a single warm Lambda function instance but is cleaned up everywhere else (e.g. between cold starts, when scaling up to more concurrent invocations, etc.).
+
+Sidecar uses the storage value from you `sidecar.php` configuration file, which defaults to 512MB.
+
+You are free to change this per function by returning a value from the `storage` method.
+
+```php
+class ExampleFunction extends LambdaFunction
+{
+    public function storage() // [tl! focus:4]
+    {
+        // 2 GB
+        return 2048;
+    }
+}
+```
+
 ## Layers
 
 Some functions require extra code or data beyond what is in your code package. From [Amazon's documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html):
@@ -139,7 +158,7 @@ class ExampleFunction extends LambdaFunction
 }
 ```
 
-It is **very important** to note that if you allow Sidecar to manage your Lambda's environment variables, any changes made to environment variables in the AWS UI will be overwritten next time you deploy your function. 
+It is **very important** to note that if you allow Sidecar to manage your Lambda's environment variables, any changes made to environment variables in the AWS UI will be overwritten next time you deploy your function.
 
 By default, Sidecar doesn't touch your Lambda's environment at all. Only when you return an array from `variables` will Sidecar take control of the env vars.
 
@@ -162,7 +181,7 @@ class ExampleFunction extends LambdaFunction
 
     public function variables()  // [tl! focus:start]
     {
-        // Values will come from the "activating" machine. 
+        // Values will come from the "activating" machine.
         return [
             'aws_key' => config('services.aws.key'),
             'aws_secret' => config('services.aws.secret'),

diff --git a/src/LambdaFunction.php b/src/LambdaFunction.php
@@ -312,6 +312,16 @@ public function timeout()
         return config('sidecar.timeout');
     }
 
+    /**
+     * The ephemeral storage, in MB, your Lambda function is given.
+     *
+     * @return int
+     */
+    public function storage()
+    {
+        return config('sidecar.storage');
+    }
+
     public function preparePayload($payload)
     {
         return $payload;
@@ -392,6 +402,9 @@ public function toDeploymentArray()
             'Description' => $this->description(),
             'Timeout' => (int)$this->timeout(),
             'MemorySize' => (int)$this->memory(),
+            'EphemeralStorage' => [
+                'Size' => (int)$this->storage(),
+            ],
             'Layers' => $this->layers(),
             'Publish' => true,
             'PackageType' => $this->packageType(),

diff --git a/tests/Unit/DeploymentTest.php b/tests/Unit/DeploymentTest.php
@@ -57,6 +57,9 @@ public function mockCreatingFunction()
             ],
             'Description' => 'test-Description',
             'Timeout' => 'test-Timeout',
+            'EphemeralStorage' => [
+                'Size' => 'test-EphemeralStorage'
+            ],
             'MemorySize' => 'test-MemorySize',
             'Layers' => 'test-Layers',
             'Publish' => 'test-Publish',

diff --git a/tests/Unit/FunctionTest.php b/tests/Unit/FunctionTest.php
@@ -23,16 +23,18 @@ public function app_name_with_a_space_gets_dashed()
     }
 
     /** @test */
-    public function memory_and_timeout_get_cast_to_ints()
+    public function memory_and_timeout_and_storage_get_cast_to_ints()
     {
         config([
             'sidecar.timeout' => '5',
-            'sidecar.memory' => '500'
+            'sidecar.memory' => '500',
+            'sidecar.storage' => '1024'
         ]);
 
         $array = (new EmptyTestFunction)->toDeploymentArray();
 
         $this->assertSame(5, $array['Timeout']);
         $this->assertSame(500, $array['MemorySize']);
+        $this->assertSame(1024, $array['EphemeralStorage']['Size']);
     }
 }
diff --git a/tests/Unit/LambdaClientTest.php b/tests/Unit/LambdaClientTest.php
@@ -172,7 +172,7 @@ public function update_existing_function()
         $this->lambda->shouldReceive('functionExists')
             ->once()
             ->withArgs(function ($f, $checksum) use ($function) {
-                return $f === $function && $checksum === 'b3422109';
+                return $f === $function && $checksum === 'e827998e';
             })
             ->andReturn(false);
 
@@ -183,8 +183,11 @@ public function update_existing_function()
                 'Runtime' => 'test-Runtime',
                 'Role' => 'test-Role',
                 'Handler' => 'test-Handler',
-                'Description' => 'test-Description [b3422109]',
+                'Description' => 'test-Description [e827998e]',
                 'Timeout' => 'test-Timeout',
+                'EphemeralStorage' => [
+                    'Size' => 'test-EphemeralStorage'
+                ],
                 'MemorySize' => 'test-MemorySize',
                 'Layers' => 'test-Layers',
                 'Architectures' => [
@@ -221,9 +224,12 @@ public function update_existing_image_function()
             ->with([
                 'FunctionName' => 'test-FunctionName',
                 'Role' => null,
-                'Description' => 'test-Description [57084773]',
+                'Description' => 'test-Description [ac420e45]',
                 'Timeout' => 300,
                 'MemorySize' => 512,
+                'EphemeralStorage' => [
+                    'Size' => 512
+                ],
                 'Layers' => [],
                 'PackageType' => 'Image',
                 'Architectures' => [
@@ -253,7 +259,7 @@ public function existing_function_unchanged()
         $this->lambda->shouldReceive('functionExists')
             ->once()
             ->withArgs(function ($f, $checksum) use ($function) {
-                return $f === $function && $checksum === 'b3422109';
+                return $f === $function && $checksum === 'e827998e';
             })
             ->andReturn(true);
 

diff --git a/tests/Unit/Support/DeploymentTestFunction.php b/tests/Unit/Support/DeploymentTestFunction.php
@@ -46,6 +46,9 @@ public function toDeploymentArray()
             'Description' => 'test-Description',
             'Timeout' => 'test-Timeout',
             'MemorySize' => 'test-MemorySize',
+            'EphemeralStorage' => [
+                'Size' => 'test-EphemeralStorage'
+            ],
             'Layers' => 'test-Layers',
             'Publish' => 'test-Publish',
             'Architectures' => ['x86_64'],

diff --git a/tests/Unit/Support/Responses/getFunction.json b/tests/Unit/Support/Responses/getFunction.json
@@ -9,6 +9,9 @@
         "Description": "Laravel [local]: Sidecar function `App\\Sidecar\\OgImage`. [ab0f64a5]",
         "Timeout": 300,
         "MemorySize": 512,
+        "EphemeralStorage": {
+            "Size": 512
+        },
         "LastModified": "2021-06-12T14:04:51.802+0000",
         "CodeSha256": "hHe3JC+E9cHvgZjRgf39adwq2yzNOftvN0qA\/1kaukU=",
         "Version": "$LATEST",
@@ -55,4 +58,4 @@
             ]
         }
     }
-}
+}
diff --git a/tests/Unit/Support/Responses/getVersions.json b/tests/Unit/Support/Responses/getVersions.json
@@ -9,6 +9,9 @@
         "Description": "Laravel [local]: Sidecar function `App\\Sidecar\\OgImage`. [ab0f64a5]",
         "Timeout": 300,
         "MemorySize": 512,
+        "EphemeralStorage": {
+            "Size": 512
+        },
         "LastModified": "2021-06-12T14:04:51.802+0000",
         "CodeSha256": "hHe3JC+E9cHvgZjRgf39adwq2yzNOftvN0qA\/1kaukU=",
         "Version": "$LATEST",
@@ -43,6 +46,9 @@
         "Description": "Laravel [local]: Sidecar function `App\\Sidecar\\OgImage`. [10f50c9b]",
         "Timeout": 300,
         "MemorySize": 512,
+        "EphemeralStorage": {
+            "Size": 512
+        },
         "LastModified": "2021-06-05T14:38:42.754+0000",
         "CodeSha256": "UarZwJqR+CAOmrPKvOaPS5qA3N6GYOZYc8Tx4\/CLH1Y=",
         "Version": "72",
@@ -72,6 +78,9 @@
         "Description": "Laravel [local]: Sidecar function `App\\Sidecar\\OgImage`. [de932341]",
         "Timeout": 300,
         "MemorySize": 512,
+        "EphemeralStorage": {
+            "Size": 512
+        },
         "LastModified": "2021-06-05T15:03:49.675+0000",
         "CodeSha256": "\/9Z\/nwfb62iCaiPi8\/l3Mo2PNl104vjRw3toastIcP8=",
         "Version": "73",
@@ -101,6 +110,9 @@
         "Description": "Laravel [local]: Sidecar function `App\\Sidecar\\OgImage`. [a5f20962]",
         "Timeout": 300,
         "MemorySize": 512,
+        "EphemeralStorage": {
+            "Size": 512
+        },
         "LastModified": "2021-06-05T20:37:19.818+0000",
         "CodeSha256": "JEc2\/ziCNUflfAu2nPlZ+iGVmj6u3w7GVNGyh\/Y\/crs=",
         "Version": "74",
@@ -130,6 +142,9 @@
         "Description": "Laravel [local]: Sidecar function `App\\Sidecar\\OgImage`. [1817e47e]",
         "Timeout": 300,
         "MemorySize": 512,
+        "EphemeralStorage": {
+            "Size": 512
+        },
         "LastModified": "2021-06-05T20:54:07.167+0000",
         "CodeSha256": "ei947HIjYFxiUN2kw4a21r8EwlP+ObLqU0c7kbV1Xe4=",
         "Version": "75",
@@ -159,6 +174,9 @@
         "Description": "Laravel [local]: Sidecar function `App\\Sidecar\\OgImage`. [0723af4c]",
         "Timeout": 300,
         "MemorySize": 512,
+        "EphemeralStorage": {
+            "Size": 512
+        },
         "LastModified": "2021-06-08T03:53:35.184+0000",
         "CodeSha256": "hHe3JC+E9cHvgZjRgf39adwq2yzNOftvN0qA\/1kaukU=",
         "Version": "76",
@@ -188,6 +206,9 @@
         "Description": "Laravel [local]: Sidecar function `App\\Sidecar\\OgImage`. [4a081bdf]",
         "Timeout": 300,
         "MemorySize": 512,
+        "EphemeralStorage": {
+            "Size": 512
+        },
         "LastModified": "2021-06-08T04:00:14.674+0000",
         "CodeSha256": "hHe3JC+E9cHvgZjRgf39adwq2yzNOftvN0qA\/1kaukU=",
         "Version": "77",
@@ -217,6 +238,9 @@
         "Description": "Laravel [local]: Sidecar function `App\\Sidecar\\OgImage`. [9fee3746]",
         "Timeout": 300,
         "MemorySize": 512,
+        "EphemeralStorage": {
+            "Size": 512
+        },
         "LastModified": "2021-06-08T04:08:39.862+0000",
         "CodeSha256": "hHe3JC+E9cHvgZjRgf39adwq2yzNOftvN0qA\/1kaukU=",
         "Version": "78",
@@ -251,6 +275,9 @@
         "Description": "Laravel [local]: Sidecar function `App\\Sidecar\\OgImage`. [ab0f64a5]",
         "Timeout": 300,
         "MemorySize": 512,
+        "EphemeralStorage": {
+            "Size": 512
+        },
         "LastModified": "2021-06-11T18:34:24.290+0000",
         "CodeSha256": "hHe3JC+E9cHvgZjRgf39adwq2yzNOftvN0qA\/1kaukU=",
         "Version": "79",
@@ -285,6 +312,9 @@
         "Description": "Laravel [local]: Sidecar function `App\\Sidecar\\OgImage`. [235d843a]",
         "Timeout": 300,
         "MemorySize": 512,
+        "EphemeralStorage": {
+            "Size": 512
+        },
         "LastModified": "2021-06-12T14:03:45.872+0000",
         "CodeSha256": "hHe3JC+E9cHvgZjRgf39adwq2yzNOftvN0qA\/1kaukU=",
         "Version": "80",
@@ -319,6 +349,9 @@
         "Description": "Laravel [local]: Sidecar function `App\\Sidecar\\OgImage`. [ab0f64a5]",
         "Timeout": 300,
         "MemorySize": 512,
+        "EphemeralStorage": {
+            "Size": 512
+        },
         "LastModified": "2021-06-12T14:04:51.802+0000",
         "CodeSha256": "hHe3JC+E9cHvgZjRgf39adwq2yzNOftvN0qA\/1kaukU=",
         "Version": "81",
@@ -343,4 +376,4 @@
         ],
         "PackageType": "Zip"
     }
-]
+]