Update Lambda Function Runtime Configuration with sigs.k8s.io/controller-runtime/pkg/controller/controllerutil Fails to Persist

[kizitonzeka]

Describe the bug
Updating lambda function runtime configuration through a custom controller that reconciles Functions k8s resource using sigs.k8s.io/controller-runtime/pkg/controller/controllerutil CreateOrUpdate method fails to persist updated runtime configuration on AWS.

I am able to see from logs the custom controller reconciles updated runtime as shown below:

manager {"level":"info","ts":"2025-01-08T11:07:32Z","msg":"Starting reconciliation for SampleCustomResource resource","controller":"SampleCustomResource","controllerGroup":"log.example.net","controllerKind":"SampleCustomResource","SampleCustomResource":{"name":"resource-name","namespace":"custom-resource-namespace"},"namespace":"custom-resource-namespace","name":"resource-name","reconcileID":"4f81143c-d965-4854-ab39-6cfdd33a2e54","name":"custom-resource-namespace"}
manager {"level":"info","ts":"2025-01-08T11:08:01Z","msg":"ACK Resources Created/Updated","controller":"SampleCustomResource","controllerGroup":"log.example.net","controllerKind":"SampleCustomResource","SampleCustomResource":{"name":"resource-name","namespace":"custom-resource-namespace"},"namespace":"custom-resource-namespace","name":"resource-name","reconcileID":"4f81143c-d965-4854-ab39-6cfdd33a2e54","ackResources":[
{"ACKResource":{"kind":"Function","apiVersion":"lambda.services.k8s.aws/v1alpha1","metadata":{"name":"log-transformer-lambda","namespace":"custom-resource-namespace","uid":"df846e9f-e976-49e8-aa28-3d3e74634c7f","resourceVersion":"296530","generation":75,"creationTimestamp":"2024-12-20T19:50:30Z","annotations":{"services.k8s.aws/region":"us-west-2"},"finalizers":["finalizers.lambda.services.k8s.aws/Function"],"managedFields":[{"manager":"controller","operation":"Update","apiVersion":"lambda.services.k8s.aws/v1alpha1","time":"2025-01-08T10:56:59Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:finalizers":{".":{},"v:\"finalizers.lambda.services.k8s.aws/Function\"":{}}}}},{"manager":"controller","operation":"Update","apiVersion":"lambda.services.k8s.aws/v1alpha1","time":"2025-01-08T11:07:05Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:ackResourceMetadata":{".":{},"f:arn":{},"f:ownerAccountID":{},"f:region":{}},"f:codeSHA256":{},"f:codeSize":{},"f:conditions":{},"f:lastModified":{},"f:lastUpdateStatus":{},"f:revisionID":{},"f:state":{},"f:version":{}}},"subresource":"status"},{"manager":"cmd","operation":"Update","apiVersion":"lambda.services.k8s.aws/v1alpha1","time":"2025-01-08T11:07:56Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:services.k8s.aws/region":{}}},"f:spec":{".":{},"f:code":{".":{},"f:s3Bucket":{},"f:s3Key":{}},"f:environment":{".":{},"f:variables":{".":{},"":{}}},"f:ephemeralStorage":{".":{},"f:size":{}},"f:handler":{},"f:memorySize":{},"f:name":{},"f:role":{},"f:runtime":{},"f:tags":{},"f:timeout":{}}}}]},"spec":{"code":{"s3Bucket":"s3-bucket","s3Key":"index.zip"},"environment":{"variables":{"key":"value"}},"ephemeralStorage":{"size":512},"handler":"index.handler","memorySize":1024,"name":"log-transformer-lambda","role":"arn:aws:iam::12345678:role/log-transformer-lambda","runtime":"**nodejs22.x**","tags":{"key":"value"},"timeout":600},"status":{"ackResourceMetadata":{"arn":"arn:aws:lambda:us-west-2:12345678:function:log-transformer-lambda","ownerAccountID":"12345678","region":"us-west-2"},"conditions":[{"type":"ACK.ResourceSynced","status":"True","lastTransitionTime":"2025-01-08T11:07:05Z","reason":"","message":"Resource synced successfully"}],"codeSHA256":"xxxx","codeSize":3106,"lastModified":"2025-01-08T10:57:03.000+0000","lastUpdateStatus":"Successful","revisionID":"1a9c1429-6882-468e-899c-a75b19ac48f1","state":"Active","version":"$LATEST"}},"Error":null}]}

AWS ack lambda picks up the diff for the updated runtime but does not persist on AWS:

{"level":"info","ts":"2025-01-08T11:07:57.414Z","logger":"ackrt","msg":"desired resource state has changed","kind":"Function","namespace":"custom-resource-namespace","name":"log-transformer-lambda","account":"12345678","role":"arn:aws:iam::12345678:role/ack-role","region":"us-west-2","is_adopted":false,"generation":75,"diff":[{"Path":{"Parts":["Spec","Architectures"]},"A":null,"B":["x86_64"]},{"Path":{"Parts":["Spec","Description"]},"A":null,"B":""},{"Path":{"Parts":["Spec","PackageType"]},"A":null,"B":"Zip"},{"Path":{"Parts":["Spec","Runtime"]},"A":"nodejs22.x","B":"nodejs16.x"},{"Path":{"Parts":["Spec","SnapStart"]},"A":null,"B":{"applyOn":"None"}},{"Path":{"Parts":["Spec","TracingConfig"]},"A":null,"B":{"mode":"PassThrough"}}]}
{"level":"info","ts":"2025-01-08T11:07:58.652Z","logger":"ackrt","msg":"updated resource","kind":"Function","namespace":"custom-resource-namespace","name":"log-transformer-lambda","account":"12345678","role":"arn:aws:iam::12345678:role/ack-role","region":"us-west-2","is_adopted":false,"generation":75}

Function resource still shows old runtime after reconciliation:

Steps to reproduce

• Create a custom controller that creates and reconciles Function k8s resource with lambda runtime nodejs v16 (or any runtime).
• Update custom controller with a different runtime version using controller/controllerutil CreateOrUpdate.

Expected outcome
It is expected that both Functions k8s resource and specified lambda on AWS get updated with new runtime configuration.

Environment

• Kubernetes version: v1.30.3
• Using EKS (yes/no), if so version? no
• AWS service targeted (S3, RDS, etc.): Lambda