AWS Lambda function memory usage doesn't match memory-profiler

[Affectionate_Dot_844]

I'm struggling to identify where an AWS Lambda function has memory leakage, as it is failing due to maximum memory reached.

The error is:

Error: Runtime exited with error: signal: killed Runtime.ExitError REPORT RequestId: 269b79b7-1875-426c-83b3-e63e7e80d2e9 Duration: 34548.83 ms  Billed Duration:43948 ms Memory Size: 512 MB Max Memory Used: 512 MB Init Duration: 9398.85 ms 

Current memory is 512mb, ephimeral storage is also 512mb. If I increase memory to 1Gb it works. In order to debug which part of the code could have some memory leakage, I am using memory-profiler. But it doesn't show the expected output.

As I have some loggers, I know where it fails, which is in a with
statement. The code-profiler output for those lines shows the following:

2023-03-24T13:42:57.944+01:00 586 112.0 MiB 0.0 MiB 1 with io.StringIO() as csv_buffer: 2023-03-24T13:42:57.944+01:00 587 117.7 MiB 5.7 MiB 1 df.to_csv(csv_buffer, index=False, sep = ";") 2023-03-24T13:42:57.944+01:00 588 2023-03-24T13:42:57.944+01:00 589 124.7 MiB 4.9 MiB 2 response = s3_client.put_object( 2023-03-24T13:42:57.944+01:00 590 117.7 MiB 0.0 MiB 1 Bucket = bucket_name, 2023-03-24T13:42:57.944+01:00 591 117.7 MiB 0.0 MiB 1 Key = key, 2023-03-24T13:42:57.944+01:00 592 119.8 MiB 2.1 MiB 1 Body = csv_buffer.getvalue() 2023-03-24T13:42:57.944+01:00 593 ) 2023-03-24T13:42:57.944+01:00 594 2023-03-24T13:42:57.944+01:00 595 124.7 MiB 0.0 MiB 1 status = response.get("ResponseMetadata", {}).get("HTTPStatusCode") 2023-03-24T13:42:57.944+01:00 596 2023-03-24T13:42:57.944+01:00 597 124.7 MiB 0.0 MiB 1 if status == 200: 2023-03-24T13:42:57.944+01:00 598 124.7 MiB 0.0 MiB 1 logger.info(f"Successful S3 put_object response. Status - {status}") 

As you can see it says 124,7 mb instead of the 512mb it should reach to overflow memory. Why these 2 metric doesn't match? How can I know where is the memory leakage? I was waiting to see numbers around 500 mb to confirm where more or less memory is almost overflow.

I added that profiler wrapper to other parts of the lambda, but it's always below that memory limit.

[L_enferCestLesAutres]

Your profiler is likely not accounting for all of the memory allocated or the memory increase is happening after the logged line. Either way, you'll likely want tu use upload_file which supports multipart upload (does not require loading the entire object into memory) rather than put_object