Storage hook on AWS Lambda

Read this if your generated media lands in S3 and you want it marked by a Lambda function instead of a long-running worker. You will deploy the storage hook as a Lambda container image fed by S3 notifications through SQS. Skip if you run a worker on a host you manage (AWS S3 with SQS). Time: about an hour, most of it building and pushing the image. Needs: the S3, SQS and KMS resources of AWS S3 with SQS, Docker with BuildKit, Amazon ECR, and an s3 (Object Lock) or postgres evidence sink.

In short

• The same IngestWorker as laidline ingest, called by an SQS event-source mapping with batches of messages.
• Run it as a container image on x86_64: 3008 MB, 120 s timeout, 1024 MB ephemeral storage, batch size 5 with ReportBatchItemFailures.
• Keep watermark.image.scheme: trustmark: PixelSeal's weights do not fit Lambda's /tmp.
• The first mark after a deploy takes about a minute because Lambda reads the image layers on first access.
• The evidence sink must be s3 (Object Lock) or postgres; a file sink is refused.

The storage hook (AWS S3 with SQS) as a Lambda function instead of a long-running worker: S3 ObjectCreated notifications go to an SQS queue, an event-source mapping invokes the function with batches of messages, and laidline.ingest.lambda_handler.handler marks each object in place with the same IngestWorker as laidline ingest.

What you need

• Build host. Docker with BuildKit, and a native Linux filesystem such as ext4 for the build context. From a network or Windows-mounted drive the copy of the large model files can fail with "cannot allocate memory".
• Image contents. A Python 3.12 image with ffmpeg, pip install "laidline[image,audio,video,server,evidence,kms,ingest,lambda]" (CPU torch keeps it small), the TrustMark weights baked in, awslambdaric as the entrypoint with laidline.ingest.lambda_handler.handler as its handler, and HOME=/tmp, LAIDLINE_HOME=/tmp and LAIDLINE_CONFIG=/etc/laidline/config.yaml. Your deployment's layer adds the configuration and certificate chain under /etc/laidline and makes them world-readable (chmod -R a+rX), because Lambda runs the image as a non-root user it picks. Prefer a KMS or Vault signing backend: a local key file baked into an image is readable by anyone who can pull it. Do not set USER, VOLUME or HEALTHCHECK in the image: Lambda picks the user itself, and once a derived image inherits such a setting it cannot be removed.
• Image push. Build the function image with --provenance=false --sbom=false. With attestations on, BuildKit pushes an OCI image index that Lambda refuses at CreateFunction ("media type ... is not supported").
• Function. x86_64, 3008 MB memory, 120 s timeout, 1024 MB ephemeral storage, HOME=/tmp and LAIDLINE_HOME=/tmp; event-source mapping with batch size 5 and ReportBatchItemFailures.
• Environment. AWS_LAMBDA_LOG_LEVEL=INFO, or no ingest ... outcome= line reaches CloudWatch. To mark audio and video, LAIDLINE_INGEST__MIME_ALLOWLIST (a JSON list naming audio/wav, video/mp4 and the image types) and LAIDLINE_WATERMARK__VIDEO__ENABLED=true; the function's environment overrides the config baked into the image.
• Evidence. An s3 (Object Lock) or postgres sink as evidence.sinks[0] (below).

Setup

1. Write the configuration

Bake this config.yaml into the image, with your own key, bucket and queue. The Lambda queue's visibility timeout is 720 s, six times the function timeout.

system: {name: acme-flux, version: "2026.10"}
signing:
  backend: aws-kms
  aws: {key_id: "arn:aws:kms:<region>:<account>:key/<id>", region: <region>}
  cert_chain_path: ./certs/dev-chain.pem
  tsa: {mode: checkpoint_only}
# paste the watermark and payload_keys blocks that laidline init wrote; never hand-write watermark.issuer_id
watermark:
  image: {scheme: trustmark}
evidence:
  sinks:
    - {type: s3, bucket: <bucket>-evidence, prefix: lambda-it/, object_lock: true, region: <region>}
  checkpoint: {every_n: 50, every_seconds: 300}
ingest:
  prefixes: ["lambda/generated/"]
  dlq_url: "https://sqs.<region>.amazonaws.com/<account>/<name>-lambda-dlq"
  s3: {region: <region>}

What just happened: you pinned the TrustMark scheme (see PixelSeal on Lambda), pointed signing at KMS and evidence at an Object Lock bucket, and limited the function to one prefix.

2. Create the queue and the notification

Follow AWS S3 with SQS, step 1 for a second queue (<name>-lambda, with its dead-letter queue, 720 s visibility) and a bucket notification on the prefix lambda/generated/. The notification call replaces the bucket's whole configuration, so keep your other entries in the file, and the prefixes must not overlap.

3. Build and push the image

Write this Dockerfile next to your config.yaml and certs/ (the chain from Production):

FROM python:3.12-slim
RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg ca-certificates && rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/cpu \
 && pip install --no-cache-dir "laidline[image,audio,video,server,evidence,kms,ingest,lambda]==1.0.1"
RUN python -c "from trustmark import TrustMark; [TrustMark(verbose=False, model_type=v, encoding_type=TrustMark.Encoding.BCH_5, loadRemover=False) for v in 'QBC']" \
 && chmod -R a+rX "$(python -c 'import trustmark, os; print(os.path.dirname(trustmark.__file__))')"
COPY config.yaml /etc/laidline/config.yaml
COPY certs /etc/laidline/certs
RUN chmod -R a+rX /etc/laidline
ENV HOME=/tmp LAIDLINE_HOME=/tmp LAIDLINE_CONFIG=/etc/laidline/config.yaml
WORKDIR /tmp
ENTRYPOINT ["/usr/local/bin/python", "-m", "awslambdaric"]
CMD ["laidline.ingest.lambda_handler.handler"]

An image installed from pip has no ONNX graph for the TrustMark Q keyframe reader, so that reader runs on torch (slower, same result): see Third-party licences.

The image pins laidline==1.0.1; change the version to the release you want. It holds Laidline with its TrustMark weights, your configuration and your certificate chain. Lambda runs the image as a non-root user it picks, so the weights and /etc/laidline are made world-readable; signing stays on KMS, never a key file in the image. Relative paths in config.yaml resolve against /etc/laidline; the function's writable space is /tmp, which is why HOME and LAIDLINE_HOME point there. The image sets no USER, VOLUME or HEALTHCHECK, and awslambdaric is the entrypoint that calls the handler named in CMD. The function is x86_64, so the build line names --platform linux/amd64 (an Apple Silicon or Graviton host would otherwise build arm64). Build and push it:

docker build --platform linux/amd64 --provenance=false --sbom=false -t <account>.dkr.ecr.<region>.amazonaws.com/<repo>:latest .

aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account>.dkr.ecr.<region>.amazonaws.com

docker push <account>.dkr.ecr.<region>.amazonaws.com/<repo>:latest

What just happened: the function image is in your ECR repository. Pin the function to the pushed digest (aws ecr describe-images --repository-name <repo> --image-ids imageTag=latest --query 'imageDetails[0].imageDigest').

4. Create the role and the function

Give the role a trust policy for lambda.amazonaws.com, attach the managed AWSLambdaBasicExecutionRole for its logs and an inline policy as under IAM. Then:

aws lambda create-function --function-name <function> --package-type Image \
  --code ImageUri=<account>.dkr.ecr.<region>.amazonaws.com/<repo>@<digest> --role <role ARN> \
  --memory-size 3008 --timeout 120 --ephemeral-storage Size=1024 \
  --environment "Variables={LAIDLINE_HOME=/tmp,HOME=/tmp,AWS_LAMBDA_LOG_LEVEL=INFO}"

A role created a moment ago can take a few seconds to reach Lambda; if the call fails with "cannot be assumed" or "execution role does not have permissions", wait five seconds and retry. Lambda adds the repository policy that lets it pull the image when a principal allowed to pull from the repository creates the function.

5. Connect the queue

aws lambda create-event-source-mapping --function-name <function> --event-source-arn <queue ARN> \
  --batch-size 5 --function-response-types ReportBatchItemFailures

What just happened: Lambda now polls the queue and invokes the function with up to 5 messages at a time; the function reports the ones to retry.

Check it worked

aws s3 cp in.png s3://<bucket>/lambda/generated/in.png

Wait for the invocation (the first one after a deploy takes about a minute, see Sizing), then:

aws s3api head-object --bucket <bucket> --key lambda/generated/in.png --query Metadata

Expected output:

{
    "laidline-asset-id": "...",
    "laidline-manifest-id": "..."
}

In CloudWatch the function's log shows an ingest ... outcome=marked line for the object.

How it works

• Image. The shared base (the extras image,audio,video,server,evidence,kms,ingest and the TrustMark Q/B/C weights) plus awslambdaric, with HOME=/tmp and LAIDLINE_HOME=/tmp, the only writable path on Lambda. Your deployment's image adds the config and certificate chain on top (two COPY lines into /etc/laidline and a chmod).
• Audio and video objects. The image carries the audio and video extras (they add about 150 MB on disk). Audio and video objects are skipped unless ingest.mime_allowlist names their types, and video also needs watermark.video.enabled: true: set them in the config, or as the function's environment variables LAIDLINE_INGEST__MIME_ALLOWLIST (a JSON list) and LAIDLINE_WATERMARK__VIDEO__ENABLED=true. A 5 s WAV was AudioSeal-marked (unkeyed; it read back as grade C when run, before 2026-10-28, since when unkeyed audio shows presence only, grade D) and a 5 s 352x288 MP4 VideoSeal-marked (grade B) with both set, and with the variables removed the same WAV was skipped (outcome=skipped_filtered reason=mime:audio/wav). The AudioSeal and VideoSeal weights are not in the image and download on first use into /tmp/models, once per execution environment (Audio).
• Logs. The per-object ingest ... outcome= lines need AWS_LAMBDA_LOG_LEVEL=INFO (or the JSON log format with ApplicationLogLevel INFO) on the function. Without it only WARNING and above reach CloudWatch.
• Ephemeral storage: give the function at least 1 GB (EphemeralStorage: {Size: 1024}) when it marks audio or video. Lambda's default /tmp is 512 MB, and /tmp holds the weights and the working files: the VideoSeal checkpoint is 228 MB and the AudioSeal generator and detector 94 MB (/tmp/models), and a video mark spools the object and writes the re-encoded stream and the final mp4 next to it, about three times the object's size (up to about 200 MB at the default ingest.max_bytes of 64 MiB). An image-only function fits in the default. This sizing is arithmetic from the file sizes; 1024 MB was enough for a 5 s WAV and a 5 s 352x288 MP4.
• Event-source mapping. FunctionResponseTypes: [ReportBatchItemFailures] is required. The handler returns {"batchItemFailures": [...]} listing the message ids to retry (a retryable failure before ingest.max_attempts, counted from ApproximateReceiveCount); Lambda deletes every other message of the batch. A message whose object failed for good is forwarded to ingest.dlq_url when set, then deleted.
• Cold and warm. The Marker (signer, watermark models, evidence sinks) is built on the first invocation in an execution environment and kept for the warm ones; importing the module builds nothing.
• Durable evidence. evidence.sinks[0] must not be a file sink: the instance's disk is gone when Lambda recycles it. The handler refuses a file primary with a ConfigError on the first invocation, so the batch fails and is retried, and the error is in the function's log. Use s3 (Object Lock) or postgres.
• Per-record time guard. Before each record after the first, the handler compares the invocation's remaining time (context.get_remaining_time_in_millis()) with LAIDLINE_LAMBDA_RECORD_BUDGET_MS (the time one record may need; default 20000, a negative value counts as 0, a value that is not an integer falls back to the default). When less is left, it stops, lists that record and every later one in batchItemFailures for redelivery and logs a WARNING with the time left, the budget and how many records were handed back. The first record of a batch always runs, so a function timeout below the budget still makes progress. Each redelivery counts as a receive against the queue's redrive policy (maxReceiveCount).
• Checkpoints. Lambda freezes the process between invocations, so the checkpoint timer thread is not started. With ingest.checkpoint_per_invocation: true (the default) the unanchored tail is checkpointed before each invocation returns, waiting at most the remaining time less 5 s (clamped to 1 to 30 s).
• Signing keys. The keyring is read at the first invocation. A rotation (laidline keys rotate) reaches Lambda with a new image or configuration, which starts new execution environments; SIGHUP and /admin/reload do not apply here.

PixelSeal on Lambda

The image scheme defaults to pixelseal, but the Lambda image bakes only the TrustMark weights. PixelSeal's pinned checkpoint is 1.24 GB, and its slimmed copy, which is what mark and detect load, is 309 MB more. They would download into /tmp on first use, and the download alone exceeds a 1,024 MB /tmp. Pin watermark.image.scheme: trustmark for the function, as the configuration above does, and keep that pin on Lambda.

Running PixelSeal on Lambda needs a container image with the slim weights (pixelseal/model.pth and its .json sidecar, from laidline models fetch --no-audio --no-video) baked into a read-only LAIDLINE_HOME path, with no checkpoint, and an ephemeral storage size measured for it. Google Cloud Run runs the same image and pins TrustMark for the same reason.

Sizing

Measured in eu-central-1 (x86_64, 3008 MB, 120 s timeout; the figures are in Benchmarks):

With the audio and video extras added (same region, 3008 MB, 120 s timeout, 1024 MB ephemeral storage; measured on the Lambda run, not in the benchmark tables), the ECR image was 911.1 MB compressed (3.27 GB on disk) and the first image mark after a deploy took 43 to 48 s. A 5 s 352x288 MP4 took 21.4 s and 1,726 MB on its first mark (the VideoSeal download included) and 11.2 s and 1,773 MB warm. A 5 s WAV took 10.8 s plus 3.0 s init cold (the AudioSeal download included) and about 1 s warm. The in-region S3 evidence write (Object Lock, n=200) measured p50 87.5 ms and p95 110.3 ms warm. Each in-place write-back raises one more invocation of about 20 ms that skips the object by its metadata.

A cold batch of 5 messages at 512 px took about 71 s against the 120 s timeout. That margin is thin for larger images or batches. The per-record time guard hands the rest of a batch back for redelivery when less than LAIDLINE_LAMBDA_RECORD_BUDGET_MS is left, but it cannot shorten a record already running, so set the budget above your slowest single mark (a cold first mark is the slowest), keep the batch size small (5 above), raise the timeout if one record can take most of it, and set the queue's visibility timeout to six times the function timeout (720 s above).

IAM

The function's role needs what laidline ingest needs (AWS S3 with SQS, IAM), narrowed to the prefixes it touches, plus AWSLambdaBasicExecutionRole for its logs: s3:GetObject and s3:PutObject on lambda/generated/* of the media bucket, the Lambda queue and its dead-letter queue, kms:Sign and kms:GetPublicKey on the one key, and on the evidence bucket s3:PutObject, s3:GetObject and s3:PutObjectRetention on the sink's prefix only. On the Object Lock evidence bucket, object reads, writes and retention can be limited to the sink's prefix, but s3:ListBucket must be granted on the whole bucket: the sink checks the bucket with HeadBucket at start, which needs s3:ListBucket and carries no s3:prefix for a condition to match (a prefix-conditioned grant fails with 403 Forbidden). The function can therefore list every key in the evidence bucket but reads and writes only objects under the sink's prefix. The role needs no ECR statement.

Limits

• S3 sends a s3:TestEvent when the notification is created. On a cold function it can start execution environments that each build the Marker (about 18 s) only to drop the test message. It costs time once and marks nothing.
• Removing the function leaves its CloudWatch log group (/aws/lambda/<function>); delete it by hand.
• After you delete the function, the resource tagging API can still list the deleted event-source mapping for a while; aws lambda get-event-source-mapping is the check that it is gone.
• Very large objects near the timeout: the first record of a batch is never cut short, so one object that needs longer than the function timeout still times out the invocation, and every message of the batch comes back after the visibility timeout (objects already marked are then skipped by their metadata).
• A cold start of about a minute after each deploy (Sizing); the in-region evidence write adds 55 to 77 ms to a mark.

What was proven

Live on 21 Oct 2026 on AWS Lambda in eu-central-1: a container image in ECR, an SQS event-source mapping with partial batch responses, and an image rebuilt from the current code with the audio and video extras. PNG, JPEG, WebP, WAV and MP4 objects were marked in place and detected, and a WAV was skipped by default. The evidence chains verified and each carried a checkpoint with a timestamp token. The same image also ran locally under the Lambda Runtime Interface Emulator. x86_64 only.

Troubleshooting

Reference

• AWS S3 with SQS: the worker this function runs, the IAM policy and the retry rules.
• Configuration: every ingest key, including checkpoint_per_invocation.
• Evidence: the s3 and postgres sinks.
• Benchmarks: the Lambda measurements.
• Audio and Video: marking those media.