Production setup

Read this if the Quickstart works and you are preparing a deployment for real traffic. You will move the signing key into AWS KMS or Vault, give every host one evidence sink, register for an issuer number, lock down the sidecar, open only the outbound connections Laidline needs, and wire up health checks. Skip if you are still trying Laidline out (Quickstart). Time: an afternoon, plus your certificate authority's lead time for a trust-list certificate. Needs: pip install "laidline[image,audio,server,kms,evidence]", an AWS KMS key or a Vault/OpenBao Transit key, an S3 bucket with Object Lock or a Postgres database, and an email address at your organisation's domain.

Before you start

  • A laidline.yaml from laidline init, tried through the Quickstart. You change it in place; keep payload-k1.key beside it and never commit either to a repository.

  • The production extras on every host:

    pip install "laidline[image,audio,server,kms,evidence]"

    Add video, text or ingest for what you mark (Install).

  • The weights on every host or in your image (laidline models fetch), so no request waits for a download.

The order matters in one place: registration binds your licence to the signing key, so move the key first (steps 1 and 2) and register after (step 4). The certificate can change later without a new registration as long as the key stays the same.

1. Move the signing key into AWS KMS or Vault

In production the private key that signs your Content Credentials lives in a key manager and never on disk. Create an ECC P-256 signing key in AWS KMS:

aws kms create-key --key-spec ECC_NIST_P256 --key-usage SIGN_VERIFY --description "Laidline signing key"

Expected output (shortened):

{
    "KeyMetadata": {
        "Arn": "arn:aws:kms:<region>:<account>:key/<id>",
        "KeySpec": "ECC_NIST_P256",
        "KeyUsage": "SIGN_VERIFY",
        ...

Give the role your hosts run as kms:Sign and kms:GetPublicKey on that key, nothing more. Then point laidline.yaml at it:

signing:
  backend: aws-kms
  aws:
    key_id: arn:aws:kms:<region>:<account>:key/<id>
    region: <region>
  cert_chain_path: ./certs/dev-chain.pem   # step 2 writes it; later your trust-list chain

What just happened: the key was created inside KMS and cannot be exported; Laidline sends each manifest's digest to KMS and gets the signature back. With HashiCorp Vault or OpenBao, use a Transit key of type ecdsa-p256 and signing.backend: vault-transit instead, with signing.vault.addr, key_name, an explicit key_version and the token in the variable signing.vault.token_env names (Configuration, Certificates).

2. Get a certificate for the key

Every manifest carries a certificate chain for the signing key. Until your trust-list certificate arrives, certify the KMS key with a development CA, so the deployment can start:

laidline keys dev-cert --kms-key-id "arn:aws:kms:<region>:<account>:key/<id>" --region <region> --org "Example Org" --cn "Example image signer"

Expected output:

wrote ./certs/dev-chain.pem and dev-ca.pem for KMS key arn:aws:kms:<region>:<account>:key/<id>; set signing.backend: aws-kms and signing.aws.key_id: arn:aws:kms:<region>:<account>:key/<id>

signing.cert_chain_path already points at it. For a manifest that other validators trust, you need a certificate from a CA on the C2PA trust list. Such a CA certifies only a generator product listed with the C2PA Conformance Program, so list your AI system first, then send the CA a signing request made by the key itself:

laidline keys csr --kms-key-id "arn:aws:kms:<region>:<account>:key/<id>" --region <region> --org "Example Org" --cn "Example image signer" --out signer.csr

The command prints the key it used and the SHA-256 of its public key, and writes signer.csr; only the public key leaves KMS.

What just happened: the development chain lets the deployment run now; your own detect trusts it, and other validators report the manifest as valid but not trusted. When the CA returns the chain, save it leaf first, set signing.cert_chain_path to it and restart. Laidline checks the chain at start (organisation in the subject, digitalSignature, an accepted extended key usage, the public key matching the KMS key) and names the failing check if one fails. The profile, the accepted key usages and key rotation without a restart are in Certificates.

3. Give every host one evidence sink

The evidence log is your record of every mark, the meter for credits, and the registry detect reads to confirm a watermark (grade B). A file sink on each host splits it: another host then reads your registered marks as grade D with registry: missing. Make the first sink one that every host writes to. With S3 and Object Lock:

evidence:
  sinks:
    - type: s3
      bucket: <your-evidence-bucket>
      prefix: laidline/
      region: <region>
      object_lock: true

Or with Postgres, after creating the schema once as a separate migrator role:

laidline evidence init-db

evidence:
  sinks:
    - type: postgres
      dsn_env: LAIDLINE_PG_DSN

What just happened: sinks[0] is now authoritative: a failed write there fails the mark, and detect, the usage statement and the evidence export read it. The S3 sink checks at start that the bucket has Object Lock and needs s3:PutObject, s3:GetObject, s3:ListBucket and s3:GetBucketObjectLockConfiguration, never a delete permission. The Postgres app role can only read and insert. Postgres is the faster primary at high volume; the measured write latencies are in Benchmarks. Later sinks are replicas (Evidence log).

4. Register the deployment

Registration gives you a registered issuer number (1 to 32,767, never reused), a free licence and an issuer certificate that binds your signing key and your private payload key. Your marks then name you. Run it on the host that holds the production configuration:

laidline register --email you@yourcompany.com --display-name "Yourcompany Inc" --plan registered_free --accept-terms

Expected output, after you follow the link in the email:

check you@example.com for the link (registration <registration id>)
waiting for the link to be clicked
licence <licence id>: plan registered_free, valid until <date>
issuer <number>: display public as <display name>
issuer: registered <number> as <display name>
reload: send SIGHUP to laidline serve or restart it

What just happened: register signed a request with your signing key, carrying your acceptance of the Laidline Customer Licence Agreement, waited for the email link (30 minutes by default, --timeout), then verified both documents under the two Laidline root keys pinned in the package (The Laidline roots) and installed the licence in licence.d/ and the certificate as issuer-certificate.json beside laidline.yaml. Your configuration keeps its trial number: the certificate lists it, and Laidline uses the registered number for private marks. The directory publishes your display name and domain beside the number; a name you give with --display-name must contain your domain's label. Reload the sidecar, then check:

laidline licence status

The issuer: line now reads registered <number> as <display name> and state: reads licensed.

When register cannot finish, it prints the registration id and resume with: laidline licence fetch <registration id> and stops: on review (with the reason and what to do), on a pending payment, or when the wait runs out. Like register, licence fetch exits 0 when it stops waiting, so a script checks laidline licence status afterwards. Questions about a review go to support@laidline.com.

What registration changes:

  • Within the month's allowance (1,200 credits on the free plan), new marks carry your registered number and your private payload key, and read as "registered to" your name. Over it, the rest of that UTC month's marks are shared marks again; they return to private on the 1st. Marking never stops.
  • Paid plans raise the monthly credits and the number of signing keys a licence binds: Lite 15,000 credits and 1 key, Starter 150,000 and 2, Growth 1,000,000 and 5, Scale 10,000,000 and 10 (--plan lite, starter, growth, scale; --interval year bills yearly). Enterprise is sold by contract. Prices are on laidline.com/licensing.
  • Renewal is automatic. A licence covers the period you pay for plus 30 days' grace, and the deployment fetches the renewed one on its own. licence.auto_fetch: false turns that off for an air-gapped host, which you renew with laidline licence install (Automatic renewal). The deployment signs each fetch with its registered key; a rotated key must be bound before it can fetch, by running laidline register again (The documents proof).
  • Detection and verification stay free in every state.

The monthly usage report

A registered laidline serve sends one signed usage report a month to the issuing service: the month, the credits and the counts of private and shared marks per medium, your payload key fingerprints and the product version. It never holds media, prompts, email addresses or file names, and marking never waits on it. Turn it off with:

issuer:
  usage_report: false

laidline usage prints the same statement locally; laidline usage report --month YYYY-MM sends a completed month by hand. See The usage report.

5. Set LAIDLINE_ENV=production

export LAIDLINE_ENV=production

What just happened: an unset LAIDLINE_ENV already means production for every relaxed check: an inline payload-key secret, a key file readable by other users, the published test secret and extra trust roots are refused or ignored unless LAIDLINE_ENV=development is set exactly. Setting production explicitly adds one more refusal: the local PEM signing backend that laidline init writes. A host that still has the development key configured then stops at start with signing.backend=local is not allowed in production, instead of signing with it. Set the variable in the service definition of every production host (Configuration).

6. Turn on the security settings

The sidecar's defaults are safe for one machine. Before its port is reachable from anywhere else, set an inbound token and decide what to expose.

Bind. laidline serve listens on 127.0.0.1 unless you pass --host 0.0.0.0, and its first log line says which:

... INFO laidline.cli listening on 0.0.0.0:8080 (all interfaces)

A container needs --host 0.0.0.0; publish its port to the network that should reach it, not to the internet.

Inbound token. Name an environment variable in laidline.yaml and put a token of at least 32 characters in it:

server:
  mark_token_env: LAIDLINE_MARK_TOKEN

export LAIDLINE_MARK_TOKEN="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"

Restart the sidecar; its start-up log says inbound token on. A request without the token is refused:

curl -s -F file=@in.png localhost:8080/v1/mark/image

Expected output:

{"detail":{"reason":"unauthorized","detail":"a valid Authorization: Bearer token is required (server.mark_token_env)"}}

What just happened: every route except /healthz, /readyz, /metrics, /detect, /v1/assets/* and /admin/* now needs Authorization: Bearer <token>: the mark routes, the job routes, the adapter rules, the OpenAI-compatible proxy and the generated API docs. /v1/assets/* stays open because its URLs are unguessable links that your end users' browsers fetch. The variable is read on every request, so you rotate the token without a restart. Clients send it as LAIDLINE_API_KEY (MarkClient puts it in the Authorization header). The sidecar removes the token before it proxies a request, so a proxied upstream that needs a key gets it from upstream.auth: env and the variable upstream.api_key_env names (OpenAI-compatible proxy).

Upload cap. server.max_upload_bytes is 64 MiB (67,108,864 bytes) by default for every request body: mark and detect uploads, proxy and adapter requests. Above it the mark, proxy and adapter routes answer 413 upload_too_large and /detect answers 413; a request whose Content-Length is already over the cap is refused before its body is read, and a body sent without a length is cut at the cap as it arrives. Cap request bodies at your reverse proxy as well, as a second line of defence. Video uploads have their own cap, jobs.max_upload_bytes (2 GiB). Lower the cap if your outputs are small.

Admin token. /admin/reload, /admin/evidence/status and /admin/evidence/export exist only while LAIDLINE_ADMIN_TOKEN holds at least 32 characters; unset, every /admin path answers 404. Set it only where you use those routes, and keep /admin reachable from your own network only (Configuration).

Metrics and detect. /metrics has no token and shares the port; keep it on your internal network or behind your reverse proxy. /detect is open by design (detection is free) and limited to 60 requests a minute per client address (detect.rate_limit).

7. Allow the outbound connections

Laidline sends nothing you did not configure. A production host needs these, and nothing else:

Destination Port Used for When
timestamp.digicert.com, fallback timestamp.sectigo.com 80 (HTTP) RFC 3161 timestamps on every manifest and checkpoint every mark
Your KMS or Vault endpoint 443 signing every mark
Your S3 bucket or Postgres database 443 or 5432 the evidence sink every mark
issue.laidline.com 443 registration, licence renewal, the monthly usage report once a day at most
directory.laidline.com 443 laidline issuer refresh (the issuer directory) when you run it
huggingface.co 443 AudioSeal weights only when weights are missing; none after laidline models fetch
dl.fbaipublicfiles.com 443 VideoSeal and PixelSeal weights only when weights are missing; none after laidline models fetch
The host in LAIDLINE_MODELS_MIRROR 443 model weights from a Laidline mirror only if you set it, tried before the two rows above

Timestamps are RFC 3161 over plain HTTP: the tokens are signed, so TLS adds nothing. Laidline ships the public roots of both default timestamp servers (signing.tsa.roots_path: builtin) and checks every checkpoint token against them. To use another timestamp server, set signing.tsa.url and save its root certificates as one PEM file for signing.tsa.roots_path; until you do, its tokens are stored unverified and the server logs a warning at start (Certificates). signing.tsa.required: false signs without a timestamp when none answers, at the cost of grade B for the manifest instead of A. An air-gapped host sets LAIDLINE_OFFLINE=1 (no model downloads, no licence fetch) and licence.auto_fetch: false.

8. Check readiness and health

curl -s localhost:8080/healthz

Expected output:

{"status":"ok"}

curl -s localhost:8080/readyz

Expected output once the start-up self-test has passed:

{"ready":true,"error":null}

What just happened: /healthz answers as soon as the process serves: use it as the liveness probe. /readyz turns ready only after a self-test mark succeeded: the signing key, the timestamp server and the evidence sink all worked. Use it as the readiness probe, so a host joins the load balancer only once it can mark.

While the self-test fails, /readyz answers 503 with a short category, and the server log has the full message:

{"ready":false,"error":"tsa_unavailable: MarkError"}

The category is <reason>: MarkError for a refused mark (tsa_unavailable, signer_unavailable), self_test_failed: <error class> for anything else, or startup_failed when the configuration started without a usable signing key under on_mark_failure: fail_open. The probe retries the self-test 30 seconds after a failure, backing off to 15 minutes while it keeps failing, so a network that comes back turns it ready without a restart. A change to signing.tsa needs a restart: SIGHUP (or POST /admin/reload) re-reads only the signing keys. The probe that runs a retry answers only when the self-test finishes, which includes the timestamp client's own timeout; give the readiness probe a timeout of 10 seconds or more, or accept that one probe times out. Under on_mark_failure: fail_open an unreachable timestamp server does not hold readiness back, because the self-test returns the image unmarked. Up to 10 self-test marks per UTC day are not counted as credits; a server that restarts or retries more often than that pays for the extra ones.

/metrics gives Prometheus counters for alerting: laidline_marks_total{modality,status}, laidline_detects_total{verdict}, laidline_mark_seconds, laidline_jobs_queued and laidline_jobs_total{status}.

9. Add more hosts

One organisation registers once. Every further host of the same deployment carries a copy of the registered host's identity instead of registering again:

  • laidline.yaml (with its watermark.issuer_id) and payload-k1.key;
  • the signing key: access to the same KMS key (or Vault key), plus certs/;
  • licence.d/ and issuer-certificate.json;
  • the same evidence sink as sinks[0] (step 3).
cp -a laidline.yaml payload-k1.key certs licence.d issuer-certificate.json /path/to/the/new/host/config/

What just happened: cp -a keeps the key files at mode 0600 and the directories at 0700. The licence binds the signing key of the host that registered, so a host with its own key reads licence_invalid:not_bound and makes shared marks; a host whose watermark.issuer_id differs from the certificate's numbers keeps making shared marks and laidline licence status says set watermark.issuer_id: N in laidline.yaml and reload. A second register from the same email domain is refused at once on the one-key plans (registered_free, lite) with the hint to copy the identity; on plans that bind more keys it goes to review, and the second host registers with --plan starter, growth or scale (More than one host).

Check it worked

On each host, with the sidecar running:

laidline licence status

The lines that show a production deployment:

state: licensed
...
marks this month: private <n>, shared 0
...
issuer: registered <number> as <display name>
...

Then mark a file and read a copy without its manifest, as in the Quickstart: the watermark now reads as "registered to" your name, grade B once it matches the shared evidence log (Grading).

Limits

  • Registration needs network access to issue.laidline.com and an email at your domain; registered names are always published beside the number.
  • An S3 evidence sink costs three writes per mark, in two sequential round trips; for high volume, Postgres is the faster primary (Benchmarks).
  • A manifest without a timestamp grades B, not A; a certificate outside the C2PA trust list reads as valid but not trusted to other validators (Grading).

Troubleshooting

Message Cause Fix
signing.backend=local is not allowed in production LAIDLINE_ENV=production with the development key still configured Finish step 1, or for a host that must sign with a local key set signing.local.allow_in_production: true.
error: registration needs the Laidline customer licence agreement <version> at <url>; ... register ran without --accept-terms outside a terminal Read the agreement, then run again with --accept-terms.
in review (<reason>): Laidline finishes it by hand The registration waits for a manual step: for example daily_cap (the day's free registrations are used up), manual_payments (a paid plan, invoiced before approval) or a second registration from your domain Follow the hint it prints; resume with laidline licence fetch <registration id>.
state: licence_invalid:not_bound The licence binds another host's signing key Copy the whole identity (step 9), or point this host at the registered KMS key.
set watermark.issuer_id: N in laidline.yaml and reload; marks stay shared until then The configured issuer number is neither the registered number nor the trial number the certificate lists Set watermark.issuer_id to N and reload.
{"ready":false,"error":"tsa_unavailable: MarkError"} No timestamp server answered the self-test Allow outbound HTTP to the timestamp servers (step 7); the probe retries on its own.
{"ready":false,"error":"signer_unavailable: MarkError"} The signing key could not sign: KMS or Vault unreachable, or a permission missing Check kms:Sign and kms:GetPublicKey (or the Vault token and policy); the server log has the full message.
401 with a valid Authorization: Bearer token is required (server.mark_token_env) The client sent no token, or the wrong one Set LAIDLINE_API_KEY on the client to the value of the variable server.mark_token_env names.
413 with upload_too_large The upload is larger than server.max_upload_bytes Raise server.max_upload_bytes, or send video to /v1/mark/video.
object_lock is true but the Object Lock configuration of bucket ... could not be read The bucket has no Object Lock, or the role lacks s3:GetBucketObjectLockConfiguration Check the IAM policy first, then the bucket (Evidence log).
A second host reads your registered marks as grade D with registry: missing Each host writes its own file sink Give every host the same evidence.sinks[0] (step 3).

Reference