CDK Past the Proof of Concept

Every CDK project I’ve built has had the same first fork in the road: the proof of concept works in one dev account, and now someone wants a second environment. That moment is when the environment config goes in. Not after four environments and a pile of copy-paste drift. The second environment is the trigger, every time, and once I had the pattern written down I never started a CDK app without it.

The config pattern itself fits on one screen, so this post also covers the things it doesn’t solve for you: a naming convention that survives AWS’s resource name length limits, a construct-ID habit I had to unlearn, defining API routes as data instead of code, and what to do when an API Gateway with a lot of endpoints blows past CloudFormation’s limits and you have to start removing nodes from CDK’s construct tree to get a deploy through.

The project most of this comes from was written against the Go bindings for CDK. The shape is the same in every language, and TypeScript is shorter on the page, so that’s what the examples use.

When the config goes in

A proof of concept has hardcoded values in it, and that’s fine. Account ID in bin/app.ts, a domain name in a string, a certificate ARN pasted straight in. The job of a POC is to find out whether the thing works at all.

The mistake is letting the second environment inherit that. The usual move is a switch (stage) at the top of the stack that picks a VPC ID and a few subnet IDs per case, with a default that throws. It works for three environments. Then someone needs a preview environment for a branch, and the default branch grows an if that matches a stage-name prefix and quietly borrows staging’s network. Then a second prefix. Now the stack knows about every environment by name, the knowledge is in control flow instead of data, and the next person to add an environment has to read the whole stack to find every place it matters.

So the rule I follow: the first commit that adds a second environment is the commit that adds the config. Everything environment-specific moves out of the stack and into a typed record. From then on the stack is a function of its config, and adding an environment is adding an entry.

The pattern

// lib/config.ts
export interface EnvironmentConfig {
  readonly appToken: string;      // short, fixed, used in every resource name
  readonly envName: string;
  readonly account: string;
  readonly region: string;
  readonly vpcId: string;
  readonly privateSubnetIds: readonly string[];
  readonly domainName: string;
  readonly certificateArn: string;
  readonly alertEmail?: string;
  readonly retainData: boolean;   // RemovalPolicy for stateful resources
  readonly logRetentionDays: number;
}

const configs: Record<'dev' | 'staging' | 'prod', EnvironmentConfig> = {
  dev: {
    appToken: 'bil',
    envName: 'dev',
    account: '111111111111',
    region: 'us-west-2',
    vpcId: 'vpc-0a1b2c3d4e5f60001',
    privateSubnetIds: ['subnet-0a1', 'subnet-0a2'],
    domainName: 'dev.example.com',
    certificateArn: 'arn:aws:acm:us-east-1:111111111111:certificate/xxx',
    retainData: false,
    logRetentionDays: 7,
  },
  staging: {
    appToken: 'bil',
    envName: 'staging',
    account: '222222222222',
    region: 'us-west-2',
    vpcId: 'vpc-0a1b2c3d4e5f60002',
    privateSubnetIds: ['subnet-0b1', 'subnet-0b2'],
    domainName: 'staging.example.com',
    certificateArn: 'arn:aws:acm:us-east-1:222222222222:certificate/yyy',
    retainData: true,
    logRetentionDays: 30,
  },
  prod: {
    appToken: 'bil',
    envName: 'prod',
    account: '333333333333',
    region: 'us-west-2',
    vpcId: 'vpc-0a1b2c3d4e5f60003',
    privateSubnetIds: ['subnet-0c1', 'subnet-0c2'],
    domainName: 'example.com',
    certificateArn: 'arn:aws:acm:us-east-1:333333333333:certificate/zzz',
    alertEmail: 'ops@example.com',
    retainData: true,
    logRetentionDays: 365,
  },
};

export function resolveConfig(name: string): EnvironmentConfig {
  const fixed = configs[name as keyof typeof configs];
  if (fixed) return fixed;

  // Preview environments ride on staging's account and network under their own name.
  if (/^pr-\d+$/.test(name)) {
    return { ...configs.staging, envName: name, retainData: false, logRetentionDays: 3 };
  }
  throw new Error(`Unknown environment: ${name}`);
}

// bin/app.ts
const app = new cdk.App();
const config = resolveConfig(app.node.tryGetContext('env') ?? 'dev');

new BillingStack(app, `${config.appToken}-${config.envName}`, {
  env: { account: config.account, region: config.region },
  config,
});

// lib/billing-stack.ts
interface BillingStackProps extends StackProps {
  readonly config: EnvironmentConfig;
}

export class BillingStack extends Stack {
  constructor(scope: Construct, id: string, props: BillingStackProps) {
    super(scope, id, props);
    const { config } = props;

    const vpc = ec2.Vpc.fromVpcAttributes(this, 'Vpc', {
      vpcId: config.vpcId,
      availabilityZones: Stack.of(this).availabilityZones,
      privateSubnetIds: [...config.privateSubnetIds],
    });

    const table = new dynamodb.Table(this, 'Table', {
      partitionKey: { name: 'pk', type: dynamodb.AttributeType.STRING },
      removalPolicy: config.retainData ? RemovalPolicy.RETAIN : RemovalPolicy.DESTROY,
    });

    if (config.alertEmail) {
      new AlarmNotification(this, 'Alerts', { email: config.alertEmail });
    }
  }
}

Deploy with cdk deploy -c env=staging, or -c env=pr-482 for a preview. The stack never asks what environment it’s in. It builds what the config describes.

Three details that matter more than they look:

• The config carries envName. It’s tempting to leave it out because the caller already knows. But the stack needs it for naming (next section), and a config record that can’t say which environment it is becomes a source of bugs the first time someone passes the wrong one.
• Preview environments are config too. The switch with a prefix match in its default branch was doing exactly what resolveConfig does, just in a place nobody would look for it. Spreading the staging record and overriding the name makes the borrowing explicit, and the overrides for retention say out loud that previews are disposable.
• Behavioral knobs are data. retainData and logRetentionDays are the kind of thing that starts as if (env === 'prod') inside a construct. Push them into the config and the construct code stays free of environment checks. Grepping lib/ for 'prod' is a test I actually run.

Naming resources so they fit

The config pattern gets you consistent infrastructure. It does not get you readable infrastructure. That comes from naming, and naming in AWS has a trap in it: a lot of resources have short name limits, and the limits are different for every service.

The convention I use for physical names is {app}-{env}-{component}. Short app token, short environment token, a component name that means something to a human reading the console at 2 a.m.

bil-prod-api
bil-prod-api-authorizer
bil-staging-fetcher
bil-dev-backups-111111111111

The rules:

1. The app token is short and fixed. Three to five characters. Pick it once, write it in the config, never spell the project’s full name into a resource name.
2. The environment token is short and fixed. dev, stg, prod. Not development, not production-us-west-2.
3. Environment goes in the name even when environments live in separate accounts. You will have two consoles open. You will have two profiles in one terminal. The name is the only thing that tells you which one you’re about to delete.
4. Globally unique names get the account ID as a suffix. S3 bucket names are global. bil-dev-backups is probably taken; bil-dev-backups-111111111111 is yours.
5. Only name what a human looks up. Lambda functions, tables, buckets, queues, log groups. Leave IAM roles, policies, and security groups to CDK’s generated names unless something external has to reference them. Every name you set is a name that can collide on a stack rename or a construct-ID change.

Rule 5 has a story behind it. A role I named by hand from the stack name needed to change, and the new role collided with the live one mid-deploy. The fix in the moment was a -v2 suffix. That suffix outlived the reason for it by years, and every new engineer asked what happened to -v1. CDK would have generated a unique name and moved on.

The construct-ID habit I had to unlearn

There’s a second kind of name in CDK, the construct ID, and I spent a long time naming those wrong. The stack I learned on had IDs like BillingRestApiVpc, BillingRestApiLambdaRole, BillingRestApiFunction, and some with the stage appended as well: WebAppSecurityGroup-staging. Every construct announced the application it belonged to, like a filename that repeats the folder name.

Construct IDs are scoped. An ID only has to be unique among its siblings under the same parent, and the parent is the stack, which already is the application. The prefix carries no information. It does carry cost: CDK builds logical IDs from the full construct path plus a hash, and builds default physical names from the stack name plus that logical ID. So bil-prod plus BillingRestApiLambdaRole becomes something like bil-prod-BillingRestApiLambdaRole7F3A2B1C-K9XQ2M, and the part a human needed is the part CDK truncates when it hits the 64-character role limit.

The stage in a construct ID is the same mistake one level down. The stack is already per-stage. Appending the stage to an ID inside it changes nothing about uniqueness and costs you a logical-ID rename (and therefore a replace) if you ever fix it.

Where the application name belongs: in the stack name, and in the physical-name helper below. Exactly two places. Construct IDs describe the role a thing plays inside the stack and nothing else: Vpc, LambdaRole, Api, Table.

Length limits that bite

Here are the limits that actually bite with a prefix in front of them. Verify anything you depend on against the current service quotas page; these are the ones I’ve personally hit.

The 32-character ones are where the convention earns its keep. billing-platform-production-api-target-group is 44 characters and CloudFormation will reject it, after it has already created everything that came before it in the dependency graph. bil-prod-api-tg is 15.

I enforce all of this at synth time rather than at deploy time, with a helper that lives next to the config:

// lib/naming.ts
export function resourceName(
  config: EnvironmentConfig,
  component: string,
  opts: { max: number; lower?: boolean; suffix?: string },
): string {
  const parts = [config.appToken, config.envName, component];
  if (opts.suffix) parts.push(opts.suffix);
  let name = parts.join('-');
  if (opts.lower) name = name.toLowerCase();
  if (name.length > opts.max) {
    throw new Error(`${name} is ${name.length} chars, limit for this resource is ${opts.max}`);
  }
  return name;
}

// usage
const tg = new elbv2.ApplicationTargetGroup(this, 'ApiTg', {
  targetGroupName: resourceName(config, 'api-tg', { max: 32 }),
  // ...
});

Failing in cdk synth costs seconds. Failing in cdk deploy costs a rollback.

Routes as data

The same instinct that moves environments into config applies to API routes. A REST API with forty endpoints wired up as forty addResource and addMethod calls is forty places to get the integration, the API-key setting, and the operation name slightly different. The version that holds up is a table:

// lib/routes.ts
export type HttpMethod = 'GET' | 'POST' | 'PUT' | 'DELETE';

export interface Route {
  readonly path: string;              // relative to the group's base resource
  readonly methods: readonly HttpMethod[];
}

export interface RouteGroup {
  readonly base: string;
  readonly apiKeyRequired: boolean;
  readonly routes: readonly Route[];
}

export const routeGroups: RouteGroup[] = [
  {
    base: 'v1',
    apiKeyRequired: true,
    routes: [
      { path: 'orders',                          methods: ['GET', 'POST'] },
      { path: 'orders/{order_id}',               methods: ['GET', 'DELETE'] },
      { path: 'orders/{order_id}/invoice',       methods: ['GET'] },
      { path: 'customers',                       methods: ['GET', 'POST'] },
      { path: 'customers/{customer_id}/notes',   methods: ['GET', 'POST'] },
      { path: 'reports/usage',                   methods: ['GET'] },
    ],
  },
  {
    base: 'webhooks',
    apiKeyRequired: false,              // third parties don't have our API key
    routes: [
      { path: 'payments', methods: ['POST'] },
      { path: 'shipping', methods: ['POST'] },
    ],
  },
];

Grouping matters. The webhook endpoints are the only ones a third party calls, so they’re the only ones that can’t require an API key, and that difference belongs on the group, not in an if (prefix === 'webhooks') buried in the loop that builds resources.

The builder is small, and smaller than the one I wrote the first time:

// lib/api-routes.ts
export function addRouteGroups(
  api: apigateway.RestApi,
  integration: apigateway.Integration,
  groups: readonly RouteGroup[],
): void {
  for (const group of groups) {
    const base = api.root.addResource(group.base, {
      defaultIntegration: integration,
      defaultMethodOptions: { apiKeyRequired: group.apiKeyRequired },
    });
    for (const route of group.routes) {
      const resource = base.resourceForPath(route.path);   // creates intermediates, reuses existing
      for (const method of route.methods) {
        resource.addMethod(method, undefined, {
          operationName: operationName(method, route.path),
        });
      }
    }
  }
}

function operationName(method: HttpMethod, path: string): string {
  // GET orders/{order_id}/invoice -> getOrdersByOrderIdInvoice
  const words = path.split('/').map((part) =>
    part.startsWith('{') ? 'By' + pascal(part.slice(1, -1)) : pascal(part),
  );
  return method.toLowerCase() + words.join('');
}

const pascal = (s: string) =>
  s.split(/[-_]/).map((w) => w.charAt(0).toUpperCase() + w.slice(1)).join('');

Two things I got wrong in the first version, so you don’t have to:

• I walked the path segments by hand with a map from segment to Resource so that orders and orders/{order_id} shared a parent. The map was keyed by the last segment, which works until two routes end in the same word (orders/{order_id}/notes and customers/{customer_id}/notes) and one of them silently lands under the wrong parent. resourceForPath does the walk, keys by full path, and reuses what exists. Default integration and method options are inherited from the parent resource, so you set them once on the group’s base.
• Methods and operation names were pipe-delimited strings like 'GET|POST' paired with 'getOrders|postOrders', split and zipped by index in the loop. It saved a few characters per row and cost every reader a minute of figuring out the convention. An array of methods and a derived operation name cost nothing and can’t drift.

The API Gateway wall

The patterns above held up across every project until one of them grew a REST API with a lot of endpoints. Then CloudFormation started refusing the stack, and the fix was not in any CDK documentation I could find at the time.

Two CloudFormation limits are in play. A stack can hold at most 500 resources. A template body can be at most 1 MB. Neither sounds tight for one API. CDK’s RestApi construct makes them tight.

Here’s what one endpoint costs in resources when you wire it up with LambdaIntegration:

• One AWS::ApiGateway::Resource per new path segment.
• One AWS::ApiGateway::Method per HTTP method.
• Two AWS::Lambda::Permission per method: one for the stage ARN and one for the test-invoke ARN, because allowTestInvoke defaults to true.
• One more Method for OPTIONS on each resource if you enabled CORS preflight.

So 100 endpoints across 60 path resources with CORS comes out to roughly 60 resources, 100 methods, 200 permissions, and 60 OPTIONS methods. That’s 420 before you’ve counted the API itself, the deployment, the stage, the Lambda functions, their roles, and their log groups. The 500 limit is gone, and every one of those resources carries an aws:cdk:path metadata entry that fattens the template on the way to the 1 MB limit.

The permissions are the part that makes no sense. API Gateway only needs one permission on the function with a wildcard source ARN covering every method and path on the API. CDK generates one per method per stage because it’s being precise, and precision costs two resources per endpoint.

The cheap fixes come first:

// Halves the permission count. The test-invoke console button stops working.
const integration = new apigateway.LambdaIntegration(handler, { allowTestInvoke: false });

// cdk.json: drop the per-resource path metadata and the CDKMetadata resource
{
  "pathMetadata": false,
  "versionReporting": false
}

When that isn’t enough, you remove the per-method permissions from the construct tree and replace them with a single wildcard. LambdaIntegration binds each permission as a child of the Method construct, not the function, so a walk down from the API root finds every one of them:

// lib/strip-method-permissions.ts
import { IConstruct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';

/** Delete every Lambda permission CDK attached under this node (one or two per API method). */
export function stripMethodPermissions(node: IConstruct): void {
  for (const child of [...node.node.children]) {
    if (child instanceof lambda.CfnPermission) {
      node.node.tryRemoveChild(child.node.id);
    } else {
      stripMethodPermissions(child);
    }
  }
}

// in the stack, after every route is wired
addRouteGroups(api, integration, routeGroups);
stripMethodPermissions(api.root);

handler.addPermission('ApiGatewayInvoke', {
  principal: new iam.ServicePrincipal('apigateway.amazonaws.com'),
  sourceArn: api.arnForExecuteApi(),   // arn:...:<apiId>/*/*/*
});

The single permission is scoped to this API with sourceArn. The first version I shipped used grantInvoke on the service principal with no source ARN at all, which lets any API Gateway in any account invoke the function. It worked, and it was wrong.

Three cautions:

• This is a hack. It depends on where in the tree CDK puts the permission, which is an implementation detail. Pin aws-cdk-lib, and put a snapshot test on the synthesized template so a CDK upgrade that moves things shows up as a diff instead of a deploy failure.
• Newer CDK gives you a less fragile way to get the same effect. Import the function with Function.fromFunctionAttributes and skipPermissions: true, hand the imported reference to the integration, and add the one wildcard permission on the real function. Same outcome, no tree surgery.
• If you turn off deploy and build the Deployment yourself (common once you want deployments named per stage), it has to addDependency on every resource subtree. Otherwise the stage goes live before the methods exist, and the first request after a deploy is a 403 from an API that doesn’t know its own routes yet.

If you’re still over the limit after the permissions are gone, the remaining options are structural:

• Split the API across nested stacks by route group. Each nested stack gets its own 500 resources. The parent owns the RestApi and the Deployment; children own resources and methods, and the deployment depends on all of them.
• Define the API as an OpenAPI document and use SpecRestApi. One RestApi resource with the whole definition inline. Resource count collapses to a handful. You lose the L2 helpers, and the inline body now counts against the 1 MB template limit, so this trades one ceiling for another.
• Use HTTP API (apigatewayv2) instead of REST API for new work. Routes and integrations are separate resources, one integration can serve many routes, and the permission story is one per integration rather than two per method.

I’d rather have known about the wall before I hit it. That’s the real reason this section exists.

Why I start here

• Adding an environment is adding a config entry, and nothing else. Preview environments included.
• No if (env === 'prod') anywhere in lib/.
• The application name lives in two places, and construct IDs say what a thing does.
• Names fit, and the ones that wouldn’t fit fail in synth.
• Routes are a table. The code that turns them into resources is twenty lines and never changes.
• The API Gateway resource count is something I check before the stack gets large, not after CloudFormation tells me.

Four environments of copy-paste drift taught me the first point. Everything after it I learned one failed deploy at a time.

I help teams design and implement cloud infrastructure that doesn’t make them want to quit. fitzpatricksoftware.com

AUTHOR

Matthew FitzPatrick

I write about Go, AWS, infrastructure, and developer tools — and I help teams ship modernization without the outages.