Mastering Multi-Account AWS Deployments: Using Managed StackSets to Deploy Unmanaged StackSets with SAM [Technical]

The Multi-Account Strategy in Enterprise AWS Environments

As organizations scale their AWS presence, the single-account approach quickly becomes impractical and risky. The multi-account strategy has emerged as a best practice for enterprise AWS implementations, despite its perceived complexity. This approach creates clear boundaries between workloads, teams, and environments, enabling organizations to enforce strict security controls, manage costs effectively, and grant appropriate levels of autonomy to development teams.

Why Multi-Account Is Critical for Enterprises

Enterprise environments benefit from multi-account architectures in several critical ways:

1. Security Isolation: Separate accounts create natural security boundaries, containing potential breaches to a single account rather than exposing the entire organization.
2. Governance Controls: Different accounts can implement varying levels of security controls appropriate to the workload sensitivity, avoiding one-size-fits-all restrictions.
3. Cost Allocation: Granular billing and reporting allows precise attribution of cloud spending to specific business units, projects, or environments.
4. Resource Quotas: Each account has its own service quotas, preventing a single application from consuming all available resources.

Common Challenges and Resistance

Despite these advantages, organizations often face resistance when implementing multi-account patterns:

1. Perceived Operational Complexity: Operations teams frequently resist managing multiple accounts, concerned about multiplying their workload without equivalent staffing increases.
2. Consistent Governance: Maintaining consistent security policies, compliance controls, and operational standards across multiple accounts appears daunting.
3. Identity Management: Coordinating access management across accounts requires careful planning and implementation.
4. Deployment Complexity: Development teams worry about the added complexity of deploying applications across account boundaries.

Overcoming Operational Hesitancy

A key challenge in adopting multi-account strategies is engaging operations teams who may be comfortable with their existing single-account workflows. These teams legitimately worry about:

• Managing many environments with the same or fewer resources
• Applying changes consistently across all accounts
• Maintaining visibility into resources spread across accounts
• Troubleshooting issues that span multiple accounts

The solution lies in automation. By implementing infrastructure as code patterns that work seamlessly across account boundaries, organizations can maintain operational efficiency while gaining the security and governance benefits of multi-account architectures. This article examines one such pattern that effectively addresses these challenges.

In the realm of AWS infrastructure management, organizations often face challenges when deploying serverless applications across multiple accounts. This article examines an innovative approach that combines AWS CloudFormation StackSets with AWS Serverless Application Model (SAM) to streamline multi-account deployments.

The Challenge: Cross-Account Serverless Deployments

When deploying serverless applications across an AWS organization with multiple accounts, teams encounter several obstacles:

1. SAM templates use the AWS::Serverless transform, which isn’t directly supported in service-managed CloudFormation StackSets
2. IAM permissions must be precisely managed across accounts
3. Execution of Lambda functions requires proper cross-account trust relationships

The solution examined here leverages a “two-tier” StackSet approach that elegantly addresses these challenges.

The Two-Tier StackSet Architecture

The architecture uses two different types of CloudFormation StackSets:

1. Service-Managed StackSet: Deploys the IAM execution roles to target accounts
2. Self-Managed StackSet: Deploys the actual SAM resources to target accounts

This approach provides several advantages:

• Properly handles SAM transforms across account boundaries
• Creates a clear separation of IAM permissions from application resources
• Establishes a reliable deployment sequence

Key Components of the Architecture

The codebase implements this architecture with several critical components:

1. Service-Managed StackSet for IAM Roles

The first deployment tier uses AWS service-managed StackSets to deploy the necessary IAM execution roles:

create-execution-iam-stackset:
    aws cloudformation create-stack-set \
        --stack-set-name $(STACKSET_EXECUTION_IAM_STACK) \
        --template-body file://$(STACKSET_EXECUTION_IAM_TEMPLATE) \
        --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \
        --permission-model SERVICE_MANAGED \
        --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false

This service-managed StackSet deploys an IAM execution role to each target account with permissions to:

• Execute CloudFormation operations
• Create Lambda functions
• Manage IAM roles and policies
• Access S3 buckets for Lambda code

2. Self-Managed StackSet for SAM Resources

The second tier uses a self-managed StackSet to deploy the SAM template across accounts:

create-per-account-stackset:
    aws cloudformation create-stack-set \
        --stack-set-name $(PER_ACCOUNT_RESOURCES_STACK) \
        --template-body file://packaged-$(PER_ACCOUNT_RESOURCES_TEMPLATE) \
        --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
        --permission-model SELF_MANAGED \
        --administration-role-arn arn:aws:iam::$(ADMIN_ACCOUNT_ID):role/$(STACKSET_ADMIN_ROLE_NAME) \
        --execution-role-name $(STACKSET_EXECUTION_ROLE_NAME)

This self-managed StackSet deploys the SAM application, which includes:

• Lambda functions with SAM transforms
• CloudWatch Events rules
• IAM roles with precise permissions

The Deployment Process

The deployment follows a precise sequence:

1. Build: SAM CLI builds Lambda functions and prepares deployment artifacts
2. Package: Lambda code is uploaded to a central S3 bucket with cross-account access policies
3. Deploy Central Resources: Central S3 bucket and shared resources are deployed
4. Deploy IAM Roles: Service-managed StackSet deploys IAM execution roles to target accounts
5. Deploy SAM Resources: Self-managed StackSet deploys the SAM application to target accounts

This sequence ensures that all necessary permissions and resources are in place before deploying application code.

Key Advantages of This Approach

This two-tier StackSet architecture offers several compelling benefits:

1. Proper SAM Transform Handling

The self-managed StackSet correctly processes the AWS::Serverless transform through the IAM execution role, which service-managed StackSets can struggle with:

Resources:
  FindStaleKeysFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub KeyRen-${AWS::Region}-${Environment}-stale-keys
      CodeUri: lambdas/key_ren-karen-for-api-key-rotation/
      Handler: src/key_ren-karen-for-api-key-rotation.handler

2. Separation of IAM and Application Concerns

By separating IAM role deployment from application deployment, security teams can maintain control over permissions while application teams control functionality:

KeyRenKarenStackSetExecutionRole:
  Type: AWS::IAM::Role
  Properties:
    RoleName: !Ref ExecutionRoleName
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Effect: Allow
          Principal:
            AWS: !Sub 'arn:aws:iam::${AdministratorAccountId}:role/${AdministratorRoleName}'
          Action: sts:AssumeRole

3. Dynamic Account Targeting

The solution employs sophisticated dynamic account targeting methods that differ between the service-managed and self-managed StackSets, addressing the unique requirements of each deployment pattern.

Service-Managed StackSet Account Targeting

For the service-managed StackSet that deploys IAM roles, the solution leverages AWS Organizations integration with a specialized targeting approach:

# Create service-managed stack instances with organizational unit targeting
aws cloudformation create-stack-instances \
    --stack-set-name $(STACKSET_EXECUTION_IAM_STACK) \
    --deployment-targets "OrganizationalUnitIds=[$(ROOT_OU)],AccountFilterType=INTERSECTION,Accounts=[$(TARGET_ACCOUNT_IDS)]" \
    --regions $(REGION)

This approach combines two powerful targeting mechanisms:

1. Organizational Unit-based deployment: The deployment targets a specific OU tree in the AWS organization
2. Account intersection filtering: The deployment is further restricted to only the specific accounts in the target list

This hybrid approach provides the benefits of service-managed StackSets (automatic deployment to new accounts) while ensuring precise control over which accounts receive the IAM roles. The system uses a helper script to dynamically generate the target account list:

# Get accounts from all OUs including sub-OUs recursively
OU_ACCOUNT_IDS := $(shell ./get_accounts.sh $(OU_LIST))

This script recursively traverses the organizational structure to identify all accounts in the target OUs, ensuring complete coverage even as the organization evolves.

Self-Managed StackSet Account Targeting

For the self-managed StackSet that deploys the SAM-based application resources, a different targeting approach is needed:

# Create stack instances in target accounts using direct account targeting
aws cloudformation create-stack-instances \
    --stack-set-name $(PER_ACCOUNT_RESOURCES_STACK) \
    --accounts $(SPACE_SEPARATED_ACCOUNTS) \
    --regions $(REGION)

This approach:

1. Uses direct account ID targeting rather than organizational unit targeting
2. Relies on the pre-deployed execution roles in those accounts
3. Enables precise control over which accounts receive the application stack

The solution includes sophisticated error handling to ensure all targeted accounts have the necessary execution roles and to manage deployment failures gracefully:

check-per-account-stack-instances:
	@echo "Checking stack instances for all target accounts in self-managed stackset..."
	@missing_accounts=""; \
	for account in $(SPACE_SEPARATED_ACCOUNTS); do \
		if ! aws cloudformation list-stack-instances \
			--stack-set-name $(PER_ACCOUNT_RESOURCES_STACK) \
			--region $(REGION) \
			--query "Summaries[?Account=='$$account'].Account" \
			--output text | grep -q "$$account"; then \
			if [ -z "$$missing_accounts" ]; then \
				missing_accounts="$$account"; \
			else \
				missing_accounts="$$missing_accounts $$account"; \
			fi; \
		fi; \
	done; \
	if [ ! -z "$$missing_accounts" ]; then \
		echo "Creating stack instances for missing accounts..."; \
		aws cloudformation create-stack-instances \
			--stack-set-name $(PER_ACCOUNT_RESOURCES_STACK) \
			--accounts $$missing_accounts \
			--regions $(REGION) \
			--operation-preferences MaxConcurrentPercentage=100,FailureTolerancePercentage=100; \
	fi

This automated verification ensures that all targeted accounts successfully receive the application resources, maintaining consistency across the organization.

4. Automated Pipeline Integration

The solution integrates with CI/CD pipelines (Azure Pipelines in this case) for automated deployment:

- task: AWSShellScript@1
  name: deploy_stackset
  displayName: "Deploy Per-Account Resources as Self-Managed StackSet"
  inputs:
    awsCredentials: 'xxxxxxxxxx - AzureDevopsDeploymentRole'
    regionName: '$(REGION)'
    scriptType: 'inline'
    inlineScript: |
      make deploy-per-account-resources

Real-World Application: API Key Rotation

The example deployment function is an API key rotation system across multiple AWS accounts using this two-tier StackSet approach. The application:

1. Identifies stale IAM access keys across all accounts
2. Creates ServiceNow tickets for keys requiring rotation
3. Optionally deactivates and deletes unused keys
4. Uploads reports to a central S3 bucket

The Lambda function deployed through SAM examines IAM users in each account:

async function processUserKeys(user, totalStaleKeys, staleAndUnusedKeys, oldButRecentlyUsedKeys, actionsPerformed) {
    // Process keys for each user
    const keys = await iam.listAccessKeys({ UserName: user.UserName }).promise();
    
    for (const key of keys.AccessKeyMetadata) {
        // Check if key is stale and take appropriate action
        const keyDetails = await iam.getAccessKeyLastUsed({ AccessKeyId: key.AccessKeyId }).promise();
        // Process key based on last usage...
    }
}

Implementation Considerations

When implementing this architecture, consider these important factors:

1. S3 Bucket Policies

The central S3 bucket requires policies allowing cross-account access:

aws s3api put-bucket-policy \
    --bucket $(BUCKET)-stackset \
    --policy '{"Version":"2012-10-17","Statement":[{"Sid":"AllowCrossAccountAccess","Effect":"Allow","Principal":{"AWS":"*"},"Action":["s3:GetObject","s3:ListBucket"],"Resource":["arn:aws:s3:::$(BUCKET)-stackset","arn:aws:s3:::$(BUCKET)-stackset/*"],"Condition":{"StringEquals":{"aws:PrincipalOrgID":"$(ORG_ID)"}}}]}'

2. IAM Role Trust Relationships

The self-managed StackSet execution role must trust the administrator role:

AssumeRolePolicyDocument:
  Version: '2012-10-17'
  Statement:
    - Effect: Allow
      Principal:
        AWS: !Sub 'arn:aws:iam::${AdministratorAccountId}:role/${AdministratorRoleName}'
      Action: sts:AssumeRole

3. CloudFormation Capabilities

Both StackSets require specific CloudFormation capabilities:

--capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND

CAPABILITY_AUTO_EXPAND is particularly important for SAM templates.

Accelerating Multi-Account Solutions with LLMs

The multi-account StackSet solution described in this article represents a complex deployment pattern that would typically require weeks of development, testing, and refinement. However, by leveraging Large Language Models (LLMs) like Claude 3.7 Sonnet, enterprises can dramatically accelerate the development of such solutions while maintaining high quality and security standards.

LLM-Enhanced Infrastructure Development

The complex infrastructure code demonstrated in this article—managing IAM permissions across accounts, handling SAM transforms, and ensuring proper trust relationships—can be developed in a fraction of the traditional timeline using LLM assistance:

1. Rapid Template Generation: CloudFormation templates for both the service-managed and self-managed StackSets can be generated and refined through iterative prompting, reducing days of work to hours.
2. Error Prevention: LLMs help identify potential security issues in IAM policies and cross-account access patterns, ensuring that least-privilege principles are followed.
3. Documentation Integration: LLMs can simultaneously generate comprehensive documentation alongside code, making complex multi-account patterns more accessible to operations teams.
4. Makefile and Pipeline Optimization: The automation scripts and CI/CD pipelines that make multi-account deployments manageable are iteratively improved through AI-suggested optimizations.

For example, the intricate Makefile logic for handling account targeting and deployment sequencing—a component that would typically require careful testing and refinement—can be generated in minutes with proper error handling and edge cases considered.

Enterprise LLM Implementation for Infrastructure Development

While our team leveraged advanced LLMs for this project, enterprises with sensitive data or regulatory requirements can achieve similar benefits with proper implementation strategies:

1. Private Deployments: Organizations handling sensitive AWS account structures can deploy models like Claude on private infrastructure or leverage enterprise-grade solutions with appropriate data protection.
2. Infrastructure-Specific Training: LLMs can be specialized with company-specific infrastructure patterns, security standards, and compliance requirements.
3. Secure Development Environments: For highly sensitive cloud configurations, LLMs can operate in air-gapped environments that never expose security-critical details outside the organization.
4. Guardrails for Security Standards: Enterprise implementations can include guardrails ensuring infrastructure code meets company security standards and compliance requirements.

Measurable Development Acceleration for Multi-Account Solutions

Complex multi-account deployment patterns like the one described in this article typically require 2-3 weeks of development time from experienced cloud engineers. With LLM assistance, similar solutions can be developed in as little as 2 days—representing an 85% reduction in development time without compromising quality or security.

This acceleration provides several specific benefits for enterprises implementing multi-account strategies:

1. Faster Security Improvements: Security-enhancing patterns like cross-account key rotation can be implemented more quickly, reducing security risk.
2. Reduced Developer Resistance: By making multi-account patterns easier to implement, developers are less likely to resist the organizational shift to multi-account architectures.
3. Consistent Implementation at Scale: Infrastructure patterns can be standardized and replicated easily across many accounts, ensuring consistent security posture.
4. Focus on Business Value: Cloud engineers spend less time on the complex boilerplate of multi-account setups and more on enhancing the business value of cloud resources.

Organizations that successfully integrate LLM-assisted development while respecting their data protection requirements position themselves to achieve significant operational efficiencies while maintaining the security benefits of multi-account architectures.

Conclusion

The two-tier StackSet approach—using a service-managed StackSet to deploy IAM roles followed by a self-managed StackSet to deploy SAM resources—creates a powerful pattern for multi-account serverless deployments. It successfully addresses the challenges of cross-account SAM deployments while maintaining proper separation of security and application concerns.

This architecture is particularly valuable for organizations with strict security requirements that need to deploy serverless applications across complex multi-account environments. By leveraging the strengths of both service-managed and self-managed StackSets, you can achieve streamlined, secure, and scalable serverless deployments across your AWS organization. More importantly, this approach helps overcome the operational hesitancy often associated with multi-account architectures by providing a reliable, automated deployment pattern that scales with your organization.

The integration of LLM-assisted development further removes barriers to multi-account adoption by dramatically reducing the complexity and time required to implement these sophisticated patterns. With the right LLM implementation approach, enterprises can accelerate their cloud security posture while maintaining full control over sensitive infrastructure configurations.

Through infrastructure automation like the pattern described in this article, companies can realize the security benefits of account isolation without multiplying operational overhead. The result is a more secure, scalable, and manageable AWS environment that aligns with enterprise governance requirements while remaining operationally practical.

Full Source Code Available Soon

We’re excited to announce that the complete source code for the two-tier StackSet solution described in this article will be made available in the coming weeks. This release will include:

• CloudFormation templates for both service-managed and self-managed StackSets
• SAM application templates with cross-account functionality
• Makefile with comprehensive automation for multi-account deployments
• CI/CD pipeline definitions for Azure DevOps
• Detailed documentation on implementation and customization

Stay tuned for updates on our Gitlab repository. Once published, the code will serve as a practical starting point that you can customize for your own multi-account AWS architectures, allowing you to immediately benefit from this advanced deployment pattern while adapting it to your specific organizational needs.