Blogs on Conijn.io

Govern and Discover AI Agents and Tools with Amazon Bedrock AgentCore Registry

Fri, 10 Apr 2026 08:00:00 +0200

Have you noticed it yet? Your organization is building AI agents across multiple teams. One team creates a customer support agent, another builds an internal knowledge assistant, and a third develops a set of MCP tools for code review. But none of these teams know what the others have built. Agents and tools are scattered across repositories, accounts, and business units with no central place to discover, govern, or manage them.

This is the reality for many enterprises adopting agentic AI. The pace of agent development is accelerating, but the infrastructure to manage that growth is lagging behind. Without a centralized registry, you end up with duplicated efforts, unvetted external dependencies, and no consistent way to enforce security or quality standards.

The Problem with Siloed Agent Development

When agents and tools are built in isolation, several problems compound quickly. First, there is no discoverability. A developer in one team has no way to know that another team already built an agent that solves a similar problem. This leads to redundant work and inconsistent implementations across the organization.

Second, governance becomes nearly impossible. When external MCP tools or third-party agents are pulled into projects without any vetting process, you introduce security risks. There is no audit trail, no approval workflow, and no way to enforce organizational policies before these resources reach production.

Third, collaboration stalls. Without a shared catalog, teams cannot build on each other’s work. Every new project starts from scratch instead of composing existing, tested capabilities. The result is slower delivery and higher costs.

In this blog post, we explore how Amazon Bedrock AgentCore Registry addresses these challenges by providing a centralized catalog and governance layer for AI agents, tools, skills, and custom resources across your organization.

What Is Amazon Bedrock AgentCore Registry?

Amazon Bedrock AgentCore Registry is a fully managed, serverless service that acts as a single source of truth for all your AI resources. It provides a centralized catalog where you can register, discover, and govern agents, MCP tools, skills, and custom resources.

The service supports four resource types:

A2A Agents — aligned to the Agent Card JSON schema for agent-to-agent communication
MCP Tools — aligned to the MCP Registry open specification
Skills — based on the skills.md YAML format
Custom Resources — any valid JSON, giving you flexibility for resources that don’t fit the other categories

You can interact with the registry through the AWS Console, the AgentCore SDK, Boto3, RESTful APIs, or even as an MCP tool directly from your IDE.

Solution Overview

The architecture of Amazon Bedrock AgentCore Registry is built around three personas: Admin, Publisher, and Consumer. Each has a distinct role in the lifecycle of registry resources.

The Admin Flow

The admin is responsible for setting up and managing the registry infrastructure. This involves two primary tasks. First, the admin creates and configures the registry itself using CRUD operations. Second, the admin sets up authentication and authorization for developers, consumers, and approvers. The registry supports both AWS Identity and Access Management (IAM) and OAuth-based authorization, giving you flexibility in how you control access.

The admin also creates and manages approval workflows. When a publisher submits a new resource, the registry emits events through Amazon EventBridge. The admin can integrate these events with their own CI/CD pipelines to run security scans, deduplication checks, and validation before approving or rejecting a resource.

The Publisher Flow

Publishers are agent developers, tool developers, or teams who build and maintain resources they want to share with others. The publishing workflow starts in the IDE, where a developer builds an agent or MCP tool. The developer can first search the registry to discover what already exists, avoiding duplicate work.

Once the resource is ready, the developer integrates the registry into their CI/CD pipeline. For agents, the Agent Card schema is auto-synced to the registry. For MCP tools, the endpoint URL and schema are auto-synced as well. The publisher then creates a registry record and submits it for approval.

After submission, Amazon EventBridge emits an event that triggers the admin-controlled CI pipeline. This pipeline can run security scan scoring, deduplication checks, and validation. Based on the results, the admin or an automated approver updates the registry record status to Approved, Rejected, or Deprecated.

The Consumer Flow

Consumers are agent developers or autonomous agents that need to discover and integrate existing resources into their workflows. The consumer flow starts with a search. The registry supports both semantic and lexical search, so you can find resources by meaning or by keyword. Tag-based search enables fine-grained filtering using attribute-based access control (ABAC) policies.

The registry also features metadata auto-sync, which polls live resources to keep the catalog current. When a consumer finds a resource they need, they request access through the resource onboarding API. The registry returns the appropriate credentials — OAuth client ID and secret, mTLS certificates, or IAM credentials — depending on the authorization model. The consumer can then call the agent or tool directly with proper authentication.

The Approval Workflow

The approval workflow provides a structured governance process. Registry records transition through defined states: Draft, Pending Approval, Approved, Rejected, and Deprecated. A publisher creates a record in Draft state. When they submit it for approval, it moves to Pending Approval. If auto-approve is enabled on the registry, the record moves directly to Approved.

Otherwise, the admin reviews the submission and updates the status. Approved resources can later be deprecated when they are no longer maintained. Rejected resources can be updated and resubmitted. This state machine gives you full control over what enters your organization’s catalog.

Multi-Account Support

For organizations operating across multiple AWS accounts, the registry supports sharing through AWS Resource Access Manager (RAM). This means a central platform team can maintain the registry in one account and share it with development teams across the organization. This is particularly valuable for enterprises with strict account separation between development, staging, and production environments.

Security and Governance

Security is built into every layer of the registry. Access control is handled through IAM-based policies or OAuth with custom JWT authorizers. The registry supports AWS PrivateLink and VPC endpoints, so you can access resources hosted in customer VPCs without traversing the public internet.

The approval workflow integrates with external security scanning tools. You can plug in open-source MCP scanners or third-party security solutions into your admin-controlled CI pipeline. Every state transition emits an EventBridge event, giving you a complete audit trail of who registered what, when it was approved, and by whom.

Registry vs. Gateway

A common question is how the registry relates to Amazon Bedrock AgentCore Gateway. The distinction is straightforward:

AgentCore Gateway is the data plane. It handles runtime execution, acting as a proxy when you call an agent or tool. Think of it as the traffic layer.
AgentCore Registry is the control plane. It handles build-time concerns: cataloging, discovery, and governance of resources.

These two services are complementary, not competing. You use the registry to discover and vet resources, and the gateway to execute them at runtime.

Efficiency Gains

The centralized nature of the registry introduces measurable efficiency improvements. Teams no longer spend time rebuilding capabilities that already exist elsewhere in the organization. The deduplication feature prevents redundant registrations, and semantic search makes it easy to find existing resources that match your needs.

The approval workflow reduces the security review burden by automating checks through CI/CD integration. Instead of manual reviews for every new agent or tool, you define your policies once and let the pipeline enforce them consistently. This reduces the time from development to production while maintaining your security standards.

For organizations with dozens or hundreds of agents and tools, the registry transforms what would be a chaotic landscape into a governed, searchable catalog. Platform teams gain visibility into what is being built and used across the organization, enabling better resource allocation and strategic planning.

Conclusion

As agentic AI adoption grows within enterprises, the need for centralized discovery and governance becomes unavoidable. Amazon Bedrock AgentCore Registry provides the infrastructure to manage this complexity. It gives admins control over what enters the catalog, publishers a streamlined path to share their work, and consumers a fast way to find and integrate existing capabilities.

If you are building agentic applications on AWS, consider how a registry fits into your platform strategy. You can learn more about Amazon Bedrock AgentCore Registry in the AWS documentation. For a deeper look at the MCP Registry specification, visit the MCP Registry documentation. And if you are already using Amazon Bedrock for your agentic solutions, the registry integrates directly into your existing workflows.

Photo by Strange Happenings

Why Cedar Policies Matter for Your Amazon Bedrock AgentCore Gateway

Wed, 25 Mar 2026 10:00:00 +0100

You are building an agentic solution. Your agent has access to MCP tools, and those tools perform real actions: approving expenses, issuing refunds, updating records. You write careful prompt instructions so the large language model picks the right tool for the right situation. Everything works on the happy path. But what happens when it doesn’t?

Only thinking about the happy path gets results fast, but it leaves a lot of room for surprises. Prompt engineering can steer a model, but steering is not the same as enforcing. A well-crafted prompt injection could convince the model to approve a $2,000 expense through a tool that should only handle amounts up to $50. The model followed an instruction; it just wasn’t yours.

In this post, we look at how Cedar policies in Amazon Bedrock AgentCore Gateway give you deterministic, auditable guardrails around your MCP tools, so you can stop relying on probability alone.

Why Prompt Engineering Is Not Enough

When you describe your tools to the model, you include constraints in the tool descriptions. “This tool handles expenses up to $50.” The model reads that description and, most of the time, routes requests accordingly. But large language models are probabilistic. They do not enforce rules; they predict the most likely next token. That distinction matters when the stakes are real.

There is no contract between the prompt and the model’s behavior. The model might follow your instructions 99% of the time, but that remaining 1% is where things go wrong. And in agentic applications, “going wrong” means a tool performs an action it should not have. Unlike a traditional API where input validation is deterministic, the decision to call a specific tool is made by the model based on pattern matching and token prediction.

Consider a prompt injection attack. An attacker crafts input that overrides or confuses the model’s instructions. Suddenly the model calls the low-limit expense tool with a $2,000 amount. The tool executes, the money is approved, and your prompt-based “guardrail” did nothing to prevent it. The model did not malfunction. It simply followed the most convincing instruction it received, and that instruction came from the attacker, not from you.

This is not a hypothetical risk. As agentic applications move into production and handle financial transactions, customer data, and operational workflows, the gap between “the model usually does the right thing” and “the model is guaranteed to do the right thing” becomes a real business risk. Relying on prompt engineering alone means accepting that gap as an inherent part of your system. You need a layer that evaluates requests deterministically, before the tool ever executes. That is where Amazon Bedrock AgentCore Policy comes in.

How Cedar Policies Work in Amazon Bedrock AgentCore Gateway

Amazon Bedrock AgentCore Gateway acts as the entry point for all tool calls your agent makes. When you attach a policy engine to the Gateway, every tool invocation is intercepted and evaluated before it reaches the tool itself. Amazon Bedrock AgentCore Policy uses Cedar, an open-source policy language developed by AWS, to express those evaluation rules.

A Cedar policy is a declarative statement that either permits or forbids access to a specific tool under specific conditions. Each policy defines a scope (who is making the request, what action they want to perform, and which Gateway resource it targets) and optional conditions that inspect the request’s context, such as input parameters or user attributes from an OAuth token.

Cedar follows a default-deny model. If no policy explicitly permits a request, it is denied. And if any forbid policy matches, the request is denied regardless of any matching permit policies. This forbid-overrides-permit approach means your safety constraints always win. You do not need to anticipate every possible attack vector; you only need to define what is allowed, and everything else is blocked by default.

An Expense Approval Example

Let’s make this concrete. Imagine you have an expense approval agent with three tools:

approve_small handles expenses up to $50
approve_medium handles expenses from $50 to $200
approve_large handles everything above $200

Your tool descriptions tell the model about these ranges, and most of the time the model respects them. But to enforce them deterministically, you write Cedar policies:

// Only allow the small expense tool for amounts under 50
permit(
  principal is AgentCore::OAuthUser,
  action == AgentCore::Action::"ExpenseTools__approve_small",
  resource == AgentCore::Gateway::"arn:aws:bedrock-agentcore:eu-west-1:123456789012:gateway/expense-gateway"
)
when {
  context.input.amount < 50
};

// Block the small expense tool for amounts of 50 or more
forbid(
  principal is AgentCore::OAuthUser,
  action == AgentCore::Action::"ExpenseTools__approve_small",
  resource == AgentCore::Gateway::"arn:aws:bedrock-agentcore:eu-west-1:123456789012:gateway/expense-gateway"
)
when {
  context.input.amount >= 50
};

The first policy permits the approve_small tool only when the amount is below 50. The second policy explicitly forbids it when the amount is 50 or more. Because Cedar uses forbid-overrides-permit semantics, even if another policy were to accidentally permit the request, the forbid policy would still block it.

Now, even if a prompt injection tricks the model into calling approve_small with an amount of $2,000, the Gateway evaluates the Cedar policy, sees that the amount exceeds the threshold, and denies the request. The tool never executes. No compute is consumed, no side effects occur, and the agent receives a denial it can handle gracefully.

You would write similar policies for approve_medium and approve_large, each scoped to their respective amount ranges. The result is a complete set of boundaries that mirror your business rules but are enforced independently of the model’s behavior.

Why This Matters Beyond Guard Clauses

You might be thinking: “I can just add a guard clause in my tool code that raises an exception when the amount is out of range.” And you absolutely should. Defense in depth is always a good idea. But relying solely on guard clauses has drawbacks that Cedar policies address.

First, the tool still gets invoked. That invocation consumes compute resources and incurs cost, even if the tool ultimately rejects the request. When your agent processes thousands of requests per day, those wasted invocations add up. Cedar policies stop the request at the Gateway, before any tool code runs. No Lambda execution, no container spin-up, no database connection opened for nothing.

Second, guard clauses are scattered across your codebase. They live in different functions, written in different programming languages, maintained by different teams. If someone accidentally removes or weakens a guard clause during a refactor, you are exposed, and you might not notice until something goes wrong in production. Cedar policies are centralized and declarative. They live in one place, are easy to audit, and are independent of your tool implementations. A change to your tool code cannot weaken your Cedar policies.

Third, from a governance perspective, it is far easier to review and validate a set of Cedar policies than to inspect guard clauses across dozens of functions and repositories. Auditors can read Cedar policies directly; they don’t need to understand Python, TypeScript, or whatever language your tools are written in. This separation of concerns means your security team can own the policies while your development team owns the tool implementations, each working independently without stepping on each other’s toes.

Finally, Cedar policies give you a consistent enforcement mechanism regardless of how your tools are built. Whether a tool is an AWS Lambda function, a container behind an API, or a third-party service, the same Cedar policy applies at the Gateway layer. You define your rules once, and they protect every tool uniformly. That consistency is hard to achieve with guard clauses alone, especially as your agent’s toolset grows and evolves over time.

Conclusion

Prompt engineering is a powerful tool for guiding model behavior, but it is not a security boundary. When your agent’s tools perform real-world actions, you need deterministic enforcement that no prompt injection can bypass. Cedar policies in Amazon Bedrock AgentCore Gateway provide exactly that: a centralized, auditable, default-deny policy layer that evaluates every tool call before it executes.

Combine Cedar policies with guard clauses in your tool code, and you get defense in depth that is both cost-effective and easy to govern. If you are building agentic applications on AWS, start by defining the boundaries your tools should never cross, then express them as Cedar policies.

To get started, explore the Amazon Bedrock AgentCore Policy documentation, review the common Cedar policy patterns, and read the AWS blog post on securing AI agents with Policy in Amazon Bedrock AgentCore.

Photo by Engin Akyurt

Why Prompt Management Matters for Agentic Applications

Sat, 14 Feb 2026 10:00:00 +0100

Have you experienced it already? You are working on an agentic solution and writing prompts for large language models. Where do you put your prompts? Do you write your prompt in your Python file itself? Or are you creating a separate file containing the prompt, then loading it into a variable to pass it to the model when invoking it?

Both approaches work; they have the advantage of being in your codebase. And therefore they can be versioned using git, for example. But this also requires you to redeploy your code whenever you need to tweak your prompt, temperature, top-k, or top-p. And what if you want to do some a/b testing? Are you now duplicating the variable? Are you creating multiple files?

Why Prompt Management Matters?

When you work with agentic applications, the prompt is the beating heart. The combination of the prompt and the model determines the success of your agentic application. Having the ability to easily tweak and test different versions of your prompt lets you steer the success of your agentic solution. Having a proper setup to do prompt management helps with this. It will also give you consistency, and because you can experiment with your prompt versions, you have more control over the quality of your agentic solution’s output.

It also allows you to make your prompts more efficient, reducing token consumption and lowering costs. You also have a better grip on which guardrails are in place, thereby improving the overall security posture of your agentic solution.

Having all your prompts in a central location helps with governance and stimulates re-use. This becomes especially critical when working in team environments where multiple developers are building different features of the same agentic system. Without centralized prompt management, you risk inconsistencies, duplicated efforts, and difficulty tracking which prompt versions are deployed in production versus development environments.

The Challenges of Traditional Prompt Management

Before diving into solutions, let’s examine the pain points developers face with traditional approaches. When prompts live directly in code, several problems emerge. First, there’s the deployment bottleneck. Every minor adjustment to wording, every temperature tweak, every experiment requires a full deployment cycle. This slows down iteration and makes rapid experimentation nearly impossible.

Second, there’s the versioning challenge. While git can track changes, it doesn’t provide an intuitive way to compare prompt performance across versions. You can’t easily see which version of a prompt performed better in production without building custom tracking infrastructure.

Third, collaboration becomes difficult. When prompts are scattered across multiple files and repositories, team members can’t easily discover existing prompts that might solve similar problems. This leads to reinventing the wheel and inconsistent approaches across the organization.

Finally, there’s the security and compliance aspect. When prompts contain sensitive instructions or business logic, having them embedded in code makes it harder to implement proper access controls and audit trails.

Prompt Management in AWS

When building agentic applications on AWS, you can use Amazon Bedrock Prompt Management. Prompt management in Amazon Bedrock allows you to create and iterate on your prompts faster. You can easily select a different model and test your prompt directly from the AWS Console. And it offers options to tweak the temperature, top-p, and top-k. You can create versions of your prompt and variants and test them side by side. Make adjustments and validate them.

You can use predefined prompt variables and embed them in your prompts. These variables act as placeholders that get populated at runtime with actual values from your application. This makes prompts more reusable and maintainable, as you can use the same prompt template across different contexts by simply changing the variable values.

The versioning system in Amazon Bedrock Prompt Management is particularly powerful. Each time you make changes to a prompt, you can create a new version. This creates an immutable snapshot of that prompt configuration, including the model selection, inference parameters, and the prompt text itself. You can then reference specific versions in your code, ensuring that your production application uses a tested, stable prompt while you continue experimenting with new versions in development.

Variants take this a step further by allowing you to create multiple variations of the same prompt for A/B testing purposes. You might have one variant that uses a more formal tone and another that’s more conversational. By testing these variants side by side with the same inputs, you can determine which approach yields better results for your specific use case.

Benefits Beyond Basic Functionality

The centralized nature of Amazon Bedrock Prompt Management brings additional benefits that might not be immediately obvious. For instance, it enables better cost optimization. By having all your prompts in one place, you can analyze which prompts are consuming the most tokens and focus your optimization efforts accordingly. You might discover that a slight rewording can reduce token usage by 20% without impacting output quality.

It also improves your security posture. Instead of having sensitive prompt instructions scattered across your codebase, where they might be accidentally logged or exposed, they’re stored securely in AWS. You can leverage AWS IAM to control who can view, edit, or deploy prompts, creating a proper separation of concerns between prompt engineers and application developers.

The governance aspect becomes crucial as your agentic applications scale. With centralized prompt management, you can implement approval workflows to ensure prompts go through proper review before being used in production. You can also maintain an audit trail of all changes, which is essential for compliance in regulated industries.

How to Use Prompt Management?

You can use the prompt directly in the converse method when you use the Boto3 library. You can also fetch the prompt details and feed all the settings to any other library. This flexibility means you’re not locked into a specific SDK or framework. Whether you’re using LangChain, Strands SDK, or building a custom solution, you can integrate Amazon Bedrock Prompt Management into your workflow.

When using Boto3, you can retrieve a prompt by its identifier and version. The API returns not just the prompt text, but all associated configuration, including the model ARN, inference parameters, and variable definitions. This means your application code can remain clean and focused on business logic, while all the LLM-specific configuration lives in the prompt management system.

For teams using the Strands SDK for agentic solutions, integrating with Amazon Bedrock Prompt Management provides a powerful combination. You get the agentic capabilities of Strands while maintaining the flexibility and governance of centralized prompt management.

Best Practices for Implementation

When implementing prompt management in your organization, start by identifying your most critical prompts. These are the ones that directly impact user experience or business outcomes. Migrate these first to establish patterns and workflows that you can then apply to other prompts.

Establish naming conventions early. Use descriptive names that indicate the prompt’s purpose, the model it’s designed for, and perhaps the team that owns it. This makes discovery easier as your prompt library grows.

Create a testing strategy that includes both automated and manual evaluation. Automated tests can verify basic functionality and regression, while manual review ensures the output meets your quality standards.

Document your prompts thoroughly. Include information about what the prompt is designed to do, what variables it expects, and any known limitations or edge cases. This documentation should live alongside the prompt in your management system.

Finally, implement a promotion process. Prompts should move through environments (development, staging, production) just as code does, with appropriate testing at each stage.

Conclusion

Prompt management is no longer optional for serious agentic applications. As these systems become more complex and business-critical, having a robust approach to managing, versioning, and testing prompts becomes essential. Amazon Bedrock Prompt Management provides the infrastructure to build this capability, enabling teams to iterate faster, collaborate more effectively, and maintain higher quality standards. By treating prompts as first-class artifacts with proper lifecycle management, you set your agentic applications up for long-term success.

Photo by Clam Lo

Organizing Git Access per Customer with 1Password SSH Agent

Fri, 17 Oct 2025 14:00:00 +0200

As consultants, we must juggle multiple credentials, including those from Xebia. But we are also onboarded at the customers where we are hired for our assignments. I already blogged on how I separate my browser sessions for different clients. This gives me the ability to have relevant bookmarks for my assignment. This blog will continue to explain how I organize my git access and separate them per customer.

Password management

Since I work with various customers, I store all my credentials in a password manager. I am using 1Password, and 1Password has an awesome feature called SSH Agent. With this functionality, you can store your private SSH keys in 1Password and use them when you pull or push code to a remote repository over SSH.

I create an SSH key per customer in my 1Password vault. Once created, I configure the public key in the customer’s repository system. (example: GitHub/GitLab/BitBucket) I set both an authentication and a signing key, so I upload the same key twice.

I also create a file ~/.ssh/my-customer.pub that contains the public key; this file instructs SSH on which identity to use. Since we also uploaded a signing key, I will add the public key to the ~/.git/allowed_signers file.

joris.conijn@my-customer.com ssh-ed25519 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

The only downside here is that the key must be stored in your main vault. I always create customer-specific vaults, and the 1Password agent can only use the main vault.

More information on 1Password Agents and how to configure them with GitHub.

Git Configuration

Next is the git configuration. This section has three parts. The first two parts are something that you need to set up per customer. The other part is something you need to set up just once.

First, we will create the git configuration that will be used for this customer. I create a file per customer in the following location: ~/.git/gitconfig-my-customer. This file has the following content:

[user]
	name = Joris Conijn
	email = joris.conijn@my-customer.com
	signingkey = ssh-ed25519 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

[gpg]
  format = ssh

[gpg "ssh"]
  program = "/Applications/1Password.app/Contents/MacOS/op-ssh-sign"

[commit]
  gpgsign = true

The user definition in this file will be used to commit the code. In this example, the user is Joris Conijn, and the email address is joris.conijn@my-company.com. This ensures that you commit your code with the entity known in the customer’s environment.

Use your own public key as the signing key and make sure that the 1Password path also matches your system. Next, for each customer, you will need to set the following in the ~/.gitconfig file:

[includeIf "gitdir:~/workspace/my-customer/"]
    path = ~/.git/gitconfig-my-customer

[url "git@my-customer:my-org/“]
        insteadOf = "git@github.com:my-org/"

I need to explain a few things here. The first part ensures that any Git repository in the ~/workspace/my-customer/ path uses the configuration defined in ~/.git/gitconfig-my-customer.

The second part will transform the git@github.com:my-org/my-repo.git to git@my-customer:my-org/my-repo.git. Now this is actually the secret sauce; the domain used is not a valid domain. It is just the name of the customer, and it will not resolve to the github.com servers just yet.

The one-time setup required is to configure the ~/.git/allowed_signers file for use. You can do this by adding the following snippet to the ~/.gitconfig file:

[gpg "ssh"]
    allowedSignersFile = ~/.git/allowed_signers

This will configure the list of allowed_signers in git so that the git cli tooling shows valid signatures.

SSH Configuration

We are almost done; next up is the SSH configuration itself. We now need to update the ~/.ssh/config file:

Host my-customer
	HostName github.com
	User git
	IdentityFile ~/.ssh/my-customer.pub
	IdentitiesOnly yes

What we are doing here is the following. We define a host called my-customer and overwrite the hostname with github.com. This ensures that GitHub servers are used again, and we point to our customer-specific configuration. The primary reason for this virtual hostname is that you can only define SSH configurations on hostnames, not URI paths.

Conclusion

Managing multiple customer environments as a consultant can easily become messy if credentials and Git configurations overlap. By leveraging 1Password’s SSH Agent and a structured Git and SSH setup, you can keep your projects securely isolated while maintaining a smooth developer experience. This approach ensures professional separation, clear commit histories, and effortless switching between customer contexts.

Photo by Myburgh Roux

Fixing oversized artifacts AWS CDK Pipelines

Thu, 29 May 2025 17:00:00 +0200

I built a workload using AWS Cloud Development Kit (AWS CDK), and the AWS CodePipeline stopped working at the worst moment. It was right at the end of the sprint; we had done multiple deployments before. But at this moment, the moment you might recognize. When this last PR is released, we will have met our deadline! Kaboom, the pipeline stops working, and the reason for the failure is not related to the changes you made.

Some background

When you use AWS CDK, you can write infrastructure as code using your favorite programming language. AWS CDK comes with certain constructs that make your life easier. For example, you can easily create a CodePipeline that will fetch the code from your repository. It will then synthesize the CDK code into AWS CloudFormation. The CodePipeline will then upload all assets to an Amazon S3 bucket, and the CloudFormation templates will be deployed to the target accounts. This CloudFormation template will contain references to the assets on S3. When the template is deployed, the assets are fetched from the S3 Bucket.

What has happened?

I ran into the problem that CodePipeline will pass an artifact from step to step. This artifact is a single zipped file, which makes it easy to pass it along the pipeline.

When CodePipeline wants to deploy the CloudFormation templates. It uploads the zipped file, and it knows where in the zipped file the template is located. However, all other assets are also in the zipfile; this could be the code that you want to run in AWS Lambda functions or even Lambda Layers. You can imagine that when your workload starts to become more mature, there will be more assets in this zipfile.

Why is that a problem?

Well, the artifact has exceeded the maximum size! With CDK projects, this is something that you will run into sooner or later. Usually, you only add stuff to your projects and rarely remove things. And, as you already learned, this always happens at a moment that you don’t expect at all. And in my case, at the worst possible time, right before my deadline.

How can we fix this?

The answer is quite simple. As I already explained, the problem is with the size of the artifact being passed to the CloudFormation deploy action. We just have to reduce the size after all the assets in the artifact are uploaded to S3.

private stripAssets(pipeline: pipelines.CodePipeline) {
 const policies = [
   new iam.PolicyStatement({
     effect: iam.Effect.ALLOW,
     resources: [`arn:aws:s3:::${this.artifactBucket.bucketName}/*`],
     actions: ['s3:PutObject'],
   }),
 ];


 if (this.artifactBucket.encryptionKey) {
   policies.push(
     new iam.PolicyStatement({
       effect: iam.Effect.ALLOW,
       resources: [this.artifactBucket.encryptionKey.keyArn],
       actions: ['kms:GenerateDataKey'],
     }),
   );
 }
 pipeline.addWave('BeforeStageDeploy', {
   pre: [
     new pipelines.CodeBuildStep('StripAssetsFromAssembly', {
       input: pipeline.cloudAssemblyFileSet,
       commands: [
         'S3_PATH=${CODEBUILD_SOURCE_VERSION#"arn:aws:s3:::"}',
         'ZIP_ARCHIVE=$(basename $S3_PATH)',
         'echo $S3_PATH',
         'echo $ZIP_ARCHIVE',
         'ls',
         'rm -rfv asset.*',
         'zip -r -q -A $ZIP_ARCHIVE *',
         'ls',
         'aws s3 cp $ZIP_ARCHIVE s3://$S3_PATH',
       ],
       rolePolicyStatements: policies,
     }),
   ],
 });
}

What happens in the code sample?

We make sure that we have the s3:PutObject rights on the artifact bucket. (We need to overwrite the existing artifact.)
We need the kms:GenerateDataKey rights. (Optional, I am using AWS Key Management Service for encryption on the bucket.)
We add a CodeBuildStep that will consume the artifact.
- The artifact is downloaded and unzipped by CodeBuild.
- We figure out the origin of the artifact.
- We remove all assets.
- We will zip the artifact.
- We will upload the artifact over the original.

Because we placed the CodeBuildStep in the pre-section it will be placed right before the deployment of the CloudFormation deploy action.

Conclusion

Hitting CodePipeline’s artifact limit can stall your sprint at the worst moment. By inserting a pre-deploy CodeBuildStep that strips the bulky assets after they’re safely uploaded to S3, you shrink the artifact without changing your workflow. This quick fix keeps your CDK pipeline flowing, your CloudFormation stacks deploying, and your release deadlines intact.

Photo by Ellie Burgin

AWS CDK and the Hidden Risks to Least Privilege

Mon, 28 Apr 2025 10:00:00 +0200

Have we given up on the least privileged principle? Personally, I am a big fan of it. But let’s be honest, it can also be tough to follow the principle strictly. With the rise of AWS Cloud Development Kit (AWS CDK), it became even harder.

Why does AWS CDK make it harder?

AWS CDK is a great tool to use when you are developing your infrastructure. You can easily build resources using the level 1 and level 2 constructs. The problem lies within the level 2 constructs. They are somewhat opinionated about how you should use them. For example, when you want to store secrets in the Cloud, you create a secret in Secret Manager. This secret needs to be encrypted, so we will use AWS Key Management Service. This can easily be achieved through AWS CDK:

const secret = new secretsmanager.Secret(this, 'Secret', {
 secretName: `MySecret`,
 description: 'My super Secret',
 encryptionKey: kmsKey,
 generateSecretString: {
   secretStringTemplate: JSON.stringify({ username: 'myUser' }),
   generateStringKey: 'password',
 },
});

As you can see in the code example. We are using a customer-managed KMS key. We do this because we want to control who can use the key for decryption. The fact that you pass the key into the Secret construct means that AWS CDK will help you and adds a policy to your KMS key. The policy that gets added is the following:

{
 "Effect": "Allow",
 "Principal": {
   "AWS": "arn:aws:iam::000000000000:root"
 },
 "Action": [
   "kms:CreateGrant",
   "kms:Decrypt",
   "kms:DescribeKey",
   "kms:Encrypt",
   "kms:GenerateDataKey*",
   "kms:ReEncrypt*"
 ],
 "Resource": "*",
 "Condition": {
   "StringEquals": {
     "kms:ViaService": "secretsmanager.eu-west-1.amazonaws.com"
   }
 }
}

Those familiar with KMS policies will notice that any principal can now use the key if it is used through the secrets manager service. At first, you might think it is not that bad or convenient. But the problem with this is that any principal with an allow statement on the secretsmanager:GetSecretValue action will be able to read your secret.

It gets worse

You are storing the secret for a reason. You probably need to read it somewhere in your workload. To do that, you need to create a role and grant the role the rights to read the secret. This is easily done with CDK.

secret.grantRead(role)

But this simple statement again changes the KMS policy. It will add the following policy.

{
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::000000000000:role/MyRole"
  },
  "Action": [
    "kms:Decrypt",
    "kms:Encrypt",
    "kms:GenerateDataKey*",
    "kms:ReEncrypt*"
  ],
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "kms:ViaService": "secretsmanager.eu-west-1.amazonaws.com"
    }
  }
}

At least it’s specific to the role, and this is what you want. But remember the previous policy? It already allows all principals to use this key.

How to work around this?

Sadly, the only way around this is to use the level 1 constructs. These constructs are not opinionated and are a one-to-one mapping to the AWS CloudFormation resources. With those level 1 constructs, you can specify the KMS key without changing the key policy for you. The flip side is that you do need to allow all principals who need to read the secret in your KMS key policy.

Conclusion

In summary, while AWS CDK greatly simplifies infrastructure development, it can unintentionally weaken strict least privilege practices, especially around KMS key policies. You may need to step down to level 1 constructs to maintain tighter control, trading some convenience for the precision and security your workloads deserve.

Photo by AS Photography

Optimizing OpenSearch Ingestion: Ensuring Reliability, Efficiency, and Cost Savings

Tue, 11 Mar 2025 10:00:00 +0100

Ingesting data into an OpenSearch cluster looks easy if you read the documentation. The truth is it is easy, but it all depends on how much you care about the data you are ingesting. Let me go one step back. Why do we even use OpenSearch? With the rise of AI, you also need a knowledge base. These knowledge bases can be hosted in OpenSearch. However, to use the OpenSearch database, you must also fill it out with data.

I recently was challenged to create a database for GenAI purposes. We had a dataset that needed to be loaded into OpenSearch. Amazon OpenSearch Ingestion sounds like a service that can help you with that.

Preparing the data

First of all, the data is never in the format that you want it to be. First, we will need to transform it into something that we can ingest. For this use-case, I used StepFunctions (but avoid costly loops) each document coming in will trigger an execution. In my case, I wanted to be able to search the document with semantic meaning. In order to do this efficiently, we will need to chunk the documents into smaller bits. This way, we can generate embeddings for each chunk, which can be used for semantic search. You can create a single file with all chunks, and each chunk will have the same metadata set; the only difference is the chunk text and chunk identification. Once we have this file, we will place it on S3, S3 will send an S3 Event Trigger to an SQS Queue, and the OpenSearch Ingestion pipeline will ingest it into the OpenSearch database.

So far so good

There is nothing wrong with the approach I just described. This is a textbook example coming from the AWS documentation pages. But an ingestion pipeline will cost you money even when you don’t ingest anything. For this reason, I developed a Lambda function that would stop the pipeline when no messages are in the queue. It will also start the pipeline when there are messages in the queue. This was a nice optimization from a cost perspective.

After some testing, we expected 10k documents in the system, but we only had around 5k. Where did the other documents go? The dead-letter queues were empty, and we did not have any log traces of any failures whatsoever… It turned out that when you stop the ingestion pipeline, all messages in the buffer are lost.

Acknowledgments to the rescue

You can argue a lot about the fact that when you stop a pipeline, the documents in the buffer are lost. But that is the reality that I needed to deal with. You can enable persistent buffering, which would at least help with the data loss part. But you still need to start the pipeline to continue the job. I looked more in the direction of SQS. The whole purpose of SQS is to queue messages and ensure they are processed. After some investigation, I noticed two options I wanted to share.

version: '2'
s3_pipeline:
  source:
    s3:
      workers: 10
      notification_type: sqs
      notification_source: s3
      codec:
        json: {}
      compression: none
      acknowledgments: true
      records_to_accumulate: 100
      on_error: retain_messages
      sqs:
        queue_url: https://sqs.eu-west-1.amazonaws.com/000000000000/MyQueue
        maximum_messages: 10
        visibility_timeout: 60s
        poll_delay: 0s
        visibility_duplication_protection: true
      aws:
        region: eu-west-1
        sts_role_arn: arn:aws:iam::000000000000:role/MyRole
  sink:
    - opensearch:
        serverless: false
        hosts:
          - https://vpcxxxxxx.eu-west-1.es.amazonaws.com
        index: documents-${/index}
        bulk_size: 5
        max_retries: 3
        dlq:
          s3:
            bucket: MyBucketName
            key_path_prefix: dlq/
            region: eu-west-1
            sts_role_arn: arn:aws:iam::000000000000:role/MyRole
        aws:
          region: eu-west-1
          sts_role_arn: arn:aws:iam::000000000000:role/MyRole
        document_id: ${/chunk_id}
        actions:
          - type: delete
            when: /operation == "delete"
          - type: update
            when: /operation == "update"
          - type: index
            when: /operation == "index"

The configuration options are acknowledgments and visibility_duplication_protection. The former will keep the message in the queue until it has been ingested into the sink, and the latter will ensure that the message stays in flight as long as it is in the buffer. These options will ensure that the SQS queue is used as intended. When a failure occurs, the message will reoccur in the queue and be retried. After x number of attempts, it will go to the dead-letter queue. These settings ensured that all 10k documents in the batch were ingested as expected.

The pipeline would still be stopped once the SQS Queue is empty. To prevent this from happening I also included the ApproximateNumberOfMessagesNotVisible next to the ApproximateNumberOfMessages metric.

Dealing with updates and deletes

I also wanted to share how we deal with updates and deletions. Ingestion of a new document is easy, but updates and deletions are different. When you have ingested a document with 15 chunks, and the updated version only has 12, you need to delete 3 and potentially update some chunks. For example, this might occur when you delete a paragraph in the middle of a text.

First, you need to know if the document has already been ingested. A search action on the OpenSearch database can do this. This search action returns all the chunks already present in the database. You can use logic to determine what chunk needs to be updated and which needs to be deleted. We add three additional metadata fields in each chunk: index, chunk_id, and operation.

The index field determines the index to which the document is being ingested. The chunck_id will become the internal document id, and the operation field will determine if the chunk needs to be indexed, updated, or deleted.

Conclusion

Ingesting data into OpenSearch sounds straightforward, but the real challenge lies in ensuring reliability, efficiency, and cost-effectiveness. While the textbook approach works, real-world scenarios often demand optimizations—like dynamically stopping the ingestion pipeline to save costs or leveraging SQS acknowledgments to guarantee message processing.

By implementing these adjustments, we achieved a more resilient ingestion process that ensures all documents make it into OpenSearch as expected. The combination of Step Functions for document transformation, SQS for reliable queuing, and metadata tagging for updates and deletions provided a scalable and maintainable solution. At the end of the day, ingestion isn’t just about getting data into the system—it’s about making sure the right data gets there, stays there, and can be efficiently queried when needed.

Photo by Andrea Piacquadio

RDS User Provisioning and Schema Migrations with AWS Lambda

Tue, 04 Mar 2025 10:00:00 +0100

Have you ever been in a situation where you want to provision or configure things cross-stack? Splitting these into logical stacks is always good when dealing with more complex environments. I already shared this in one of my previous blogs. But this also introduces a different problem!

Granting access to your database

When you use Amazon DynamoDB, you have to use AWS Identity and Access Management (IAM) to grant access. However, other databases like MySQL also have an internal authentication method. You have a few options when you create the database in one stack and a different stack wants to access the database.

Use the credentials that you created at deployment time.
Use identity and access management (AWS IAM).
Create a local user per use-case.

Now, all options will work in the end, but they do have their pros and cons. I am a big fan of the least privileged principle. So, for me, using the credentials you created at deployment time is too privileged. With these credentials, you can create databases and tables. This also means that you can drop tables and delete databases. You can compare these credentials with the root credentials of a Linux system or the root account for your AWS account. The root account is there for a reason. That reason is not for day-to-day work. You should use those credentials for administrative tasks, not operational tasks.

You could use AWS IAM, and this will give us the ability to be more least privileged. This is because you need to create a user in the database. That user will receive grants on what it can do on the database and tables. You can create a read-only role if you only need to read from the database. If you also need to write, you can simply add that permission. Regardless of whether you go for option 2 or 3, you still need to provision the user and the permissions in the database.

Provisioning RDS users

After you created an Amazon Relational Database Service - managed SQL database instance, you configured the database’s “root” credentials. The AWS Secrets Manager should generate these credentials. Users should never have these credentials, but if they do, they are there just in case. Since we don’t want to use the root credentials, we need a user to access the database through our application. For this, we can use a provisioner lambda function. This lambda function creates the local users in the database. You can simply invoke the AWS Lambda function as a custom resource using the same template as the RDS instance.

The Lambda function can retrieve the “root” credentials from Secrets Manager. It can use those credentials to connect to the database, create it, and load the initial schema. Next, it will create the user and the grants that the user needs. You don’t need to assign a password if you plan to use IAM. If you don’t want to use IAM, you should create the credentials in Secrets Manager and pass them dynamically into the custom resource. Afterward, your user is ready to use your application.

Provisioning RDS users cross-stack

But what if you have multiple stacks and want to use the RDS instance created in a different stack? The same principle applies. You still need to create the provisioner with the RDS instance. But instead of creating the custom resource in the template of the RDS instance, you create it in the other template. For this, you need the ARN of the Lambda function. You can use a naming schema or store the ARN in an SSM Parameter.

The advantage of this is that you can invoke the creation of database users in the template where they are used. Plus, if you make the provisioner smart enough, you can pass on the grants the user will need. This way, you encapsulate the user, the rights, and the application into the same template. This will reduce the maintenance load on your application and its infrastructure.

Schema migrations

One of the other advantages of having a provisioner is that you can also manage schema migrations. Mutating the schema can be invoked with the same ease as the user creation. This gives you the ability to perform these migrations during the AWS CloudFormation deployments. You do need to make sure that the application is backward compatible.

Conclusion

Splitting infrastructure into multiple stacks keeps things organized, but it also introduces challenges like managing database access and schema changes. Using a Lambda function as a provisioner, you can automate user creation and schema migrations as part of your CloudFormation deployments. This approach keeps everything tightly scoped, ensures the least privilege principle, and reduces operational overhead. Whether you use IAM authentication or local users, making the provisioner smart enough to handle both ensures flexibility. With schema migrations baked into your deployments, you get a reliable and repeatable way to evolve your database without manual intervention.

Photo by Muhammed Ensar

Securing Amazon S3 Downloads with ALB and Cognito Authentication

Tue, 25 Feb 2025 10:00:00 +0100

Securing an endpoint used to be hard. Nowadays, with the cloud, it’s quite easy. You only need to know how! Assume you have files on Amazon S3 that you like to share. You could make the object publicly available. This would allow your users to download the file using their browsers simply. If you need to scale it, you can add Amazon CloudFront. This would cache the content closer to your users, making sure that your users have the best performance. But what if you want to control who can download the file? For this, you will need authentication and authorization.

Authentication vs Authorization

Authentication is all about identifying who you are. First, we need to make sure that we know who the user is. Once we establish who the user is, we can see if the user can access the content. The latter is authorization. AWS has a service called Amazon Cognito that allows you to manage a pool of users. These users can originate from other sources like Google, Facebook, and your own identity provider. Or, if you don’t want to use identity providers, you can also create users directly in the user pool.

You can also create groups, and based on these groups, you can manage the authorization. For example, you could make a group called developers. All users within this group should be allowed to fetch the build report hosted on S3.

Building the endpoint

With the Cognito User Pool in place, we need a way to validate the user during the request. I am using an Application Load Balancer to invoke a AWS Lambda function. This function can then check if the user can access the report. The logic is quite simple: if the user is part of the developer’s group, the user can read the report.

In this case, we can use the native Cognito integration of Elastic Load Balancing with the Application Load Balancer. What this will do is the following:

If the user is unauthenticated, navigate to the Cognito-hosted UI. If you use your identity provider, you will be redirected to your company’s login page. If you host the users from the user pool, a login form is shown to the user. After the user has logged a redirect, the user is now authenticated. The load balancer will now invoke the target group with the request.

We also want to check if the user is in the developer group. We will use a Lambda function to check this. The following code would do the trick:

import base64
import json
from typing import List


def decode(data: str) -> dict:
    return json.loads(base64.b64decode(data.split('.')[1]).decode('utf-8'))


def resolve_groups(groups: str) -> List[str]:
    return list(map(lambda group: group.strip(), groups[1:-1].split(',')))


def handler(event, context) -> dict:
    code = 403
    description = "403 Access Denied"
    body = "Access Denied"
    user = decode(event["headers"]["x-amzn-oidc-data"])
    groups = resolve_groups(user.get('custom:groups', '[]'))

    if 'developers' in groups:
        code = 200
        description = "200 OK"
        body = f"Hi {user.get('name')}, you should be able to download the report"

    return {
        "statusCode": code,
        "statusDescription": description,
        "isBase64Encoded": False,
        "headers": {"Content-Type": "json; charset=utf-8"},
        "body": body,
    }

The listener on the application load balancer and the user pool client can be configured as follows:

  Listener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn: !Ref LoadBalancer
      Port: 443
      Protocol: HTTPS
      Certificates:
        - CertificateArn: arn:aws:acm:eu-west-1:111122223333:certificate/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      DefaultActions:
        - AuthenticateCognitoConfig:
            OnUnauthenticatedRequest: authenticate
            Scope: openid
            UserPoolArn: arn:aws:cognito-idp:eu-west-1:111122223333:userpool/eu-west-1_xXXXxxxx
            UserPoolClientId: !Ref UserPoolClient
          Order: 1
          Type: authenticate-cognito
        - Order: 2
          TargetGroupArn:
            Ref: LambdaTarget
          Type: forward


  UserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      AllowedOAuthFlows:
        - code
      AllowedOAuthFlowsUserPoolClient: true
      AllowedOAuthScopes:
        - profile
        - phone
        - email
        - openid
        - aws.cognito.signin.user.admin
      CallbackURLs:
        - https://<MyDomainName>/oauth2/idpresponse
      ClientName: MyClient
      GenerateSecret: true
      LogoutURLs:
        - https://<MyDomainName>/logout
      SupportedIdentityProviders:
        - COGNITO
      UserPoolId: eu-west-1_xXXXxxxx

You also need to setup the lambda function as followed:

  LambdaSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      SecurityGroupEgress:
        - CidrIp: 0.0.0.0/0
          Description: Allow all outbound traffic by default
          IpProtocol: "-1"
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          Description: Allow HTTPS access
          FromPort: 443
          IpProtocol: tcp
          ToPort: 443
      VpcId: !Ref VPCAsParameter

  LambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
      ManagedPolicyArns:
        - !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
        - !Sub arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole

  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: my-bucket
        S3Key: path/to/code.zip
      Handler: index.handler
      Role: !GetAtt LambdaRole.Arn
      Runtime: python3.12
      VpcConfig:
        SecurityGroupIds:
          - !GetAtt LambdaSecurityGroup.GroupId
        SubnetIds:
          - !Ref Subnet1
          - !Ref Subnet2
          - !Ref Subnet3

  LambdaPermissions:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt LambdaFunction.Arn
      Principal: elasticloadbalancing.amazonaws.com

  LambdaTarget:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    DependsOn:
      - LambdaPermissions
    Properties:
      TargetType: lambda
      Targets:
        - Id: !GetAtt LambdaFunction.Arn

You will be prompted to log in when you navigate to the load balancer. Afterward, you can see that the lambda function was invoked. This is a very simple example, but the idea is that you can extend the lambda function with the logic you need. For example, you could create a pre-signed URL for the report on S3 and redirect the user directly to that URL. This will then download the report automatically.

Conclusion

Securing access to your Amazon S3 files doesn’t have to be complicated. By leveraging AWS Cognito, an Application Load Balancer, and a simple Lambda function, you can control exactly who gets access to your files—without exposing them to the public. With this setup, authentication is handled seamlessly, and authorization is as simple as checking group memberships. From here, you can expand the functionality further, such as generating pre-signed URLs for downloads or adding more granular permissions. The cloud makes it easy—you just need to know how!

Photo by Pixabay

Become a documentation ninja

Tue, 18 Feb 2025 10:00:00 +0100

Writing documentation sucks! But there are ways to make it easier and maybe even fun! In one of my previous blogs, I explained how you can embed your documentation in your pull requests and why you should consider doing it, too. In this blog, I want to go a bit deeper into the syntax you can use to improve your documentation pages on Confluence.

Syntax? You mean Markdown, right?

Yes, Markdown is the documentation syntax that Mark uses. However, it has some extensions that are still valid in Markdown format and allow you to use the macros from Confluence.

Linking to other pages

When you write documentation, you probably have different pages explaining different things. You can logically split areas, making it easier to navigate and read. But sometimes, you want to link from one page to another. The problem is that you don’t know the URL yet since that is a Confluence page. You could publish the page and look up the link, which kills the motivation to write documentation. Assume we have a page like this:

# My Confluence Page

My description of the page

And we want to link to this page. You can simply do this with the following syntax:

Read this on [My Confluence Page](ac:) or [read](ac:My Confluence Page) the page.

As you can see, you can link to the page title itself. Or, you can use a custom word, like “read” and link to the page. When mark uploads the page to Confluence the links will be resolved automatically.

Linking to Jira issues

Next to linking to pages, you will probably also refer to Jira issues. However, you could link to Jira issues using native markdown. When you use the Jira reference macro, the type, title, and status of the issue are shown. This gives immediate feedback to the reader on what the status is. What you need to do is to define a macro in the top of your markdown file and mention the issue as plain text. Here is an example:

<!-- Macro: MYJIRA-\d+
     Template: ac:jira:ticket
     Ticket: ${0} →

# My page that contains a Jira ticket reference

See task MYJIRA-123.

In this example, MYJIRA is the Jira project code. The macro definition matches everything that starts with MYJIRA- and is followed by a set of digits. You can then simply mention the issues in your markdown file, and they will be converted into Jira macros automatically.

Hide code snippets in collapsable macros

Especially when you write documentation, you may need to hide parts of the code snippets. One example that I use is for OS-dependent instructions. Assume you want to provide instructions on using your product on the command line. In that case, you could give instructions for Windows, Linux, and MacOS. To use this, you first need to define the macro. Then, use the macro to build the collapsable snippet.

<!-- Macro: :expand-start:(?s)(.*?):\n(.*?)\n:expand-end:
     Template: ac:expand
     Title: ${1}
     Body: ${2}  -->
# My page with command line instructions

Before you start, you must ensure you set the environment variable:

:expand-start:Unix:

Unix instructions go here

:expand-end:

:expand-start:Windows:

Windows instructions go here

:expand-end:

Page properties and status badges

This is my favorite! This is such a powerful feature of Confluence. For those of you who are not familiar with this functionality, you define a table, which contains a set of keys and values. For example, you can specify an owner and a date in this table. You also need to set a label on the page. This is required for the report page. The label is used to make a list of pages that have this label. But the cool thing is that you can also display the values of the page properties.

As you can see in the screenshot above, we have three pages. The parent page holds the report, and the child pages are Project 1 and Project 2. However, it also shows the date, owner, and status badge. This is a great way to give a nice overview of projects.

To create the page properties on the child page, you need the following syntax.

<!-- Parent: Projects -->
<!-- Label: project -->
<!-- Macro: :in-progress:
     Template: ac:status
     Title: In Progress
     Color: Blue -->

# Project 1

<ac:structured-macro ac:name="details" ac:schema-version="1">
    <ac:rich-text-body>
        <table>
            <tbody>
                <tr><th><strong>Status</strong></th><td>:in-progress:</td></tr>
                <tr><th><strong>Owner</strong></th><td>Joris Conijn</td></tr>
                <tr><th><strong>Date</strong></th><td>2025-02-15</td></tr>
            </tbody>
        </table>
    </ac:rich-text-body>
</ac:structured-macro>

The Project 2 markdown file is similar to the one above. However, the parent page has a different definition.

# Projects

<!-- Include: ac:detailssummary
     SortBy: Title
     CQL: 'space = "Projects" and label = "project"' -->

In this example, my confluence space is called Projects. The Confluence Query Language (CQL) retrieves all pages in the Projects space that have the label project. A third requirement is that the page also needs to have the page properties defined. We did this on Projects 1 and 2, so the two projects are rendered in the table.

Conclusion

Writing documentation doesn’t have to be a chore. By leveraging Confluence macros within Markdown, you can enhance your documentation with dynamic links, Jira issue references, collapsible content, and structured page properties. These features make your documentation more interactive, easier to maintain, and more useful for your team.

By embedding these enhancements directly in your markdown files, you streamline the process of writing and updating documentation, reducing friction and improving adoption. So, next time you document a feature or process, try incorporating these techniques. You might find documentation efficient and enjoyable!

Photo by cottonbro studio

ECS Fargate Persistent Storage: EFS Access Points vs. Lambda Workarounds

Tue, 04 Feb 2025 10:00:00 +0100

When running a Docker container on Amazon Elastic Container Service and AWS Fargate, persistent storage is often a necessity. I initially attempted to solve this by manually creating the required directory on Amazon Elastic File System using a AWS Lambda backed custom resource. While this worked, it introduced unnecessary complexity. Through experimentation, I discovered a more elegant solution—using EFS access points. In this post, I’ll walk through my journey, the challenges I faced, and how I ultimately simplified the setup with fewer resources and less maintenance.

Setting up EFS

When you start a container on AWS Fargate, you must define a TaskDefinition. This definition is used to start the container. Within this definition, you can define Volumes. These volumes can then be mounted in the container by specifying a mount point per container.

So far, so good! But I quickly ran into an issue. The task could not be started because the path on EFS did not yet exist. I tried to mount /my-task/data on /data in the container. However, the /my-task/data path does not exist on the EFS drive. The easy solution would be to hook it up to an Amazon EC2 instance and create the folder from there. You might think that the problem is solved, but this is not the solution to the underlying problem. As you could have read in my previous blog, I like to be able to spin up multiple versions of a stack. With this manual step in between, you can’t do that.

How about a custom resource?

We know we need to create a folder on the EFS drive. A Lambda function could do this, so I started implementing a custom resource. The idea is very simple: create a Lambda function that mounts the root of the EFS, and then the Lambda code would simply make the folder on EFS. We mark this custom resource as a dependency on the AWS Fargate service to ensure that the folder exists before we start the container and we are done.

I fully implemented this flow, and it worked perfectly. When you deploy your stack, the custom resource does its thing, and the container can be launched successfully.

Is there a better way?

When implementing the custom resource, I needed to create an access point. A Lambda function can only mount an EFS drive through an access point. This access point allows you to define a path on EFS and the user and group ID used to access this path. If you supply /my-lambda/ as the path, the function can only see the content of the my-lambda folder. This is great for protecting other paths from being accessed. From the lambda functions perspective, you don’t see the my-lambda folder you only see the content.

Why is this so important, you might ask? Well, this path is created on the EFS drive for you. I looked at the ECS documentation, and you can also use an access point for container volumes. Using the access point to define the /my-task/data, I set the uid and gid to the same ID the user executed in the container. We could run the container without the need for a custom resource, so I deleted the function from my template.

Conclusion

Experimentation often leads to better solutions. My initial approach—using a Lambda function to create the necessary EFS directory—worked but added unnecessary complexity. By leveraging EFS access points, I achieved the same result with a cleaner, more maintainable setup. This experience reinforced an important lesson: the simplest solution is often the best. When working with cloud infrastructure, it’s always worth exploring built-in features before resorting to custom implementations.

Photo by Nataliya Vaitkevich

Streamlining Workflows with Feature Branches and Logical Stacks

Sun, 26 Jan 2025 15:00:00 +0100

Efficient collaboration and streamlined deployment processes are essential in modern development workflows, especially for teams working on complex projects. Feature branches and stack-based development approaches offer powerful ways to isolate changes, test effectively, and ensure seamless integration. However, proper strategies can make managing resources, dependencies, and environments challenging. This blog explores how to optimize feature branch workflows, maintain encapsulated logical stacks, and apply best practices like resource naming to improve clarity, scalability, and cost-effectiveness.

Working with feature branches

I have a strong preference for encapsulating things into logical stacks. These stacks should have a minimal number of dependencies. And you should be able to deploy multiple instances. Let me first explain why you want to deploy multiple instances of the same stack. The answer is quite simple! Feature branches! If you work in a team with multiple people, you might work on different stories. Therefore, you don’t want to update the same environment with your changes. My change might break your change and vice versa. Detecting why something failed becomes more challenging in this case. By deploying your own environment, your changes will only impact your environment. When you are done, you can thoroughly test your changes before merging them into the main branch.

Working with multiple stacks

You can use multiple stacks, each with its area of responsibility, to keep things small and simple. However, this also introduces the danger of breaking encapsulation and introducing dependencies, such as an Amazon Relational Database Service - managed SQL database. You need to create it in a base stack and pass it to other stacks. This is needed because the components in the other stacks will need the endpoint to connect to it, creating a dependency.

This example applies to the more traditional lift and shift approaches. However, we always advise modernizing to reap all the benefits of the cloud. Why? Simple: In the example, we needed an RDS instance. If you create an RDS instance per stack, you will have multiple instances and must pay per RDS instance, driving up the cost. By switching to serverless, you pay for the usage. This will allow you to move the database into your stacks, removing the dependency. I explicitly say option because, in relational databases, it’s not always possible to split the database into two because of the nature of relational databases.

Use proper naming for your resources.

Amazon CloudFormation can handle naming resources for you. A naming schema follows this pattern: StackName + LogicalResourceId + RandomizedString. This ensures that you don’t end up with duplicate resource names. Some resources allow you to have duplicates, while others do not. So, having a guaranteed unique name is perfect from the perspective of having feature branches. But from a management perspective, this is bad. Most names are trimmed after several characters, and the generated long names are not that descriptive.

By introducing a prefix, you can have a clear name and still deploy your stacks multiple times. You simply pass the prefix as a parameter and prefix each resource name. So, for an AWS Identity and Access Management role, that could be Joris-CheckoutProcess. The CheckoutProcess name describes what it is, a role used by, for example, a AWS Lambda function that processes the checkout. The prefix Joris makes it unique and provides information about who owns and created the resource. Using this pattern you avoid creating a scavenger hunt.

Conclusion

Adopting feature branches and logical stacks is an excellent way to foster efficient, error-free collaboration in team environments. You can enhance your development workflow by deploying isolated environments for individual changes, embracing serverless solutions to reduce dependencies, and implementing descriptive naming conventions. These practices streamline debugging and testing and ensure resource management remains clear and cost-efficient. Whether you’re modernizing applications or maintaining legacy systems, these strategies will help you maximize benefits of cloud-based development.

Photo by Min An

Stop organizing scavenger hunts in your cloud infrastructure

Sat, 18 Jan 2025 16:00:00 +0100

A CloudWatch alarm is triggered. Now what? I am not the first person to tell you that observability is essential to your cloud infrastructure. You are not done when you have set up CloudWatch alarms!

Who will act on those alarms?

Observing some metrics and raising an alarm if a certain threshold is breached is just the start! You must also consider who will act if this alarm is breached. In some cases, you might be able to automate your actions. For example, you could think of a high CPU load on your application servers. The CloudWatch alarm is then triggered, and you can scale up your cluster.

These auto-remediation actions work very well for all known issues that can go wrong. You can detect them and build remediation actions that you trigger based on these CloudWatch alarms.

But in some cases, you just need a human to jump online and figure out what happened. When this happens, you don’t want to start a scavenger hunt in your AWS account to find the problem. You want a clear starting point and, from there, guidance in the right direction.

CloudWatch Dashboards

CloudWatch dashboards can guide you in the right direction during production issues. For example, assume we have a SQS Queue that contains messages. A lambda function is processing these messages. If the process fails, it will retry, and after three attempts, it will deliver the message to the dead-letter queue.

When one or more messages are in the dead-letter queue, a CloudWatch alarm is triggered, and we know something went wrong in our application.

This alarm can trigger an SNS topic, and from there, you can reach an engineer who can investigate this issue. A dashboard that shows these components makes it easy to navigate to the issue.

Here, you can see the dead-letter queues from our application. MyLambda is in an alarm state, and on the right, you see a link that brings you directly to that function’s LogGroup.

It turns out that someone pushed code with an exception. We fixed the code and started re-driving the messages in the dead-letter queue back to the original queue. The lambda function is now processing the messages as it should, and the alarm will go back to an OK state.

Conclusion

When you build cloud infrastructure, you also need to think about what can go wrong. Design for failure, build auto-remediation for all known failures and loop in a person when unknown issues appear. But be kind for your fellow engineer and don’t sent him on a scavenger hunt. Guide him to problem and give context about the failures in a CloudWatch dashboard.

Photo by Gill Heward.

Boost Your Productivity with awscurl: Simplifying IAM-Secured API Testing in AWS

Sat, 21 Dec 2024 14:00:00 +0100

With modern clouds, you can build awesome things. You can bring your ideas to life within the hour, enabling experimentation and driving innovation. One of the significant advantages of the cloud is that you get a lot of security controls out of the box. But these security controls can also block you from being productive!

How are these security controls blocking me?

When you use AWS, you can interact with it through the console, sdk, or cli. But these all have one thing in common. They all use the same set of APIs to perform the actions requested by the user. These APIs are protected, and how authentication and authorization are done through the service IAM. When you develop a workload or work on a PoC, you will also use the IAM service.

Assume you want to build an internal API and use the API Gateway service. Enabling IAM authentication on the methods you define is easy. You can then give the appropriate roles access to your API. The consumer of this API only needs to add the AWSSigv4 header, and as long as the role policy allows the invocation of the API, it will work.

Various programming languages can be used to add the signature. But what if you want to test the API from your local machine or the cloud shell from the console? In those cases, you have a shell, and generating the signature becomes much harder. This will impact your productivity.

How can we unblock ourselves?

As explained in the previous section, the challenge is to generate the signature based on the current payload. In the past, I used a simple Python script to perform these API calls, but that always took some time and energy to build. I also felt demotivated when I encountered such situations.

The problem is not the script that you need to write. The problem is that you need to do it repeatedly because each situation is different. This goes against one of my principles, which is to automate everything! There should be a solution to this problem, and the good news is there is one!

Igor Okulist created a small tool called awscurl. This tool allows you to perform a curl command that automatically signs your API call. For example, when we use the scenario described in the previous section, you can simply do the following command:

export AWS_PROFILE=my-profile
awscurl --region eu-west-1 --service execute-api -X GET https://my-url-to-my-api

The example assumes that you have a valid profile configured called my-profile. Then, you do the same command as the original curl command. But you only add a region and a service option to the command. For the comparison, the normal curl command would look like this:

curl -X GET https://my-url-to-my-api

But this command will result in the following error:

{"message": "Missing Authentication Token"}

The awscurl command will inject the Authorization header that looks something like this:

Authorization: AWS4-HMAC-SHA256 Credential=XXX, SignedHeaders=XXX, Signature=XXX

When you try this out yourself, you can add the -v option, which will print out the headers sent and received. You will see that the API now accepts the signature and responds with the actual results.

You can use it to perform any API call that supports sigv4, but for the majority of services, the AWS cli tool is the best tool for the job. But for your own APIs or calls to your OpenSearch cluster, this tool is definitely a timesaver.

Conclusion

When you develop an internal API or have an OpenSearch cluster that uses IAM for authentication and authorization, testing calls towards these endpoints can be challenging as you will need to create a signature. The awscurl cli tool helps you create this signature, allowing you to focus on your business value instead of building signatures for tests.

Photo by Pixabay

Relative Python imports in a Dockerized lambda function

Sat, 14 Dec 2024 00:00:00 +0000

Relative Python imports can be tricky for lambda functions. I wrote a blog on this 3 years ago. But recently, I ran into the same issue with Dockerized lambda functions. So, I figured it was time for a new blog!

You can follow along with the steps or look at the result directly on GitHub.

Project setup

Make sure you installed the AWS CDK cli.

brew install aws-cdk

Initialize the project:

cdk init app --language=typescript

Lambda setup

First we will need to create the file and folder structure:

mkdir -p lib/functions/hello-world/hello_world
touch lib/functions/hello-world/hello_world/__init__.py
touch lib/functions/hello-world/hello_world/business_logic.py
touch lib/functions/hello-world/requirements.txt
touch lib/functions/hello-world/Dockerfile

Now you will need to fill the Dockerfile, like this:

FROM public.ecr.aws/lambda/python:3.12
COPY requirements.txt .
COPY hello_world ${LAMBDA_TASK_ROOT}/hello_world
RUN pip install --no-cache-dir -r requirements.txt
CMD ["hello_world.handler"]

We are using a Python base image that is based on Python 3.12. Next, we will copy in the requirements.txt file and the source code. We will install all dependencies listed in the requirements.txt file and make sure that the handler method is set as the CMD.

Next, we will need to fill our Python files with some code. In the __init__.py file, you can place the following content:

from typing import Dict, Any
# Example of the relative import
from .business_logic import business_logic


def handler(event: Dict[str, Any], context: Any) -> Dict[str, str]:
    # User the method that is imported based on a relative path
    name = business_logic(event)

    return {
        "Name": name,
        "Message": f"Hello {name}!",
    }

__all__ = [
    "handler"
]

Afterward, we will need to fill the business_logic.py file:

from typing import Dict, Any


def business_logic(event: Dict[str, Any]) -> str:
    return event.get("name", "World")

NOTE: The code used here could use multiple relative imports. This is possible because it is in a separate package. This example only shows one example in the __init__.py file. However, you can use multiple files here to improve the maintainability of your project.

For this example, I don’t need any dependencies, so we can keep the requirements.txt file empty. I included it in this example to illustrate how you can include dependencies as well.

Create the Lambda function using IaC

Our folders and files are in place, so it is time to add the Lambda function to the CDK construct. You can simply add it like this:

    new lambda.Function(this, 'Function', {
      functionName: "hello-world",
      code: lambda.Code.fromAssetImage("lib/functions/hello-world", {
        platform: ecr_assets.Platform.LINUX_ARM64,
      }),
      runtime: lambda.Runtime.FROM_IMAGE,
      handler: lambda.Handler.FROM_IMAGE,
      architecture: lambda.Architecture.ARM_64,
      timeout: cdk.Duration.seconds(15),
      memorySize: 128,
    });

For this to work, you also need the following imports:

import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as ecr_assets from 'aws-cdk-lib/aws-ecr-assets';

Note that we make sure that the code directory points to the directory that contains the Dockerfile and that we select the ARM platform for both the code and the function itself.

Testing the lambda function locally

Fast feedback is important, so there might be cases where you need to run the container locally. For this, you first need to build the container:

docker build --platform linux/arm64 \
  -t hello-world:latest \
  -f ./lib/functions/hello-world/Dockerfile \
  ./lib/functions/hello-world

Note that this command can be executed from the project’s root. Next, we need to make sure it’s running before we can invoke it:

docker run --platform linux/arm64 -p 9000:8080 hello-world:latest

Afterwards, you can invoke the function as follows:

curl http://localhost:9000/2015-03-31/functions/function/invocations -d '{"name": "Joris"}'

Conclusion

Relative imports can be tricky! You need to place your code in a package. This allows you to do relative imports within your own package. This will enable cleaner code, as you can split responsibilities into multiple files, making it easier to manage and maintain.

Photo by Kaique Rocha

Serverless, it can help you brew beer

Fri, 25 Oct 2024 13:00:00 +0200

I like to combine technology with something more practical. This helps me understand the technology much better. Plus, when you have a practical example, it’s also easier to explain to my wife and friends. During the coronavirus pandemic, I started brewing beer. Brewing beer involves a lot of steps, and the brewer needs to take many steps. But some steps can be automated!

Brewing 101

Before we dive into the automation part, let’s do a quick brewing introduction. To brew a beer, you need the following ingredients:

Malted grains
Hops
Water
Yeast

The grains are crushed just enough to be broken inside. The other layer needs to be somewhat whole. We can then start the mashing process in water around 60 degrees. This will extract the sugars from the grain. When this process is done, you must separate the water from the grain bed. Afterward, you can sparge some additional water through the grain bed to grab the last bits of sugar from the grain.

The product we now have is called wort. Now it’s time to boil the wort. Depending on the recipe, you need to add hops and other ingredients. The general rule for hops is that when you add hops in the beginning, it makes the beer more bitter. Adding more towards the end adds more aroma. The type of hops that you add also plays a role here.

After you boil the wort, it’s time to cool it down. You need to bring the wort back to around 20 degrees. Again, this highly depends on the yeast you are using. When the wort is cool enough, you want to add as much oxygen as possible to it and the yeast.

Close the kettle holding the beer and make sure it has a water lock. The yeast will start eating the sugars in the wort and producing CO2 and alcohol. The CO2 will escape through the water lock, and the alcohol will stay in the wort.

How does Serverless help?

During the fermentation process, the density of the wort changes. It’s slowly changing from wort to beer. The sugars will decrease, and the alcohol will increase over time. But when is this process done?

Density can be measured, so by putting a device in the wort, you can send updates about the density. When you combine this with some cloud expertise, you can build some cool things! Since this device would only send some reading with a value, I don’t need to wait for a response. The API needs to be as fast as possible! When the device waits for confirmation, it consumes more power. Since this device is floating in a fermenting chamber, you can’t hook it up to the power grid or change batteries.

Due to this requirement, I used the API Gateway service from AWS. It allows you to place the incoming payload directly into an SQS Queue. This creates a really fast API for the device floating in my fermenting chamber. Using a queue completely decouples it. A secondary process can just read the messages from the queue and process them accordingly.

How about security

When the device sends its data, it also contains the identifier of the device itself. I used this device ID to link it to a registered device. If the device is unknown, I would just stop processing the message. This would allow you to tie a device to a particular tenant in your solution. However, when you know the endpoint, you can still inject messages. To prevent this from happening, you could use mTLS. The downside in my case is that that has an impact on the battery life. Another other option would be a custom authorizer. This allows you to use a Lambda function to use business logic to decide whether the call can be performed.

In my case, I used the usage plans functionality. The documentation clearly states that you should not use the usage plans for authentication. And I am not! I only use it to limit the pressure on my backend, just in case someone figures out my API endpoint and tries to DDOS me. When the payload is malformed or the device ID is unknown, I will directly drop the messages anyway. So, the impact is limited unless they know the identifiers of my devices.

Conclusion

Real-world examples help illustrate our options for serverless technology. They help you consider other dimensions, like do I have enough battery life? These pros and cons need to be considered during your design. Based on those questions, you might pivot your solution’s architecture.

If you want to know more about what I have automated, make sure you follow me on my social media.

Avoid Costly Loops in AWS Step Functions

Sat, 12 Oct 2024 13:00:00 +0200

We all played around with AWS StepFunctions in our careers. It is a fantastic orchestration tool! But there are some scenarios where your cloud bill can explode in your face! In this blog, I will walk you through how you can end up in this situation and, maybe more importantly, how to avoid it!

Looping

When you look at this pattern, you might think, okay, we wait for 5 seconds and use lambda to query something. If the response is true, we are done; if not, we will wait again and check the status once more. The problem with this pattern is that the response might never be true, causing an endless loop, which will generate cost.

There are no endless loops in StepFunctions

If you find yourself in such an “endless” loop, AWS will stop executing once you have exceeded 25,000 events. Even if you implement it, it will eventually be stopped by AWS. But each loop has 3 transitions, and you must pay for these transactions. They are pretty cheap, but the problem lies within this loop, especially if this pattern is part of a map distribution that allows you to do 10,000 parallel executions.

Assume your StepFunction is invoked based on an S3 event trigger. We upload 100 files in our S3 bucket. This will trigger 100 executions resulting in 3,000,000 state transitions each 5 seconds until they exceed the 25,000 events.

You can already see that this will ramp up your AWS bill.

The callback pattern

You can solve this by implementing a callback pattern. It’s quite simple: You will place a message in an SQS Queue. You can then use any consumer to do the task at hand. When it’s done, the consumer notifies the step function execution and passes the result back.

This is a known pattern, but we design our systems for failure! The cool thing about a SQS Queue is its retry mechanism.

Now, we can design it as follows:

The StepFunction will place the task in the SQS Queue.
The Processor will receive a batch of messages
The function will process each message, and for each, it will:
1. Send a heartbeat to the StepFunction execution. This prevents the StepFunction from timing out.
2. If the message is invalid, we can fail the StepFunction execution directly.
Depending on the outcome, the function will:
1. When a failure occurs, the message will be kept in the queue.
  - This will trigger the retry mechanism based on the visibility settings on the queue.
2. When the message is processed successfully, we will:
  - Send a success to the StepFunction execution.
  - Remove the message from the queue.
When the message has reached the maximum number of retries, it will be delivered to the dead letter queue.
The Dead Letter Queue Processor will receive the message.
It will fail the StepFunction execution with some additional context.
Remove the message from the dead letter queue.

Why is this a good design? It accounts for failure! We left the actual processing job out of scope here. However, the lambda processing can fail for many reasons:

It can reach AWS Service limits.
It might depend on another external system that is unavailable.
You’re scaling so fast that the serverless components can’t handle the load yet.

You have a better chance of succeeding with the SQS Queue and the retry mechanism. If the retries are exhausted, the dead letter queue will mark the failure in the step function itself. This will keep all the execution information in a single place for you to analyze.

Conclusion

Avoid creating loops in your StepFunctions. Use the callback pattern instead! Design your flows to be robust and expect failures to happen!

Photo by Steve Johnson

Make writing documentation part of your pull request

Wed, 25 Sep 2024 20:00:00 +0200

We are all guilty of this one! We changed something, and we forgot to update the documentation. Even if you have the best intentions to do your best, it’s way too easy not to do it.

Why is it so easy to skip?

Assume you changed something, created a pull request, and followed all the defined processes before you can merge the change. When is the best time to update the documentation in this process? If you update the documentation while the pull request is open, the documentation is ahead. It refers to a state that might not be true until you merge your pull request. What if you decide not to merge? Will you remember to revert the documentation?

The other way around is also far from ideal. Before you are ready to merge the pull request, it might take some iterations. Your peers give feedback and comments. So, by the time you have your pull request ready to be merged, you are already mentally done with it. So you need discipline to do the extra step and write the documentation.

Of course, you can make this part of a definition of done and create all sorts of processes around it. But these highly rely on the professionalism of the team itself.

Make it part of your pull request.

I began my career as a software engineer, where I learned concepts like encapsulation. The idea is simple: You place all the things that belong to each other as close as possible.

Let’s apply this concept to the documentation of our solution. We can place the documentation in our git repository and change the logic and behavior in a pull request. The developer can also include any documentation changes in this same pull request. For the reviewer, it becomes easier. They see the code changes and the documentation changes, and they can see the correlation between them. When a developer forgets to update the documentation, peers can ask for the documentation changes as well. So, it’s a win-win situation already!

But my audience does not use git!

Most companies that I worked for use Confluence for their documentation. However, this does not have to stop you from using git! You can sync your Markdown-based documentation to Confluence using mark!

In this example, I use the GitHub actions to perform the sync action when we push to our main branch:

name: Deploy the documentation to confluence
on:
  push:
    branches:
      - main
jobs:
  Documentation:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 2
      - name: Setup Golang
        uses: actions/setup-go@v5
        with:
          go-version: '^1.21.13'
          cache-dependency-path: "**/go.sum"
      - name: Install dependencies
        run: go install github.com/kovetskiy/mark@latest
      - name: Sync Documentation
        env:
          MARK_SPACE: MySpace
          MARK_BASE_URL: https://xebia.atlassian.net/wiki
          MARK_USERNAME: ${{ secrets.CONFLUENCE_USERNAME }}
          MARK_PASSWORD: ${{ secrets.CONFLUENCE_PASSWORD }}
        run: |
          MESSAGE=$(git log -1 --pretty=format:"%s (by %an in %h)")
          UPDATED_DOC_FILES=$(git diff-tree --no-commit-id --name-only -r HEAD --root docs | grep '\.md$' || true)
          for FILE in ${UPDATED_DOC_FILES}; do
            if [[ -e "${FILE}" ]]; then
              echo "> Sync ${FILE}";
              mark --version-message="${MESSAGE}" --drop-h1 --title-from-h1 -f ${FILE} || exit 1; echo;
            else
              echo "> Skipping ${FILE} as it was deleted in the last commit";
            fi
          done

Let’s break it down into smaller steps:

Check out the code with a depth of 2. This enables us to detect the changes made in the last commit.
Ensure we have a Golang environment, and we will cache the dependencies to speed up the build process.
Installing mark into the environment.
Grab the last commit message.
Grab Markdown files from the last commit that have been updated.
Sync each file that has been changed, use the commit message in the confluence version management.

Pretty cool, right? Now, when the pull request is merged, the changes made, including the documentation, are applied when they are merged! Unfortunately, this tool does not support deleting pages.

Conclusion

By combining the documentation with your code, you will encapsulate them. This will make it easier to keep them up to date and thus improve the overall quality.

Want to become a documentation ninja, read my follow-up blog?

Photo by Pixabay

Abuse CDK functionality in CloudFormation

Wed, 18 Sep 2024 19:42:00 +0200

Each “Infrastructure as Code” framework has pros and cons. My go-to framework is CloudFormation, but I like CDK’s tree view, so I started thinking about borrowing this.

What is the tree view?

AWS announced this functionality on September the 14th, 2022.

Customers using CDK want a simple way to map the resources synthesized in a CloudFormation template back to the source CDK Construct.

Normally, the Resources tab looks something like this:

But when you deploy a CDK stack, you get a new option, the Tree view! When you navigate to the stack in the console, you will see the following:

How is CDK using it?

When you write your constructs, each resource will be nested in the construct. You can use a construct inside of a construct. This will result in a nested tree structure like this:

As you might have noticed, there is some unneeded nesting here. At the root level, we have only one entry, the environment name from the pipeline. Then, we have the name of the stack that we use. Going back to the intention of this feature, it makes sense, but we are looking at a stack deployment in CloudFormation. So yes, this will be in the development account we already knew because we logged into it. Next, the stack name, we needed to click the stack name to get to this view. From that point of view, it has no value in showing it here.

The third level becomes more interesting. Here are the constructs that we use inside the stack itself. On this level, CDK does a pretty good job of organizing the resources into logical nested groups. A bucket policy applies to a bucket, so it’s nested under the bucket itself, for example.

How can we abuse it?

When you look at the resource definitions in the synthesized version, you can see how it is done. The bucket will contain some additional metadata like this:

ApplicationBucketPolicy:
 Type: AWS::S3::BucketPolicy
 Metadata:
   aws:cdk:path: Development/AppStack/StaticWebsiteConstruct/Bucket/BucketPolicy/Resource
 Properties:

You can play around with this and organize the resources any way you want. Here is an example that I did to experiment with:

As you can see, this is organized more logically. We have the following major components:

Database
DNS
Authentication
API

Going into the Authentication, you can see that I have some triggers. Diving deeper, you can see what kind of triggers. From there, you will have the resources needed for the operation, like permissions, log groups, and the function itself.

Known requirements

While playing around with the organization of this tree view, I discovered the following requirements:

The value in the aws:cdk:path must be unique across the processed template!
When you use the AWS Serverless Application Model, you need to ensure that the resource definition does not result in multiple resources. (This will lead to multiple values being the same, which will break the rendering of the tree view.)
When you omit the /Resource at the end, the LogicalId in the template is used. This could lead to uglier names in the tree view.

Conclusion

If you like organizing your CloudFormation templates, check out the tree view. It might have been built for CDK, but you can use it directly in your CloudFormation templates too.

Photo by Pixabay

Share your configuration across AWS accounts

Sat, 31 Aug 2024 10:00:00 +0000

It’s a best practice to split your environments into separate AWS accounts. It does not cost you anything extra and you can guarantee that your development environment is completely isolated from the higher environments. With this setup, it’s also common to have a build or tooling account to be responsible for the deployment of your application.

But it does bring some other challenges, for example: When you have a web application it usually has a frontend and an API that it uses. If you use separate pipelines to deploy these components you need a way to share the endpoint. You could use a predictable schema to determine the endpoint programmatically when you build the web application. This works well for a URL, but this does not work for a Cognito Client ID.

Use SSM Parameters

SSM Parameters are a key/value store, they are perfect for configuration values. When you deploy your API, Cognito user pool and client. You can store the value in an SSM parameter like this:

UserPoolClientIdParameter:
 Type: AWS::SSM::Parameter
 Properties:
   Name: /<Environment>/cognito/user-pool-client-id
   Tier: Advanced
   Type: String
   Value: !Ref AppClient

Next, you can share this parameter with the build/tooling account:

ShareUserPoolClientIdParameter:
 Type: AWS::RAM::ResourceShare
 Properties:
   AllowExternalPrincipals: true
   Name: UserPool Client ID - <Environment>
   PermissionArns:
     - arn:aws:ram::aws:permission/AWSRAMDefaultPermissionSSMParameterReadOnly
   Principals: 
     # This is the account id of your build/tooling account
     - 111122223333
   ResourceArns:
     - !Sub arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${UserPoolClientIdParameter}

Resolve the values in CodeBuild

Now that we have shared the configuration with the build/tooling account. We need to resolve the value. First, make sure that the principle is allowed to resolve the parameter:

{
  "Effect": "Allow",
  "Action": [
    "ssm:GetParameter",
    "ssm:GetParameters"
  ],
  "Resource": [
    "arn:aws:ssm:eu-west-1:444455556666:parameter/<Environment>/cognito/user-pool-client-id"
  ]
}

Next, we need to add the parameter to the buildspec:

version: 0.2
env:
  parameter-store:
    USER_POOL_CLIENT_ID: arn:aws:ssm:eu-west-1:444455556666:parameter/<Environment>/cognito/user-pool-client-id
phases:
   ...

Now in theory this should work… However, this parameter will not resolve. I assume that CodeBuild can only resolve parameters stored in the same account. But you can simply work around this issue by fetching the parameter yourself:

export USER_POOL_CLIENT_ID=$(aws ssm get-parameter --name arn:aws:ssm:eu-west-1:444455556666:parameter/<Environment>/cognito/user-pool-client-id --output text --query 'Parameter.Value')

I preferred the native solution but unfortunately, this is not working. Until it is this would be my recommended solution.

Conclusion

Following best practices is always a good idea. But they can introduce new challenges. In the example given, I have shown you how to use an SSM parameter to share configuration across accounts.

Photo by Mostafa Meraji

How to Speedup Your CDK Test Feedback

Wed, 08 May 2024 19:00:00 +0200

When you deploy lambda functions using CDK and a test-driven approach, you might have noticed that the test feedback takes longer each time you add a new function. That’s because each function will be bundled with its own set of dependencies. These dependencies need to be downloaded before CDK can bundle them into a single package.

CDK Synth

Before you can deploy your infrastructure you need to synthesize your code. This will transform your code into a CloudFormation template plus some assets. When you have a lambda function the content of this function will be bundled into one of these assets.

This all makes sense when you deploy your code. You need to ship the actual lambda code to AWS in the correct format with the correct dependencies. Otherwise, your solution will not work.

But the synth step is also used when you test your infrastructure code. The process is like the deploy step. Instead of deploying your templates, you will inspect the outcome. For example, you are expecting a Lambda function. This function will use a certain runtime. And, for production environments, you want to confirm that the backup tags are applied.

Skip the bundling when you run the tests!

It might be an obvious answer. Skip the bundling when you run the tests! Assume we use the following code to create a Lambda function:

const sampleFunction = new lambda.Function(this, 'SampleFunction', {
  code: lambda.Code.fromAsset('./sample_function/', {
    bundling: {
      command: ['bash', '-c', 'pip install --target /asset-output -r ./requirements.txt && cp -au . /asset-output'],
      image: lambda.Runtime.PYTHON_3_12.bundlingImage,
    },
  }),
  runtime: lambda.Runtime.PYTHON_3_12,
  handler: 'index.handler',
  timeout: cdk.Duration.seconds(60),
  memorySize: 1024,
});

You can see that we are loading the code from a folder called sample_function. Then we pass the options on how CDK needs to bundle the function:

First, the dependencies will be downloaded and installed.
Next, the content from the sample_function is copied.
The content of the asset_output folder will be the bundled function.

Now, these steps will become part of the test. You can skip the bundling by defining the test as followed:

import * as cdk from 'aws-cdk-lib';
import { Template } from 'aws-cdk-lib/assertions';
import { AppStack } from './AppStack';


test('Sample unit test on how to skip bundling', () => {
 // Setup the application
 const app = new cdk.App({
   context: {
     'aws:cdk:bundling-stacks': [],
   },
 });


 // Setup the CDK stack
 const stack = new AppStack(app, 'MyTestStack', {
   backups: true, // For production we enable backups
 });


 // Synth the code
 const template = Template.fromStack(stack);


 // Confirm that the Lambda function uses python 3.12 runtime
 template.hasResourceProperties('AWS::Lambda::Function', {
   Runtime: 'python3.12',
 });


 // Confirm that the DynamoDB table has the Backup tag
 template.hasResourceProperties('AWS::DynamoDB::Table', {
   Tags: [
     {
       Key: 'Backup',
       Value: 'YES',
     },
   ],
 });
});

A few things are happening in this example:

When the app object is being created, some specific context is supplied. This context will overwrite the aws:cdk:bundling-stacks option. This means that the bundling options are overwritten with nothing.
Then the stack object is created and a template is synthesized.
We will confirm that we have a lambda function with the Python 3.12 runtime.
We will also confirm that the DynamoDB table has the Backup tag with the value YES.

Because we skipped the bundling the test will execute immediately. Giving you instant feedback on the code that you have been writing. You can now run the test after each change. This enables you to create a nice development cycle:

Adding a test implementation.
Adding the business logic.
Running the tests to confirm the implementation.
Repeat

Why is it ok to skip the bundling?

When you write tests you should confirm your business logic. The bundling logic has already been tested by the team that is maintaining CDK. So you don’t need to test it and it’s safe to skip the bundling.

You could argue that the options in the bundling need validation too. And yes, that is now skipped. Yet the bundling of these functions is usually always the same for each runtime. And when you deploy you will catch any bundling mistakes anyway. Next to that, you have tests for your lambda code as well. These tests are not executed against the bundled version of your code. These are executed against the source version of your code. So there is no need to bundle them from a testing perspective.

Conclusion

When it takes too long to run tests, you will run them less. This has a negative impact on your quality and efficiency. You can solve this by excluding heavy operations. Operations like downloading dependencies and bundling your lambda functions. This also helps with focussing on the tests that matter. The tests that actually confirm your business value.

Follow me if you want to learn more about improving your efficiency and code quality.

Photo by Pixabay

Use CloudWatch LogGroups for EC2 logging

Wed, 10 Apr 2024 10:00:00 +0000

You can protect yourself from losing logs on Amazon EC2 by using CloudWatch Logs. Configure the CloudWatch Agent to stream your logs to a LogGroup. This protects you from losing logs. For example, when the instance is replaced by autoscaling. You are also protected against tampering of the logs. An attacker who has gained access to your system can remove the logs. But the logs in the LogGroup will contain the original log lines.

Preparations

We will need the following resources:

 InstanceSecurityGroup:
   Type: AWS::EC2::SecurityGroup
   Properties:
     GroupDescription: Security group for the test instance
     VpcId: "{{resolve:ssm:/landingzone/vpc/vpc-id}}"
     SecurityGroupEgress:
       - Description: Allow outbound connectivity to port 443.
         IpProtocol: tcp
         FromPort: 443
         ToPort: 443
         CidrIp: 0.0.0.0/0

 Role:
   Type: AWS::IAM::Role
   Properties:
     PermissionsBoundary: !Sub arn:aws:iam::${AWS::AccountId}:policy/landingzone-workload-permissions-boundary
     AssumeRolePolicyDocument:
       Version: 2012-10-17
       Statement:
         - Effect: Allow
           Action: sts:AssumeRole
           Principal:
             Service: ec2.amazonaws.com
     ManagedPolicyArns:
       - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
       - arn:aws:iam::aws:policy/AmazonInspector2ManagedCispolicy
       - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

 InstanceProfile:
   Type: AWS::IAM::InstanceProfile
   Properties:
     Roles:
       - !Ref Role

 LogGroup:
   Type: AWS::Logs::LogGroup
   Properties:
     LogGroupName: !Sub ${AWS::StackName}-Instance
     RetentionInDays: !!int 365

InstanceSecurityGroup, allows an outbound connection on 443. The CloudWatch Agent needs to stream the logs to the CloudWatch endpoint.
Role, holds the policies that allow the instance to send the logs to the LogGroup.
InstanceProfile, the profile used to attach the role to the EC2 instances.
LogGroup, the LogGroup used to store the logs in.

Setting up the EC2 Instance

In my previous blog I wrote about how you can create an EC2 instance using Infrastructure as Code. We will continue on that example. Instead of a single instance we will create an autoscaling group.

First we will need a LaunchTemplate. This LaunchTemplate will contain some metadata called AWS::CloudFormation::Init. The metadata has so called configSets and blocks of config. Here is an example that we will use:

 LaunchTemplate:
   Type: AWS::EC2::LaunchTemplate
   Metadata:
     AWS::CloudFormation::Init:
       configSets:
         default:
           - CloudFormationInit
           - CloudWatchLogs
       CloudFormationInit:
         files:
           /etc/cfn/cfn-hup.conf:
             owner: root
             group: root
             mode: 000400
             content: !Sub |-
               [main]
               stack=${AWS::StackId}
               region=${AWS::Region}
           /etc/cfn/hooks.d/cfn-auto-reloader.conf:
             owner: root
             group: root
             mode: 000400
             content: !Sub |-
               [cfn-auto-reloader-hook]
               triggers=post.update
               path=Resources.LaunchTemplate.Metadata.AWS::CloudFormation::Init
               action=https://conijn.io/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --region ${AWS::Region} --resource LaunchTemplate
               runas=root
         services:
           sysvinit:
             cfn-hup:
               enabled: true
               ensureRunning: true
               files:
                 - /etc/cfn/cfn-hup.conf
                 - /etc/cfn/hooks.d/cfn-auto-reloader.conf
       CloudWatchLogs:
         packages:
           yum:
             amazon-cloudwatch-agent: []
         files:
           /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json:
             owner: root
             group: root
             mode: 000400
             content: !Sub |-
               {
                 "agent": {
                   "region": "${AWS::Region}",
                   "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
                   "debug": false
                 },
                 "logs": {
                   "logs_collected": {
                     "files": {
                       "collect_list": [
                         {
                           "file_path": "/var/log/user-data.log",
                           "log_group_name": "${LogGroup}",
                           "log_stream_name": "{instance_id}/user-data.log"
                         },
                         {
                           "file_path": "/var/log/cfn-hup.log",
                           "log_group_name": "${LogGroup}",
                           "log_stream_name": "{instance_id}/cfn-hup.log"
                         },
                         {
                           "file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
                           "log_group_name": "${LogGroup}",
                           "log_stream_name": "{instance_id}/amazon-cloudwatch-agent.log"
                         }
                       ]
                     }
                   },
                   "log_stream_name": "default_log_stream"
                 },
                 "metrics": {
                   "append_dimensions": {
                     "AutoScalingGroupName": "${!aws:AutoScalingGroupName}"
                   },
                   "metrics_collected": {
                     "disk": {
                       "measurement": [
                         "used_percent"
                       ],
                       "metrics_collection_interval": 60,
                       "resources": [
                         "/"
                       ]
                     }
                   }
                 }
               }
         services:
           sysvinit:
             amazon-cloudwatch-agent:
               enabled: true
               ensureRunning: true
               files:
                 - /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json
   Properties:
     LaunchTemplateData:
       BlockDeviceMappings:
         - DeviceName: /dev/xvda
           Ebs:
             DeleteOnTermination: !!bool true
             Encrypted: !!bool true
             KmsKeyId: !GetAtt EncryptionKey.Arn
             VolumeSize: !!int 32
             VolumeType: gp2
       DisableApiTermination: !!bool true
       IamInstanceProfile:
         Arn: !GetAtt InstanceProfile.Arn
       ImageId: "{{resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2}}"
       InstanceType: t3.micro
       SecurityGroupIds:
         - !Ref InstanceSecurityGroup
       MetadataOptions:
         HttpTokens: required
         InstanceMetadataTags: enabled
       UserData:
         Fn::Base64: !Sub |-
           #!/bin/bash -x
           /opt/aws/bin/cfn-init --stack ${AWS::StackName} --region ${AWS::Region} --resource LaunchTemplate
           /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --region ${AWS::Region} --resource AutoScalingGroup

CloudFormationInit

The CloudFormationInit holds a configuration. This configuration is used by the cfn-init agent. It comes pre-installed with the AmazonLinux2 and AmazonLinux2023 AMIs. Under the files section you can see that we are defining 2 files:

/etc/cfn/cfn-hup.conf
/etc/cfn/hooks.d/cfn-auto-reloader.conf

The syntax is quite explanatory. You set an owner, group, mode and the content of the file. The content of the tells the EC2 instance where it can find the metadata. During the initial boot this is given in the user-data. But you might change the metadata overtime. This makes sure that the running instances checks if there is an update available.

To recap this will make sure that the running EC2 instances stay in sync with the given metadata.

CloudWatchLogs

The Amazon Linux AMIs do not come with the CloudWatch Agent you need to install the agent first. You can do this by providing the amazon-cloudwatch-agent package, under packages.

Now it’s time to configure the agent to stream the log files to CloudWatch LogGroups. We have defined 3 files that will be watched by the agent:

/var/log/user-data.log
/var/log/cfn-hup.log
/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

You can extend this list to your own needs. In this example each EC2 instance will have 3 streams. 1 per file and the instance id is used as a prefix, example: i-0000000000/user-data.log.

You can also do measurements on the instance itself and expose this as a CloudWatch Metric. In this example we are capturing a percentage of the used space of the root volume.

We have now configured the agent. But we also need to make sure it’s running. When changes are made to the configuration file, we also need to restart it. We do this by pointing to the config file. Now cfn-init knows when it changes the configuration that it also needs to restart the agent.

User Data

The user data will make sure that the configuration is applied. Since we configure the cfn-init to check the configuration it will stay in sync from here on. The second thing it will do is it will send a signal to the AutoScalingGroup resource. If the cfn-init is executed successfully it will signal a success and if not it will signal a failure.

This means that if the configuration fails the autoscaling group will receive a failure. As a result it will rollback the stack.

AutoScalingGroup

The autoscaling group looks as followed:

  AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    CreationPolicy:
      ResourceSignal:
        Timeout: PT5M
    UpdatePolicy:
      AutoScalingRollingUpdate:
        MinInstancesInService: 1
        MaxBatchSize: 1
        PauseTime: PT5M
        WaitOnResourceSignals: !!bool true
    Properties:
      LaunchTemplate:
        LaunchTemplateId: !GetAtt LaunchTemplate.LaunchTemplateId
        Version: !GetAtt LaunchTemplate.LatestVersionNumber
      MinSize: !!int 1
      MaxSize: !!int 2
      DesiredCapacity: !!int 1
      VPCZoneIdentifier:
        - "{{resolve:ssm:/landingzone/vpc/private-subnet-1-id}}"
        - "{{resolve:ssm:/landingzone/vpc/private-subnet-2-id}}"

We will have 2 policies defined, one for creation and one for updates. If the signal configured in the user-data fails the resource will fail and trigger a rollback. If the instance is configured successfully it will continue. In the auto scaling group you can define the amount of instances required. In this example the maximum is 2 and the minimum 1. This makes sure that at least one EC2 instance is running. When a replacement is needed a new instance will be added first and if it is deployed successfully the old one is removed. This is why the max is set to 2 so that we can replace the old one gracefully.

Conclusion

Configuring a LogGroup to stream your log files is not hard. Using CloudFormation Init you can install and configure applications. In our example we configured the CloudWatch agent that will stream the logs.

When you have to deal with autoscaling events and/or malicious attackers. You can prevent the loss of logs by centralising your application log files.

Photo by Pixabay

Compliant infrastructure using infrastructure as code

Mon, 18 Mar 2024 10:00:00 +0000

When you are using compute you have a lot of options. One of these options is Amazon EC2. In a world where more and more workloads become serverless. You might still have this use-case that is better off on EC2. But, how do you combine EC2 with compliance and security? In this blog post we will explore how we can build a compliant and secure EC2 stack.

Compliance in AWS

When we talk about compliance we are actually saying AWS Config. AWS Config is the service that enables you to confirm if the infrastructure is compliant. I wrote a blog in the past on how you can write your own rules. But out of the box AWS has the AWS Foundational Security Best Practices v1.0.0. This standard comes with a few config rules for EC2:

ec2-instance-multiple-eni-check, this control checks if Amazon EC2 instance uses multiple ENI/EFA. This control will pass if single network adapters is used.
ec2-paravirtual-instance-check, this control checks if the EC2 virtualization type is paravirtual. The control fails for an EC2 instance if virtualizationType is set to paravirtual.
ec2-instance-no-public-ip, this control checks whether EC2 instances have a public IP address. The control fails if the publicIp field is present in the instance configuration. This control applies to IPv4 addresses only.
ec2-imdsv2-check, this control checks if your EC2 instances have IMDSv2 configured. The control passes if HttpTokens is set to required for IMDSv2. The control fails if HttpTokens is set to optional.
ec2-instance-managed-by-ssm, this control checks if the EC2 instances are managed by SSM

The first 3 are easy, with a proper VPC design and using sane defaults you will be compliant. But for the last 2 you need to perform some extra actions.

Security in AWS

Security is a broad topic and can include a lot of different things. In this blog I will only focus on the EC2 instance itself. This means that patch management and hardening is in scope. The service that AWS has for this is called AWS Inspector. This tool will scan the EC2 instance for known vulnerabilities. Inspector will report the findings through the AWS Console. It also integrates with Security Hub. Here you can manage all your security related findings from a single place.

Inspector can also scan your instance against the CIS hardening baseline.

Why should I use infrastructure as code?

When you use IaC (infrastructure as code), you create repeatable and predictable infrastructure. Now why is this important? Assume you create an EC2 instance, and you will use the console to launch it. Now AWS Config will start reporting that IMDSv2 is not enforced. And that the instance is not managed by SSM (AWS Systems Manager). This is because you forgot all those extra steps. Humans are not designed to think about all the options that we need to include. IaC is designed to do this for you, you define a template. You deploy the template and the end result is what is defined in the template.

If you design your templates to be compliant and secure. The infrastructure that you deploy will be compliant and secure.

Let’s build a compliant and secure template!

Now my IaC technology of choice is AWS CloudFormation. So the examples that I will present are built in CloudFormation. But, this can also be achieved in any other IaC technology that you want to use.

Setting up an instance profile

We want the EC2 instance to be managed by SSM. For this we need to create an IAM Role and an Instance Profile. We will attach 2 managed policies:

AmazonSSMManagedInstanceCore, enables the instance to be managed by SSM.
AmazonInspector2ManagedCispolicy, enables the ability to perform the CIS baseline scan.

 Role:
   Type: AWS::IAM::Role
   Properties:
     AssumeRolePolicyDocument:
       Version: 2012-10-17
       Statement:
         - Effect: Allow
           Action: sts:AssumeRole
           Principal:
             Service: ec2.amazonaws.com
     ManagedPolicyArns:
       - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
       - arn:aws:iam::aws:policy/AmazonInspector2ManagedCispolicy


 InstanceProfile:
   Type: AWS::IAM::InstanceProfile
   Properties:
     Roles:
       - !Ref Role

Setting up the EC2 Instance

When you launch an EC2 instance we will be needing a security group. We also want our EC2 instance to be managed by SSM. To SSM Agent comes pre-installed in Amazon Linux 2. Needs to be able to connect outbound via HTTPS, for this reason we add 0.0.0.0/0 on port 443 on the egress traffic.

We also want to make IMDSv2 required. We can configure this in the launch template MetadataOptions. To ensure that all security updates are installed during the first boot. We include the yum update -y --security option in the user-data.

You can now use the launch template to create an EC2 instance. Or, use an auto scaling group to launch one or more EC2 instances.

 InstanceSecurityGroup:
   Type: AWS::EC2::SecurityGroup
   Properties:
     GroupDescription: Security group for the test instance
     VpcId: "{{resolve:ssm:/landingzone/vpc/vpc-id}}"
     SecurityGroupEgress:
       - Description: Allow outbound connectivity to port 443.
         IpProtocol: tcp
         FromPort: 443
         ToPort: 443
         CidrIp: 0.0.0.0/0

 LaunchTemplate:
   Type: AWS::EC2::LaunchTemplate
   Properties:
     LaunchTemplateData:
       IamInstanceProfile:
         Arn: !GetAtt InstanceProfile.Arn
       ImageId: "{{resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2}}"
       InstanceType: t3.micro
       SecurityGroupIds:
         - !Ref InstanceSecurityGroup
       MetadataOptions:
         HttpTokens: required
       UserData:
         Fn::Base64: !Sub |-
           #!/bin/bash -x
           yum update -y --security

 Instance:
   Type: AWS::EC2::Instance
   Properties:
     LaunchTemplate:
       LaunchTemplateId: !GetAtt LaunchTemplate.LaunchTemplateId
       Version: !GetAtt LaunchTemplate.LatestVersionNumber
     SubnetId: "{{resolve:ssm:/landingzone/vpc/private-subnet-1-id}}"

Spinning up the instance

If you combine the two snippets you can spin-up a compliant EC2 instance. This instance has all the latest security patches installed. But when you look into Inspector you will notice that there are still some findings reported.

The findings that popped up during the time of writing were kernel related. So for me to fix this I need to update the kernel. When you lookup your EC2 instance in the console you will notice the Connect button. This will create a shell session to your instance:

Updating the kernel

First, let’s see what kernel we have running:

sh-4.2$ uname -r
4.14.336-253.554.amzn2.x86_64
sh-4.2$

Now let’s see what kernels are available:

sh-4.2$ amazon-linux-extras | grep kernel
49  kernel-5.4               available    [ =stable ]
55  kernel-5.10              available    [ =stable ]
62  kernel-5.15              available    [ =stable ]
sh-4.2$

We will install the latest version of the kernel 5.15 with the following command:

sh-4.2$ sudo amazon-linux-extras install kernel-5.15

Because a kernel update requires a reboot we need to reboot the instance:

sudo reboot

All done!

Avoiding the kernel update

The problem with the kernel update is the required reboot. You only have two options to remove this step:

Create your own up to date base AMI.
Use the Amazon Linux 2023 AMI with a newer kernel version.

Maintaining a base AMI comes with a lot of extra overhead. So my recommendation is to always use the newer version that uses the newer kernel versions.

But when you need to harden your images using the CIS benchmarks. You might need to start maintaining your own images anyway. There are CIS Hardened images available through the marketplace for an extra charge.

Conclusion

IaC (Infrastructure as Code) can help you spin up compliant and secure infrastructure. It removes the human aspect of forgetting to configure certain options. Using IaC you will have a predictable outcome every time you spin up the same template. This makes it reliable and is less time consuming. As a bonus you are not triggering any config rules to be marked as non compliant.

Photo by Hitesh Choudhary

Tracking your security posture in AWS

Fri, 01 Mar 2024 09:00:00 +0000

How do you track your security posture in AWS? You have services like Security Hub, but it will only show you the scores of a limited set of standards. This blog post will continue on the previous 2 blog post that I wrote:

In this blog we will look at how you can track your security posture in your organization.

AWS Security Hub

AWS Security Hub is the service for your cloud security posture management. And this works for smaller organizations and when you don’t have your own config rules. Security Hub will only show you the compliance scores of standards that AWS provides. Plus, it will not give you historical insight of the scores. This does not need to be a problem. Having historical insight can help a lot when you interact with the workload teams.

For example, a workload team could say: “Yesterday the score was fine, it’s only lower because of <insert reason here>”. Having the historical insight will actually tell you if this is true or not.

How about conformance packs?

In my previous blog I also wrote that conformance packs give you this insight. Yes this is true but there is a subtle difference between the scores in AWS Config and AWS Security Hub. In config the state is black and white, a resource is either compliant or non compliant. The calculated score is the percentage of compliant resources.

In AWS Security Hub when 1 resource is non compliant in the control, the control is non compliant. When 9 controls are compliant and you have one that is not. The score would be 90%, regardless of the amount of resources.

Security hub also allows you the ability to suppress the non compliant findings. This is useful where you have a resource that has a good reason to not adhere to a control. You can suppress the finding on the resource. These suppressed findings are excluded when the compliance score is being defined.

How do you calculate the score then?

If you have set up a landingzone you have an audit account. This account collects all config rule results. But also all security hub findings into a single account.

By querying the security hub for findings we can calculate the compliance score. You can create handy dashboards. These dashboards can display all scores of a workload across environments.

This requires an AWS Account per environment and you need a naming schema. For example: xebia-my-workload-development.

When you expose these dashboards to your teams. You will increase the visibility into the compliance scores. Visualized in a single place and easy to access. Giving the product owners the insights they need to plan the backlog. This will increase the adoption of built-in security responsibilities.

How does it work?

For the full implementation details I refer to the github repository. But in essence we will perform the following steps:

Use the filter to retrieve all findings, we are fetching 100 findings per invocation.
When there is a NextToken we need to collect the rest of the findings.
When we have 100 files, containing each 100 findings. We will add them into a single file containing 10K findings. We are repeating this until we have all the findings.
Split the findings per AWS Account ID.
In parallel, calculate the score based on the findings.
Publish the results to CloudWatch metrics. The metric will have the workload name and environment as dimensions.

There are 3 supported flavors:

The security standards from AWS Security Hub.
Conformance packs deployed in the organization.
List of custom AWS Config rules.

For the conformance packs there is one more step. We need to get the number of rules available within the conformance pack. The reason for this is that controls that are compliant are not visible in Security Hub. This would lead to an incorrect calculation of the security score.

Conclusion

When you need one of the following:

Historical insight in yous security posture
Compliance scores of your conformance pack.
Dashboards with the scores of all environments of a workload

You need to do some manual labor as it is not supported by AWS. I hope I peaked your interest to check out the aws-security-posture project.

Photo by ThisIsEngineering

Deploy AWS Config rules across your organization

Thu, 15 Feb 2024 11:00:00 +0100

In my previous blog I showed you how you can write your own config rules. But it will only bring you value if you deploy the rule in your AWS Accounts. In this blog we will dive into the distribution of these config rules.

What are my options?

There are many ways you can deploy these rules in your member accounts. In this blog I will only focus on the 2 real options:

Deploy the rules using CloudFormation StackSets in each account.
Use conformance packs.

Using CloudFormation StackSets

When you look at what CloudFormation StackSets does, you will see the reason why this would be a good fit:

You need to have a delegated administrator. This admin has the ability to deploy the template in other accounts. You define all the account ids and regions you want to deploy the config rules in.

If you combine this with a landingzone you most likely already have this in place.

Using Conformance Packs

A conformance pack is nothing more than a bunch of config rules grouped together. But having this bundle gives you some more benefits. For example, you can see the compliance score of your bundle in your account. You can compare it to the compliance scores that you can see in Security Hub. Assume you have 10 config rules. Now one has resources that are non-compliant you will have a compliance score of 90%.

But this alone would not address the need to distribute the rules across all your accounts. But the conformance packs come in 2 flavors, regular and organizational conformance packs. Regular rules deploy the conformance pack in a single account. The organizational conformance pack will deploy the rule in all accounts.

Note, You can deploy the regular config rules using an organization config rule. This would not give you the grouping of all your rules. For this reason I tend to use conformance packs.

Show me an example

It’s always better to use an example, this snippet will only work when you follow the prerequisites. When you deploy this template in the account that is setup as the delegated administrator. All member accounts will have the config rules available.

Resources:

 ConformancePackBucket:
   Type: AWS::S3::Bucket
   DeletionPolicy: Retain
   UpdateReplacePolicy: Retain
   Properties:
     BucketName: !Sub awsconfigconforms-${AWS::AccountId}-${AWS::Region}
     BucketEncryption:
       ServerSideEncryptionConfiguration:
         - ServerSideEncryptionByDefault:
             SSEAlgorithm: AES256
     PublicAccessBlockConfiguration:
       BlockPublicAcls: True
       BlockPublicPolicy: True
       IgnorePublicAcls: True
       RestrictPublicBuckets: True

 ConformancePackBucketPolicy:
   Type: AWS::S3::BucketPolicy
   Properties:
     Bucket: !Ref ConformancePackBucket
     PolicyDocument:
       Version: 2012-10-17
       Statement:
         - Sid: AWSConfigConformsCheckAcls
           Action: s3:GetBucketAcl
           Effect: Allow
           Resource: !GetAtt ConformancePackBucket.Arn
           Principal:
             AWS: !Sub arn:aws:iam::${AWS::AccountId}:role/aws-service-role/config-conforms.amazonaws.com/AWSServiceRoleForConfigConforms
           Condition:
             StringEquals:
               aws:PrincipalOrgID: ${aws:PrincipalOrgID}
             Bool:
               aws:SecureTransport: true
         - Sid: AWSConfigConformsReadWriteBucket
           Action:
             - s3:PutObject
             - s3:GetObject
           Effect: Allow
           Resource: !Sub ${ConformancePackBucket.Arn}/*
           Principal:
             AWS: !Sub arn:aws:iam::${AWS::AccountId}:role/aws-service-role/config-conforms.amazonaws.com/AWSServiceRoleForConfigConforms
           Condition:
             StringEquals:
               aws:PrincipalOrgID: ${aws:PrincipalOrgID}
             Bool:
               aws:SecureTransport: true

 OrganizationConformancePack:
   Type: AWS::Config::OrganizationConformancePack
   Properties:
     OrganizationConformancePackName: lz-framework
     DeliveryS3Bucket: !Ref ConformancePackBucket
     ExcludedAccounts: [ "000000000000" ] # AccountID to exclude
     TemplateBody: |-
       Resources:
         S3AccessLogging:
           Type: AWS::Config::ConfigRule
           Properties:
             ConfigRuleName: lz-my-custom-rule
             Description: My custom rule
             EvaluationModes:
               - Mode: DETECTIVE
             Scope:
               ComplianceResourceTypes:
                 - AWS::S3::Bucket
             Source:
               Owner: CUSTOM_POLICY
               SourceDetails:
                 - EventSource: aws.config
                   MessageType: ConfigurationItemChangeNotification
                 - EventSource: aws.config
                   MessageType: OversizedConfigurationItemChangeNotification
               CustomPolicyDetails:
                 EnableDebugLogDelivery: 'true'
                 PolicyRuntime: guard-2.x.x
                 PolicyText: |-
                   # The rule definition goes here

The template only includes 3 main resources, but 2 actual logical components. A bucket with a policy and the conformance pack definition. Let’s go over all the components in the example.

S3 Bucket and the bucket policy

This bucket needs to start with awsconfigconforms. The config-conforms.amazonaws.com service will use this bucket to upload the template. The template comes from the OrganizationConformancePack resource. The policy allows the service to put and get objects as long as it is within the organization. This allows the service to deploy the template in all member accounts.

Organization Conformance Pack

The organization conformance pack definition looks big but it’s quite simple. If you have a closer look you will notice it has only 4 properties:

OrganizationConformancePackName
DeliveryS3Bucket
ExcludedAccounts
TemplateBody

Now the first 3 options are pretty straight forward. The template itself is a bit more complicated. In my example I used an inline template, I did this for the sake of this blog. But you can also reference an existing object on S3. This way you can use linting tools like cfn-lint on your conformance pack. This will reduce errors during deployment as you can catch them before you commit and push your code.

If you need help with setting up the conformance pack template, have a look at these samples.

Conclusion

When you combine custom rules in a conformance pack. It will allow you to simplify the distribution of these rules. You don’t need to maintain any pipelines or stack sets. And you outsource the distribution of your rules to AWS.

This removes all the maintenance on the infrastructure to do this distribution yourself. And you don’t need to think about scenarios like, adding/removing accounts.

Photo by ELEVATE

Use custom rules to validate your compliance

Sat, 03 Feb 2024 10:00:00 +0100

AWS has a lot of controls built in, but what if you need more? AWS Config allows you to create your own rules. These rules can then inspect your resources and determine if they are compliant. This is useful when you want to enforce certain configuration settings. Giving you an overview of how compliant your workloads are.

Let’s use a specific example, but you can apply this concept to other scenarios.

Check S3 access logs configuration

When an engineer creates an Amazon S3 bucket, they need to enable access logging on the bucket. AWS has a built-in config rule for this called s3-bucket-logging-enabled. When you enable the AWS Foundational Security Best Practices v1.0.0 standard in Security Hub it becomes available.

When we create a new account, part of the bootstrapping is the deployment of a bucket. The engineers who use the account only have read rights on this bucket. When an engineer creates a bucket, we expect him to use this bucket for the access logs.

Creating rules: Lambda vs CloudFormation Guard

You can build a custom config rules in 2 ways, using AWS Lambda and CloudFormation Guard. Lambda gives you a lot of flexibility, but it also brings complexity of maintaining. CloudFormation Guard is a bit more lightweight in that regard. Yes, you still need to maintain the logic to determine when your resource is compliant or not. But you need to do this in both cases, thus my go to preference is CloudFormation Guard.

Config Rule

When you start to define your custom config rule you need to think about the scenarios that happen:

When the resource is not an S3 bucket, we should skip the rule.
When the resource is the actual logging bucket, we should skip the rule.
When the resource does not have logging configured, we should fail the rule.
When the resource has logging configured on a different bucket then the one we want. We should fail the rule.
When the resource has logging configured with the expected bucket, we should pass the rule.

With AWS Config you can define InputParameters. We will use this to pass in the name of the bucket that will store the access logs. We named the parameter loggingBucket. You can reference that value of this parameter using: CONFIG_RULE_PARAMETERS.loggingBucket.

Deployment

I use CloudFormation for the deployment of the rule here is a snippet of the template:

Resources:
  S3AccessLogging:
    Type: AWS::Config::ConfigRule
    Properties:
      ConfigRuleName: lz-s3-access-logging
      Description: Validate that access logging has been enabled and that the correct logging bucket is used.
      EvaluationModes:
        - Mode: DETECTIVE
      InputParameters:
        Fn::Sub: '{"loggingBucket": "s3-access-logs-${AWS::AccountId}-${AWS::Region}"}'
      Scope:
        ComplianceResourceTypes: ["AWS::S3::Bucket"]
      Source:
        Owner: CUSTOM_POLICY
        SourceDetails:
          - EventSource: aws.config
            MessageType: ConfigurationItemChangeNotification
          - EventSource: aws.config
            MessageType: OversizedConfigurationItemChangeNotification
        CustomPolicyDetails:
          EnableDebugLogDelivery: 'true'
          PolicyRuntime: guard-2.x.x
          PolicyText: |-
            rule s3_logging_configuration when resourceType == "AWS::S3::Bucket" resourceName != CONFIG_RULE_PARAMETERS.loggingBucket {
                supplementaryConfiguration.BucketLoggingConfiguration exists
                <<
                Violation: S3 Bucket needs to have access logging configured
                Fix: Configure the destinationBucketName on your S3 bucket
                >>
            }

            rule s3_logging_correct_bucket when s3_logging_configuration {
                supplementaryConfiguration.BucketLoggingConfiguration {
                    destinationBucketName == CONFIG_RULE_PARAMETERS.loggingBucket
                    <<
                    Violation: S3 Bucket needs to have access logging configured
                    Fix: Configure the destinationBucketName on your S3 bucket
                    >>
                }
            }

Let’s walk over this policy and explain what is happening. The first rule only applies when the resource is a S3 bucket and the name is not equal to the name of the logging bucket. The name of the loggingBucket is configurable via the InputParameters. Next, we will check if the bucket has used the logging bucket for the access logs configured. When the logging configuration exists the destinationBucketName needs to match the given name.

You can test the scenarios I mentioned earlier using unit tests. Read: This is how you can test your cfn-guard rules for more information on how you could do that.

After you deployed this rule into your accounts. Config will check all your S3 Buckets if they are compliant with this custom config rule.

Limitations

Currently, AWS Config does only support CloudFormation Guard 2.x.x. I am waiting on the support for 3.x. Since that brings even more powerful capabilities like the json_parse method. This allows you to also check JSON structures like for example the bucket policy.

Conclusion

You can create custom config rules using CloudFormation Guard. Using CloudFormation Guard you don’t have to maintain dependencies from your functions. And you don’t need to upgrade your lambda runtime when it gets deprecated.

Do you want to know how you can distribute your rules? Read my next blog to learn more.

Photo by Pixabay

Using design patterns in AWS Lambda

Mon, 15 Jan 2024 13:00:00 +0100

When you speak with software developers, they will probably tell you that they use design patterns. But when the world first shifted to the internet the general feeling was that these design patterns would not work for the web. This is not true, and today you see these patterns being used more and more.

I have noticed the same behavior with serverless. In this blog post I will go over some reasons why you should be using design patterns in your Lambda functions

Getting started

To get started with AWS Lambda is quite easy, and this is also the reason why some crucial steps are skipped. For example, you can use the console to create a function and type your code in an editor via your browser. Or, you create a CloudFormation template and you put your code in the template itself. Obviously, this is fine for experimentation and learning purposes. But when you start developing systems that need to be reliable you need to take a different approach.

The biggest objection that I have with the editor in the console is that it does not allow you to run tests. Also using inline code that is part of a CloudFormation template has some downsides. Again you cannot run tests against your inline code and the code becomes vulnerable to indenting and syntax errors due to the lack of proper IDE highlighting.

Both options do not allow you to add dependencies, or at least not in an easy way. So you default to the installed dependencies, and you have no control over these dependencies.

Separate your infrastructure and application code

By storing your application code in separate files you will have all the benefits of using an IDE. You can run linters, formatters and tests and you have syntax highlighting while you develop your code. This also allows you to run these steps in the CI/CD pipeline before you actually deploy your code to production. This gives you a quality gate and a decision moment to say yes this can go to production.

When you combine this with the AWS Serverless Application Model you can also very easily include your dependencies. Or use a compiled language like golang for your Lambda functions. You simply run sam build before you run the aws cloudformation package and aws cloudformation deploy commands. SAM will build the binary and update the template to point to the newly built binary. Package will then upload it to S3 and replace the local reference to the S3 location. Deploy can then create or update the stack or you can use the CloudFormation integration in CodePipeline.

Why should I use design patterns

Testing AWS calls can be challenging, I wrote a blog on this for golang. By splitting the business logic from the infrastructure in your code you gain 2 important things:

You can focus on the business logic without having to deal with stubbing your infrastructure.
The used infrastructure can be changed without the need to change your business logic.

The former makes it easier to add test scenarios. When it’s easy to add scenarios you are more likely to add more scenarios. Increasing the reliability of your business logic. The latter gives you flexibility, you can swap a RDS database with a DynamoDB table relatively easily. Without changing the business logic, this also means that you have less of a vendor lock-in.

Golang example

Let’s use the Observer pattern, first you will need some interfaces:

type (
   Event struct {
      name string
   }

   Observer interface {
      NotifyCallback(Event)
   }

   Subject interface {
      AddListener(Observer)
      RemoveListener(Observer)
      Notify(Event)
   }

   eventObserver struct {
      id   int
      time time.Time
   }

   eventSubject struct {
      observers sync.Map
   }
)

In this example we define an Event, Subject and an Observer. We can use the event to store all the relevant data. When you pass the Event in the Subject the Event is sent to all the registered observers.

Next we need some logic to register the observers on the Subject:

func (s *eventSubject) AddListener(observer Observer) {
   s.observers.Store(observer, struct{}{})
}

func (s *eventSubject) RemoveListener(observer Observer) {
   s.observers.Delete(observer)
}

func (s *eventSubject) Notify(event Event) {
   s.observers.Range(func(key interface{}, value interface{}) bool {
      if key == nil || value == nil {
         return false
      }


      key.(Observer).NotifyCallback(event)
      return true
   })
}

As you can see the Notify method will iterate over all observers and pass the event to all registered observers. The observer could be as simple as:

func (e *eventObserver) NotifyCallback(event Event) {
   fmt.Printf("Received from observer %d: %s after %v\n", e.id, event.name, time.Since(e.time))
}

For your tests you can simply implement a testObserver:

type testObserver struct {
   event Event
}

func (e *testObserver) NotifyCallback(event Event) {
   e.event = event
}

This is a very simplified version of the business logic:

func doStuff(observer Observer) {
   n := eventSubject{observers: sync.Map{}}
   n.AddListener(observer)
   n.Notify(Event{name: "Joris Conijn"})
}

We create an Event with Joris Conijn as the name value. We want to test this business logic that it indeed creates this Event with Joris Conijn as the name value.

func TestObserver(t *testing.T) {
   t.Run("Run doStuff", func(t *testing.T) {
      var obs1 = testObserver{}
      doStuff(&obs1)
      assert.Equal(t, obs1.event.name, "Joris Conijn")
   })
}

For the actual implementation the observer would store it in DynamoDB or RDS. To have a good testing coverage you do need to test the observer. Per observer you now test the actual implementation of the database of your choice.

Conclusion

You can use design patterns in your Lambda functions. It is a great way to separate your business logic from the used infrastructure. And it makes you business logic easier to test. However, you might have noticed that you will need some boilerplate code to implement these patterns.

My advice would be to use these patterns when you need portability and/or when you have more complex business logic. For very simple Lambda functions it might not be worth it.

Thanks Tensor Programming for the inspiration. Photo by Soloman Soh

Setting up my own landing zone on AWS

Mon, 25 Dec 2023 12:50:00 +0100

As a consultant I am used to a certain level of quality that I need to deliver to our customers. For this reason I have built a landing zone for my own website, initiatives and experiments. By using the same structure that our customers have, I can test and build my ideas and apply them in customer environments if they are successful.

At first it might feel like a massive overkill to set up a complete landing zone. But once you have set it up you will be capable of performing experiments very easily.

My landing zone

For my landing zone I used the Customizations for AWS Control Tower (CfCt) project. When I created my landing zone the Landing Zone Accelerator was not yet available. I still need to evaluate if I can switch, but my initial concerns are the cost of running it. CfCt has low costs and depending on your usage it might even fit in the free tier. The Landing Zone Accelerator comes with VPCs and transit gateways and some other services that will generate costs regardless of what you are doing with the landing zone.

A default landing zone typically has the following accounts:

Management - Used for billing for all accounts in an organization, to create new accounts, and to manage access to all accounts.
Audit - A restricted account for your security and compliance teams to gain read and write access to all accounts.
Logs - Used as a repository of logs of API activities and resource configurations from all accounts.

Next to the default accounts I have some additional accounts setup:

Build - Used for all CodeCommit repositories and CodePipelines that are deployed within the landing zone.
Domains - Used to host the domain names that I have registered. If you are interested in why I use a separate account for this you can read my Cross account DNS management.
Service Catalog - Used to host all my AWS Service Catalog products used within my landing zone.
Team - Used to perform Temporary elevated access management (TEAM) within my landing zone.

For the rest I have some workload accounts that have an account per environment. Each workload has at least a development account, but it can be extended to have a test, acceptance and production account as well.

Account creation

When I want to build something new, I need a new development account. For account creation I use aws-samples/aws-control-tower-automate-account-creation. I altered the solution to directly inject a record into the DynamoDB table that triggers the account creation process. I have a simple command that I can run on the command line that requires the following arguments: workload name, description and the environment type.

Go build!

After I created a development account I login to the build account and navigate to the Service Catalog service. Depending on the idea I require a different product but up until now I created the following products:

Generic Project - Contains a CodePipeline including a CodeCommit repository, CodeBuild step and a CloudFormation deployment step for each environment.
S3 Hosted Website - Contains a CodePipeline including a CodeCommit repository, CodeBuild step and a CloudFormation deployment step for each environment and a S3 deployment step.

I use these Service Catalog products to create the CI/CD pipelines to host for example, my own website Conijn.io.

The CodeCommit repository will be pre-filled with the following files:

.pre-commit-config.yaml - contains the cfn-lint and cfn_nag pre-commit hooks.
buildspec.yml - installs the aws-sam-cli and builds and packages the template.
parameters-development.json - The CloudFormation parameters, tags and policies for the development environment.
parameters-production.json - The CloudFormation parameters, tags and policies for the production environment.
README.md - Initial read me file.
template.yaml - Initial CloudFormation template.

After the Service Catalog product has been successfully launched. I only need to execute the following command:

git clone codecommit::<AWS_REGION>://<MY_PROFILE_NAME>@<PROJECT_NAME>

And I am ready to start building, I can start with adding resources to my template. If I need to use parameters I can use the parameters-development.json file. If I need to do additional steps I can add them in the buildspec.yml. When I commit the changes and push them to the remote repository CodePipeline will be triggered and deploy the changes into the development account.

For visibility of the pipelines I have set up a NotificationTopic, this topic is a SNS Topic that has AWS ChatBot as a subscriber. Chatbot will then send the updates to my Slack workspace that I have set up. This way when the pipeline is triggered I will get the notifications on my phone and laptop.

Conclusion

Setting up your own landing zone is quite some work, but you will learn a lot from it. And, when you have one it makes it very easy to start a new initiative in an empty AWS Account.

Photo by Vincent Albos

Using Golang for your Serverless projects

Tue, 19 Dec 2023 11:00:00 +0100

In one of my previous blogs I wrote why I switched to compiled languages for my lambda functions. But using Golang for your lambda functions does add some challenges. In this blog I would like to share the challenges that I have seen and how to mitigate them.

Spoiler, I use a Makefile

I always use a Makefile in my projects, If you are interested in why I wrote a blog on that as well. I am mentioning this before we dive into the challenges to keep the focus on the solution and not the technology.

Every function is its own module?

When you develop a CLI tool, your project structure is quite simple. You have a go.mod file in the project root directory, and at least a main.go file that contains the business logic. When you are creating a serverless project, this changes. This is because each lambda function needs to be its own module. As a result you will need to create a folder for each function and each folder contains its own go.mod and main.go files, forming its own module.

Managing dependencies

The first challenge that I noticed is dependency management. This is now scattered over your lambda functions. This adds more maintenance to your project. It will also cause issues in your IDE. Since your IDE would expect a single module and not multiple it gets confused.

You can solve this by setting up a multi-module workspace. In essence you need to create a go.work file in the root directory of your project. Then you can point to your folders that contain the modules. Your IDE should now be able to understand all your modules and handle the dependencies for you.

To ensure that your dependencies are up-to-date and we only include the one that we actually use you can run go get -u and go mod tidy. But you will need to do this for each individual module. For this reason I use the following Makefile target:

.PHONY: tidy
tidy: ## Run `go get -u` and `go mod tidy` for all modules
   $(info [+] Running `go get -u` and `go mod tidy`)
   find . -name go.mod -execdir go get -u \;
   find . -name go.mod -execdir go mod tidy \;

This will look for each folder that contains a go.mod file and executes those 2 commands in that folder for you. This makes it a simple and easy command to run more often.

Running tests

I am a fan of TDD (test driven development), so obviously I wrote tests for my lambda functions. An example can be found in the “Stubbing AWS Service calls in Golang” blog I wrote. But with this multi-module workspace setup it’s hard to run all tests.

If you run go test ./... it will only run the tests that you might have in the root of your project. And, not in the modules that are used by your lambda functions. An option would be to navigate into your lambda folder and execute the command there. But since it requires additional steps the likelihood of you doing that is smaller than a single simple command.

For this reason I use the following Makefile target:

.PHONY: test
test:
   $(info [+] Running unit tests)
   find . -name go.mod -execdir go test ./... -coverprofile=coverage.out -covermode count \;

Again same as we did with the dependency management, we will look for each go.mod file. And execute the go test command in that folder. It will also generate a coverage.out file. But since the file only contains the coverage of the tested module we need to combine them together:

reports/coverage.out:
   find . -name coverage.out -type f -exec sh -c 'cat {} | if [ "$1" = "{}" ]; then cat {}; else tail -n +2 {}; fi' sh {} \; > reports/coverage.temp
   echo "mode: count" | cat - reports/coverage.temp > reports/coverage.out
   rm reports/coverage.temp

We are now looking for each coverage.out file and take all the content minus the first line. We are placing all those lines in a single file and then we add the first line back in there.

Now if you want to visualize your coverage in for example GitLab. You will need a Cobertura XML file:

reports/coverage.xml: reports/coverage.out
   $(info Collecting Code Coverage)
   go run github.com/boumenot/gocover-cobertura < reports/coverage.out > reports/coverage.xml

We will just use the gocover-cobertura module to convert the file to the needed XML format and you are good to go.

Conclusion

It’s possible to have a multi-module setup and have a simple and single command to perform actions like running your tests. However, you do need to maintain some scripting in for example a Makefile. A full working example can be found on Github.

Photo by alleksana

Avoid using the default profile

Sat, 21 Oct 2023 11:00:00 +0200

When you start working with AWS one of the easiest ways to interact with the APIs is through the IAM User. Although it’s not the advised way it’s the easiest way hence the most used one and the most abused one. In this blog I will dive a little deeper into this topic.

Easy to use

Using an IAM User is the oldest way to interact with the AWS API. You only need an AccessKey and a SecretAccessKey and you are good to go! When you have the AWS CLI installed you only need to execute the aws configure command. This will as for some inputs via prompts:

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [eu-west-1]: eu-west-1
Default output format [yaml]: yaml

This will result in the following ~/.aws/credentials file:

[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

And you are good to go! You can now perform any aws cli command. Assuming the IAM User has the correct policies in place to perform the needed actions. Having these credentials in place makes your life easy. For example: the Boto3 SDK will use your credentials. Super convenient, super dangerous!

Boto3 SDK

When you are developing you often run your code to test if it works. Now I would love to think that everybody uses Test Driven Development but I know this is not true. This could have many reasons, experience is the most obvious one. When you start your development journey there are already a lot of things that you need to learn. Writing unit tests is not the highest priority for someone who wants to learn how to write code. That being said, yes it will slow you down. But only in the beginning! When you master proper testing you will increase your productivity tenfold!

So why are these credentials dangerous again? Assume you are writing some code to clean-up objects from a S3 bucket. If you have a bug in your code, and you run your script it will be using the default credentials as a default. Resulting in potential data loss in your AWS Account! And this is of course one of the examples that can go wrong. Even when you do use test driven development, and you forget to stub one of the API calls. Boto3 will use the default credentials and execute the API call.

Avoid using a default profile

So, should you not use the IAM User credentials then? Well there are “better” alternatives, like Single Sign On. When you configure those credentials as your default credentials, you will face the same problem! So avoid the default profile! Either don’t configure it or use mock credentials like I used in the example. They will result in an access denied but that is better than losing data. Next to that you should use the principle of the least privileged. Because when your principal does not have the access rights, it cannot perform them!

Conclusion

Make sure you use the least privileged principle. And avoid sticking your credentials in the default profile.

Photo by Min An

Stubbing AWS Service calls in Golang

Tue, 15 Aug 2023 09:00:00 +0000

I recently switched to Golang for my language of choice. (In my previous blog you can read why.) But I am also a big fan of test driven development. With Python you have a stubber that helps you mock the AWS API. So how do you do this in Golang? I found 2 ways to do this. One via dependency injection and one via a stubber. In this blog I will share my experience so far.

Using dependency injection

My first experiment was with dependency injection. I used the following code to do this:

package main

import (
	"context"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"time"
	"log"
	"os"
)

type Request struct {}
type Response struct {}

type Lambda struct {
	s3Client *s3.Client
}

func New() (*Lambda, error) {
	cfg, err := config.LoadDefaultConfig(context.TODO())

	m := new(Lambda)
	m.SetS3Client(s3.NewFromConfig(cfg))
	return m, err
}

func (x *Lambda) SetS3Client(client *s3.Client) {
	x.s3Client = client
}

func (x *Lambda) Handler(ctx context.Context, request Request) (Response, error) {
	data := []byte("Hello World")

    _, err := x.s3Client.PutObject(x.ctx, &s3.PutObjectInput{
		Bucket: aws.String("my-bucket"),
		Key:    aws.String("my-object-key"),
		Body:   bytes.NewReader(data),
	})

    return Response{}, err
}

As you can see, I use the SetS3Client method as a setter. It allows me to set the client from the outside. This is useful when doing unit tests. You can use this in your tests as followed:

package main

import (
	"context"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"testing"
	"time"
	"log"
	"os"
)

type mockS3Client struct {
	s3.Client
	Error error
}

func (m *mockS3Client) PutObject(input *s3.PutObjectInput) (*s3.PutObjectOutput, error) {
	return &s3.PutObjectOutput{}, nil
}

func TestHandler(t *testing.T) {
    lambda := New()
    lambda.SetS3Client(&mockS3Client{})
	var ctx = context.Background()
	var event Request

    t.Run("Invoke Handler", func(t *testing.T) {
		response, err := lambda.Handler(ctx, event)

	    // Perform Assertions
    })
}

We inject a mocked object that acts as the client used to perform the API calls. With this approach I could now write some tests. But I realized that this approach creates another problem. For example, what if you have 2 API calls that perform a PutObject call. In this example I return an empty PutObjectOutput. But I want to test more than one scenarios, so how do you control this behavior in your mocked object?

Using a stubber

So I did some more research and I found the awsdocs/aws-doc-sdk-examples repo. This repository used a testtools module. So I started an experiment to see how I could use this module. I refactored the code as followed:

package main

import (
	"context"
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

type Request struct {}
type Response struct {}

type Lambda struct {
	ctx      context.Context
	s3Client *s3.Client
}

func New(cfg aws.Config) *Lambda {
	m := new(Lambda)
	m.s3Client = s3.NewFromConfig(cfg)
	return m
}

func (x *Lambda) Handler(ctx context.Context, request Request) (Response, error) {
    data := []byte("Hello World")

    _, err := x.s3Client.PutObject(x.ctx, &s3.PutObjectInput{
		Bucket: aws.String("my-bucket"),
		Key:    aws.String("my-object-key"),
		Body:   bytes.NewReader(data),
	})

	return Response{}, err
}

I added a cfg parameter to the New method, so I also need to pass this in my main method.

package main

import (
	"context"
	"github.com/aws/aws-lambda-go/lambda"
	"github.com/aws/aws-sdk-go-v2/config"
	"log"
)

func main() {
	cfg, err := config.LoadDefaultConfig(context.TODO())
	if err != nil {
		log.Printf("error: %v", err)
		return
	}
	lambda.Start(New(cfg).Handler)
}

The test itself looks like this:

package main

import (
	"bytes"
	"context"
	"encoding/json"
	"errors"
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/awsdocs/aws-doc-sdk-examples/gov2/testtools"
	"io"
	"os"
	"strings"
	"testing"
)


func TestHandler(t *testing.T) {
	var ctx = context.Background()
	var event Request

	t.Run("Upload a file to S3", func(t *testing.T) {
		stubber := testtools.NewStubber()
		lambda := New(*stubber.SdkConfig)

		stubber.Add(testtools.Stub{
			OperationName: "PutObject",
			Input: &s3.PutObjectInput{
				Bucket: aws.String("my-sample-bucket"),
				Key:    aws.String("my/object.json"),
				Body:   bytes.NewReader([]byte{}),
			},
			Output: &s3.PutObjectOutput{},
		})

		response, err := lambda.Handler(ctx, event)
		testtools.ExitTest(stubber, t)

		// Perform Assertions
	})
}

As you can see, we now moved the mock in the test itself. This enables you to let the AWS API react based on your test. The biggest advantage is that it’s encapsulated in the test itself. For example, If you want to add a scenario where the PutObject call failed you add the following:

t.Run("Fail on upload", func(t *testing.T) {
    stubber := testtools.NewStubber()
    lambda := New(*stubber.SdkConfig)
    raiseErr := &testtools.StubError{Err: errors.New("ClientError")}

    stubber.Add(testtools.Stub{
        OperationName: "PutObject",
        Input: &s3.PutObjectInput{
            Bucket: aws.String("my-sample-bucket"),
            Key:    aws.String("my/object.json"),
            Body:   bytes.NewReader([]byte{}),
        },
        Error: raiseErr,
    })

    _, err := lambda.Handler(ctx, event)
    testtools.VerifyError(err, raiseErr, t)
    testtools.ExitTest(stubber, t)
})

The main advantage of the stubber is that you validate the Input out of the box. But in some cases you want to ignore certain fields. For example, if you use a timestamp in your Key. Or the actual Body of the object you are uploading.

You can ignore these fields by setting the IgnoreFields. (Example: IgnoreFields: []string{"Key", "Body"})

The second thing I like about using the stubber is the VerifyError method. This will verify if the error is returned that you raised in your test scenario.

The last one is the ExitTest method. This will make sure that all the defined stubs are actually called. In other words, if you still have a stub. Your test will fail because there is still a uncalled stub.

Conclusion

The testtool is a good replacement of the stubber I used in Python. It allows you to encapsulate scenario data in your test. Avoiding hard to maintain mock objects. The testtool works from the configuration, so you don’t need to stub every client. Resulting in less code that is needs to test your implementation.

Photo by Klaus Nielsen

Using Golang for your AWS Lambda Functions

Fri, 04 Aug 2023 00:00:00 +0000

I was sparked on a XKE to do a short experiment with using Golang for my AWS Lambda Functions. The trigger for this was something my colleague Mark van Holsteijn said. We where talking about sustainability. During this talk Mark made the comment that we should drop Python for our functions then. (He also recently wrote a blog on Golang using Golang for custom providers.)

Let me compare some numbers between Python and Go. (source: Which Programming Languages Use the Least Electricity?))

Energy: 75.88 vs 3.23
Time: 71.90 vs 2.83
Memory: 2.80 vs 1.05

As you can see, based on these numbers Golang is clear winner. But are there other advantages? So, like always everything comes with it’s own pros and cons. Lets explore those when you do chose for Golang.

Pre-compiled binary vs compiled at run time

Let’s start with the obvious one, a pre compiled binary vs code that is compiled at run time. This is one of the reasons why Golang is more energy efficient as you only compile once vs every time you run it. But the advantage of Python is that you can actually see the source code in the AWS Console and tweak it. This makes it a great candidate for easy development and fast testing. This also a problem! When you are running production workloads. You don’t want your developers altering your code through the console. Yes, IAM can help you here but altering a binary is not possible through the console.

You need to compile

Why is the need to compile an advantage? Next to the performance improvement you actually check if your program compiles. If it does not compile you cannot deploy it. Compare this with Python, you can upload malformed code. This would than break your workload because of an indent issue. You can build in checks to prevent this from happening, and you should! But you need to do something extra, with Golang this comes out of the box.

Dependencies

In python you have the option to do inline code in AWS CloudFormation templates. I would never recommend doing this. When you needs other dependencies than that there are available. You will need to bundle them yourself and upload them as a zipfile. When you are missing a dependency the invocation of your function will fail at runtime. When you use Golang the dependencies are all build into the binary. Especially when you set the CGO_ENABLED=0 option in your compile options. This will make sure that all the needed libraries are included.

Tutorial

Let’s deploy a small stack including a single Lambda function. We will be using AWS SAM for the deployment, so please make sure you have that installed.

Prepare the file structure

Create the folders and files we will be needing:

mkdir aws-lambda-sample-golang
cd aws-lambda-sample-golang
touch template.yaml
mkdir my-function
touch my-function/Makefile
touch my-function/main.go

SAM Template

A SAM Template is nothing more that a CloudFormation template. But it contains the AWS::Serverless-2016-10-31 transformation. This allows you to use the AWS::Serverless::Function resource. When CloudFormation parses this resource it will converted to CloudFormation resources.

Add the following to the template.yaml file:

Transform: AWS::Serverless-2016-10-31
Resources:

  MyFunction:
    Type: AWS::Serverless::Function
    Metadata:
      BuildMethod: makefile
    Properties:
      Runtime: provided.al2
      CodeUri: ./my-function
      Handler: bootstrap

  MyFunctionLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub /aws/lambda/${MyFunction}
      RetentionInDays: 7

A few things are going on here. You can see the BuildMethod set in the Metadata property. With this configuration you are saying to SAM that you want to use a Makefile. This Makefile is used to compile the lambda function.

CodeUri will point to the root of your Lambda functions code.
Handler will point to your binary output file.

Lets create the Makefile, we need to place it in the root folder of the function my-function. Add the following content to the file:

build-MyFunction:
	GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o bootstrap
	cp ./bootstrap $(ARTIFACTS_DIR)/.

The syntax used here is build-<LOGICAL-ID-OF-CLOUDFORMATION>. So, if your LogicalId in the template is HelloWorld. Your Makefile target would be: build-HelloWorld. The go build command will build your binary. The copy command will copy the binary in the artifacts directory. This is used when you deploy your template.

Note the GOARCH if you use an ARM function you need to specify it here as well.

Lambda Code

We need to initialize our go module, and pull in the github.com/aws/aws-lambda-go dependency:

cd my-function
go mod init my-function
go get github.com/aws/aws-lambda-go
cd ..

Now we can fill the my-function/main.go with some business logic:

package main

import (
	"context"
	"log"

	"github.com/aws/aws-lambda-go/lambda"
)

type MyEvent struct {
	Name string `json:"name"`
}

func Handler(ctx context.Context, event MyEvent) {
	log.Printf("Welcome %s", event.Name)
}

func main() {
	lambda.Start(Handler)
}

And done!

Deploy the template

To deploy the template we first need to build the binary. AWS SAM as build in support for building and packaging Lambda functions.

Note: AWS SAM will create a bucket in your account used for uploading the artifacts.

sam build
sam deploy \
    --stack-name aws-lambda-sample-golang \
    --capabilities CAPABILITY_IAM \
    --resolve-s3

Testing

You can now invoke the Lambda function, with a {"name": "Joris Conijn"} payload. The logs should now show:

Welcome Joris Conijn

For a full working copy you could also clone my Nr18/aws-lambda-sample-golang repository.

Conclusion

For me Golang would become my default choice for AWS Lambda functions. The fact that it’s “more sustainable” and that it protects you from obvious failures convinced me. That beings said I have a software engineering background. So, for me it’s relatively easy to switch languages. How about you, would you use Golang as your default choice?

Photo by Narcisa Aciko

Invalidate CloudFront cache for Hugo websites

Thu, 13 Jul 2023 21:00:00 +0200

Hugo is a great tool that let you build website fast and easy. It will render a complete static HTML version of your website. This makes it an ideal candidate to host it on a S3 Bucket. You can then use CloudFront to serve the content across the globe.

CloudFront Cache

Cache can be a wonderful thing! But it can also be an annoying thing. So I decided to dive a little bit deeper into this topic for my own personal website. So by default, when you use S3 as an origin by default the TTL (Time To Live) is set to 24 hours. This means that when you visit your site that version is than cached for 24 hours on the edge location.

This means if you would update your website, the changes will not be visible for you. This is because you will most likely end up on the same edge server. You can invalidate the cache but objects like photos and videos benefit from this cache.

Only invalidate what has changed

So that brings us to the next logical step. Invalidate only the things that have been changed. I am using CodeCommit as a source repository you can ask git for what files are changed:

git diff-tree --no-commit-id --name-only -r HEAD

This will give you a list of files changed in your last commit, with this information you can do smart things:

Only select the files changed in the content folder.
Replace the parts that contain any _index to index.
Replace the parts that contain any .md to .html.
Detect where we have post aggregation.

So for this blog post that would result to:

$ git diff-tree --no-commit-id --name-only -r HEAD
content/blog/2023/07/invalidate-cloudfront-cache-for-hugo-websites/index.md

So Hugo uses _index.md files to generate pages that list all child pages. These overview pages are also cached and also need to be invalidated.

By using the following script I will include these pages in my “to be invalidated” list:

#!/usr/bin/env python3
from typing import List
import os
import subprocess

command = ['git', 'diff-tree', '--no-commit-id', '--name-only', '-r', 'HEAD']
output = subprocess.check_output(command, cwd=os.getcwd()).decode('utf-8')
changed_files = output.strip().split('\n')


def content_changed(path: str) -> bool:
    return path.startswith("content/")


def convert_html_endpoints(path: str) -> str:
    path = path.replace("content/", "")
    path = path.replace("_index", "index")
    path = path.replace(".md", ".html")

    if not path.startswith("/"):
        path = f"/{path}"

    return path


def find_list_pages(path: str) -> List[str]:
    existing_path = ""
    list_pages = []

    for part in path.split("/"):
        list_page_path = os.path.join(existing_path, part, "_index.md")

        if os.path.exists(list_page_path):
            list_pages.append(os.path.join(existing_path, part, "_index.md"))

        existing_path = os.path.join(existing_path, part)


    return list_pages


def flatten_list(list_of_lists: List[List[str]]) -> List[str]:
    return [item for sublist in list_of_lists for item in sublist]


changed_files += list(flatten_list(map(find_list_pages, changed_files)))
changed_files = list(dict.fromkeys(changed_files))
changed_files = filter(content_changed, changed_files)
changed_files = list(sorted(map(convert_html_endpoints, changed_files)))
list(map(print, changed_files))

Let’s examine what that would bring us:

$ ./retrieve_files.py
/blog/2023/07/invalidate-cloudfront-cache-for-hugo-websites/index.html
/blog/index.html
/index.html

The blog post itself is listed, but also the front page and the blog section of my website. Since these pages are listing my page I include them in the pages to invalidate.

Putting it together

In my build phase I will call this script since I will have access to the commit history there. A simple command will capture these files in a changelog.txt file:

./retrieve_files.py > public/changelog.txt

Because my pipeline runs in a separate AWS account I could not use a Lambda function. Instead I upload the changelog file to the S3 bucket. And I use a S3 Event Notifications to trigger the create_invalidation call.

Conclusion

You can use your existing systems to keep track what has changed. You only need to expose it in the correct way so that it becomes usable in a downstream actions.

This way you make use of a decoupled event driven pattern.

Photo by Magda Ehlers

Automate your workload insights

Wed, 28 Jun 2023 11:00:00 +0200

Managing more than one AWS accounts can become challenging. AWS Organizations provides a way to manage many accounts. You can automate account creation and management. If you use a naming schema you can leverage landingzone-organization!

Naming schema?

Using a naming schema improves your ability to find the correct account faster. For example, let say your company Xebia has some workloads. Each workload requires a full DTAP environments. When you follow the best practices of AWS, you will end up using an account per environment.

Using a naming schema your account name would look like this: xebia-<workloadname>-<environment>

When you make use of AWS SSO you will see a list of accounts. In each account you will have one or more roles that you can assume. Depending of your permissions that could be none or all accounts.

When you are a workload engineer you will have a short list. When you are a platform operator or an auditor you might have access to all accounts. In the later, it helps when you have applied this schema. For example, you can search on the workload name. This will filter out the list only showing the environment account for that workload. Or you can search on environment, or even a combination.

Going a step further

Combining a naming schema with an OU layout wil result in even more flexibility.

First, lets address obvious advantages. When you use a OU structure as followed:

workloads/development/xebia-myworkloadname-development
workloads/testing/xebia-myworkloadname-testing
workloads/acceptance/xebia-myworkloadname-acceptance
workloads/production/xebia-myworkloadname-production

This allows you to place a SCP on the workloads OU this one applies to all workloads. But you also have the ability to apply a SCP on each individual environment OU. This SCP is then only applied to the development accounts for example. I will not go into SCP that is a topic on its own.

The other advantage is that you now know where your workloads live. Combine that with landingzone-organization and you can do things like this:

from landingzone_organization import AWSOrganization

# Fetch the organization information and instantiate the organization object
organization = AWSOrganization().parse()

# Define the OU structure where the workload accounts live
nested_ou = ["Workloads"]

# Iterate over all workloads in the provided OU location
for workload in organization.workloads(nested_ou):
    # Display the workload name
    print(f"Workload: {workload.name}")

    # Iterate over each account in the workload
    for account in workload.accounts:
        # Display the environment name and id of the account (aka the workload environment)
        print(f"\tEnvironment: {account.environment} has {account.account_id}")

Why does this help me?

Keeping track of workloads that are actually running is hard. But if it does not have a production account you know the workload is not in production yet. Next to that keeping track of what workloads exists within your organization is hard. Most of the times this comes from spreadsheets, how do you keep those up to date?

With this tool you can write all the needed data to a CSV file for later consumption. It could become the source of truth within your company and you can automate!

Conclusion

Automation makes are life easier! So stop keeping track of your workloads the old fashion way and start using automation tools.

There are various applications for this that I will hope to address in a future blog, so stay tuned!

Photo by Sangria Lips

Use StepFunctions to simplify your serverless applications

Fri, 09 Jun 2023 08:00:00 +0200

Use StepFunctions to simplify your serverless applications

AWS StepFunctions is a great orchestrating tool for your serverless applications. When you write lambda functions that only contain logic to perform a single task they are easier to test. Especially when there is no orchestration logic within your function.

The orchestration is then performed by StepFunctions by defining a state machine. This state machine is nothing more than a JSON definition based on Amazon State Language. It describes the flow between the functions for your specific use-case.

Lets use an example

Lets say one of the input parameters is an object location on S3. We will feed this to your state machine. A lambda function can then be used to read this object and perform some business logic on this data.

Or you can use the AWS SDK Service integration. By using the service integration you can again simplify your business logic. Instead of needing to read an object from S3 and do some actions. You only need to do the action on the data that you receive. This will remove the need of mocking S3 GetObject calls in your unit tests. Next to that you do not have to pay for the time it takes to read the data from S3. This is done outside of the scope of the lambda function, and therefor you do not have to pay for it.

Sounds great, do you have an example for me?

Lets assume the following payload it given:

{
  "Bucket": "my-bucket-name",
  "ObjectKey": "my-path/to-my-file.json"
}

And we have a the following task in our state machine:

{
  "Read From S3": {
    "Type": "Task",
    "Resource": "arn:aws:states:::aws-sdk:s3:getObject",
    "Parameters": {
      "Bucket": "$.Bucket",
      "Key": "$.ObjectKey"
    },
    "OutputPath": "$.Body",
    "ResultPath": "$.ObjectBody",
    "Next": "My Lambda"
  }
}

This will perform a s3:getObject action on s3://my-bucket-name/my-path/to-my-file.json. The action returns the content of the file in the Body parameter. And we will place this content in the ObjectBody parameter of the event. So the event will now look like:

{
  "Bucket": "my-bucket-name",
  "ObjectKey": "my-path/to-my-file.json",
  "ObjectBody": "{\"Foo\": \"Bar\"}",
  "TransformedBody": "{\"Foo\": \"Bar\"}"
}

We can now invoke a lambda function to do something with this payload:

{
  "My Lambda": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "InputPath": "$.ObjectBody",
    "Parameters": {
      "FunctionName": "arn:aws:lambda:eu-west-1:123456789012:function:my-function"
    },
    "ResultPath": "$.TransformedBody",
    "Next": "Write to S3"
  }
}

Assume that in the lambda function we enriched the object with an extra parameter. So the event will look like this now:

{
  "Bucket": "my-bucket-name",
  "ObjectKey": "my-path/to-my-file.json",
  "ObjectBody": "{\"Foo\": \"Bar\"}",
  "TransformedBody": "{\"Foo\": \"Bar\", \"Bar\": \"Foo\"}"
}

We can now store the transformed event on S3:

{
  "Write to S3": {
    "Type": "Task",
    "Resource": "arn:aws:states:::aws-sdk:s3:putObject",
    "Parameters": {
      "Bucket": "$.Bucket",
      "Key": "$.ObjectKey",
      "Body.$": "$.TransformedBody"
    }
  },
  "End": true
}

And done, the existing object is updated with the extra parameter in the json structure.

Conclusion

You can use StepFunctions to remove the following from your lambda functions:

Orchestration logic
AWS API Calls

This enables you to focus on the task at hand, and simplifies the code that you need to write and maintain.

Photo by Baskin Creative Studios

Why should I set a canonical URL?

Wed, 10 May 2023 00:00:00 +0000

If you like blogging like me, you probably post your content on multiple places. For example, I post my content on xebia.com, conijn.io and dev.to. When you do not provide a canonical url search engines like Google are less likely to show your blog.

What is a canonical URL

I describe a canonical url the same way as you do other source disclosures. For example, when you publish your blog on site A. This would be the original content. By posting it on site B you are duplicating the content. Since you are the author that is all fine. But the search engine does not know this. So by pointing to the source, the search engines know what page they should recommend.

How can you use a canonical URL

There are 2 relatively easy ways to set a canonical url. Via a header or using a html tag. For example the headers of your website request could return:

Link: <https://xebia.com/blog/why-should-i-set-a-canonical-url/>; rel="canonical"

For this method you need control over how the headers are generated. The other method is simpler, using a html tag:

<link rel="canonical" href="https://xebia.com/blog/why-should-i-set-a-canonical-url"/>

The reason why I like this method, is because it is easy to setup using Hugo.

What is Hugo?

According to the website:

Hugo is one of the most popular open-source static site generators. With its amazing speed and flexibility, Hugo makes building websites fun again.

Hugo uses simple Markdown files as input and renders a complete static website out of that. So you will have a single markdown file for your blog.

Setup a canonical URL in Hugo

Each post is a separate markdown file, so you can define some metadata using front-matter. In this metadata you can also add the canonical url:

---
author: Joris Conijn
canonicalUrl: https://xebia.com/blog/why-should-i-set-a-canonical-url/
categories:
- Blog
date: 2023-05-10 00:00:00+00:00
excerpt: Learn why you should set a canonical url when you do cross-posting of your
  blogs.
featured_image: images/banner.jpg
focus-keywords: set canonical url
slug: why-should-i-set-a-canonical-url
subtitle: ''
title: Why should I set a canonical URL?
---

If you like blogging like me, you probably post your content on multiple places. For example, I post my content on [xebia.com](https://xebia.com/blog), [conijn.io](https://conijn.io/blog/) and [dev.to](https://dev.to/aws-builders). When you do not provide a canonical url search engines like Google are less likely to show your blog.
...

Next, you need to specify in the layout of your site where the canonical url used. I have placed it in the baseof.html file of my layout as followed:

{{ if .Params.canonicalUrl }}
<link rel="canonical" href="{{ .Params.canonicalUrl }}">
{{ else }}
<link rel="canonical" href="{{ .Permalink }}">
{{ end }}

This will point the page to itself if no canonical is set. But if it is set it points to the provided url.

Conclusion

Setting the canonical url can improve the likelihood of your page being found. And when you use Hugo it is easy to put in place!

Photo by Brett Sayles

How to update your repository from a gitlab pipeline

Tue, 18 Apr 2023 11:00:00 +0200

Did you ever ran into the scenario where you wanted to update files in a repository from your pipeline? I did, a couple of times actually!

Updating documentation

In one of my previous blogs I wrote on how you can manage diagrams as code. Well as a human you sometimes forget to update images that you might be using in your README. So, updating these images during the CI process will help you with that.

Building containers

There are several scenarios where you might want to do this. For example, you are building a container and you want to deploy this in your environment. The approaches that I often is that you tag the container as latest and trigger a redeployment.

The problem with this approach is that you might already have a cached container on your host. With the result that you deploy the previous version. And what if you need to roll back to the previous version, for whatever reason? Or you want to know what version was running at a specific moment?

You can solve this by pointing to the actual digest of the container. This will always deploy the given version, or it will rollback. And if you need to do a manual rollback? You can revert the digest to the previous value. And as a bonus you can use your git history as an audit log.

If you want to know how you can update your CloudFormation templates. We created a tool called cru that can help you with that!

Creating a commit from CI on GitLab.

To get this to work, you do need to do some configuration. You will need to generate a SSH Key pair:

ssh-keygen -t ed25519 -C "<comment>"

You can add the public key in your repository under Settings and then Repository. In the Deploy keys section you can add your public key. Please make sure you grant write permissions to this key.

Under Settings and then CI/CD you will have the section Variables. Here you will need to create a File type variable called DEPLOY_SSH_KEY as value use the private key.

The private key will be available on the runner as a file. The location of the file is available as the DEPLOY_SSH_KEY environment variable.

Before you can commit you first need to:

Configure the SSH client.
Configure the git user.
Configure the git remote.
Commit and push the changes if there are any

To make it a bit easier I placed all those actions in the following bash script:

#!/usr/bin/env bash

failed () { echo "Failed:" >&2 "$@" && exit 1; }

GIT_USER=$1
GIT_EMAIL=$2
GIT_COMMIT_MESSAGE=$3

[[ -z "${CI_SERVER_HOST}" ]] && failed "The environment variable CI_SERVER_HOST is not set, this is needed to configure git and ssh to be enable to write back to the repository."
[[ -z "${CI_COMMIT_REF_NAME}" ]] && failed "The environment variable CI_COMMIT_REF_NAME is not set, this is needed to configure git and ssh to be enable to write back to the repository."
[[ -z "${DEPLOY_SSH_KEY}" ]] && failed "The environment variable DEPLOY_SSH_KEY is not set, this is needed to configure git to be enable to write back to the repository."
[[ ! -f "${DEPLOY_SSH_KEY}" ]] && failed "The ${DEPLOY_SSH_KEY} is not available on the filesystem, this is needed to configure git to be enable to write back to the repository."
[[ -z "${GIT_USER}" ]] && failed "Please use: ./commit-changes.sh '<USER NAME>' '<EMAIL>' '<COMMIT MESSAGE>'"
[[ -z "${GIT_EMAIL}" ]] && failed "Please use: ./commit-changes.sh '<USER NAME>' '<EMAIL>' '<COMMIT MESSAGE>'"
[[ -z "${GIT_COMMIT_MESSAGE}" ]] && failed "Please use: ./commit-changes.sh '<USER NAME>' '<EMAIL>' '<COMMIT MESSAGE>'"

run_function () {
    GREEN='\033[1;32m'
    RED='\033[1;31m'
    NC='\033[0m' # No Color

    echo -n "Running $1 "
    ERROR=$( $1 2>&1 )
    [ $? -eq 0 ] && echo -e "[ ${GREEN}DONE${NC} ]" ||  (echo -e "[ ${RED}FAILED${NC} ]" && failed "Unexpected error occurred: ${ERROR}")
}

configure_git_client() {
    git config user.name "${GIT_USER}"
    git config user.email "${GIT_EMAIL}"
}

cleanup_ssh_client() {
    rm -rf ~/.ssh
    mkdir ~/.ssh
}

configure_ssh_client() {
    local KNOWN_HOSTS=~/.ssh/known_hosts
    local CONFIG_FILE=~/.ssh/config

    ssh-keyscan $CI_SERVER_HOST > $KNOWN_HOSTS
    chmod 0644 $KNOWN_HOSTS
    echo "Host $CI_SERVER_HOST" > $CONFIG_FILE
    echo "    IdentityFile $DEPLOY_SSH_KEY" >> $CONFIG_FILE
    chmod 0600 "$DEPLOY_SSH_KEY"
}

configure_git_remote () {
    git remote remove update 2>/dev/null || echo "update remote was not present yet"
    git remote add update git@$CI_SERVER_HOST:$CI_PROJECT_PATH.git
}

commit_and_push_changes () {
    git fetch update
    git branch -D $CI_COMMIT_REF_NAME || echo "branch does not exist"
    git switch -c $CI_COMMIT_REF_NAME update/$CI_COMMIT_REF_NAME
    git diff --quiet && git diff --staged --quiet || (git commit -am "${GIT_COMMIT_MESSAGE}" && git push --set-upstream update $CI_COMMIT_REF_NAME)
}

run_function "configure_git_client"
run_function "cleanup_ssh_client"
run_function "configure_ssh_client"
run_function "configure_git_remote"
run_function "commit_and_push_changes"

You can use it as followed:

./commit-changes.sh "GitLag CI" "noreply@acme.com" "chore: my commit message"

Conclusion

Committing from the a GitLab pipeline works the same as you would from your own machine. The difference is that you need to configure your runners everytime and you need to grant the access to do so.

Photo by Jill Burrow

Cross account DNS management on AWS

Thu, 30 Mar 2023 11:00:00 +0200

I recently created a few blog post on how you can build, deploy and host an application on AWS. This blog post is all about handling DNS records in a cross account setup. You might think AWS got you covered, but the sad reality is that they don’t. (when dealing with cross account use cases.)

For example, if you have a domain called example.com. You will need to create the hosted zone in an AWS Account. Then from the context of that account you are able to manage the records through infrastructure as code. This will work until your using more than 1 AWS Account.

The problem

Because the Route53 hosted zone lives within a specific account. You cannot add records to the hosted zone from another account. So there are 3 options how you can solve this problem:

Option 1: Manage your DNS records by hand.
Option 2: Create a Route 53 hosted zone in each AWS Account. You will need entire different domains or subdomains to get this working.
Option 3: Use the cfn-cross-account-dns-provider

So Option 1, I like to use infrastructure as code. So this is not an option. And for Option 2 you will need, either a different domain. Or you will need to use subdomains. Using a different domain is a bit of an overkill. So when you use subdomains you will need a hosted zone in every account. A single hosted zone will cost you $ 6 a year. Let say you have 3 workloads and 4 environments per workload. Now you pay $ 72 per year only for the hosted zone. This might be an option for your use case but not for my personal pet projects.

So that brings us to option 3. In a nutshell its my solution to option 2.

My solution

I created a single AWS Account that I am using for DNS management. All the domains that I own are in this account, and are using Route53. I also deployed the cfn-cross-account-dns-provider in this account. This provider comes with a SNS Topic and all member accounts are allowed to publish to this topic.

Now you can use the resource definition in CloudFormation to create DNS records. It will use the SNS topic to invoke the Custom Cross Account DNS Provider in the AWS account that have the hosted zones. The provider will manage the records for you and will notify CloudFormation with the end result.

DNSRecord:
  Type: Custom::CrossAccountDNS
  Properties:
    ServiceToken: !Sub arn:aws:sns:${AWS::Region}:${HostedZoneAccountId}:binxio-cfn-cross-account-dns-provider
    HostedZoneId: !Ref HostedZoneId
    Name: mysubdomain.example.com
    Type: CNAME
    Value: my-value-from-cloudfront-for-example.amazon.com

Cool, so now that we can create DNS records cross account you can do more fun stuff. For example, you also need DNS records for certificates that you request through ACM.

When you are using CloudFront the certificate needs to exist in the us-east-1 region. This will become a problem when your application lives in a different region. For this Mark van Holsteijn created the cfn-certificate-provider. With this provider you can request a certificate in the us-east-1 region. While your stack lives in the eu-west-1 region. When you combine this with the cfn-cross-account-dns-provider your certificates will be issued automatically.

Conclusion

You can manage certificates and DNS records across account and region. All you need is the cfn-certificate-provider and the cfn-cross-account-dns-provider. This will remove any manual steps from your deployment making them more predictable and reliable.

Photo by BOA.vision

Deploy S3 hosted application using CodePipeline

Tue, 14 Mar 2023 11:00:00 +0100

A while ago I wrote how you can host your own single page application on S3. But how will you get your application on the S3 bucket? There are a couple of options here, you could upload it by hand? But we both know that is not the real solution here. No we want to automate this process! In this blog post I will show you how you can automate this using AWS CodePipeline.

The Pipeline

AWS Codepipeline uses different stages, I often use Source, Build and Deploy stages. In some cases I split the Deploy stage into a Development, Testing, Acceptance and Production deployment (also known as DTAP). If you want to know more about how you can set this up you can read my building applications with pipelines blog. But in the end it is up to you and what makes sense to your use-case.

When you deploy your infrastructure using CloudFormation. You can make use of the outputs within the CodePipeline. Another option is to use a naming convention. I like to use the outputs as it removes the need to define a name upfront. Making it more robust when you re-use snippets or deploy your infrastructure more than once.

Outputs:
  ApplicationBucketName:
    Value: !Ref ApplicationBucket

The next thing you need to define is a namespace on the action that deploys your infrastructure.

- Name: ExecuteChangeSet
  Region: eu-west-1
  RunOrder: 2
  RoleArn: !Sub arn:aws:iam::${DevelopmentAccountId}:role/cross-account-role
  Namespace: DevelopmentVariables
  ActionTypeId:
      Category: Deploy
      Owner: AWS
      Provider: CloudFormation
      Version: "1"
  Configuration:
      ActionMode: CHANGE_SET_EXECUTE
      RoleArn: !Sub arn:aws:iam::${DevelopmentAccountId}:role/cloudformation-execution-role
      StackName: !Sub ${ProjectName}-development
      ChangeSetName: !Sub ${ProjectName}-development-ChangeSet

By default CodePipeline will load the outputs in the given namespace. In this example that is DevelopmentVariables, so the ApplicationBucketName is available as: #{DevelopmentVariables.ApplicationBucketName}.

Deploy to S3

AWS provides a S3 Deploy action you can use this action to deploy an artifact in your pipeline to S3. You can create this artifact in a CodeBuild Project or you can use the source artifact.

I am using a cross account deployment strategy. For this reason I need to allow my cross-account-role to allow uploads to the S3 buckets. I am using a BucketPolicy for this:

- Sid: AllowPipelineToUpload
  Effect: Allow
  Action: s3:PutObject
  Principal:
    AWS: !Sub arn:aws:iam::${AWS::AccountId}:role/cross-account-role
  Resource: !Sub ${ApplicationBucket.Arn}/*

Note that the role and the bucket are living in the same account. The pipeline lives in my build/deployment account. So in the pipeline we need to configure the upload to S3:

- Name: Ireland-Uploadapplication
  Region: eu-west-1
  RunOrder: 3
  RoleArn: !Sub arn:aws:iam::${DevelopmentAccountId}:role/cross-account-role
  InputArtifacts:
    - Name: application
  ActionTypeId:
    Category: Deploy
    Owner: AWS
    Provider: S3
    Version: "1"
  Configuration:
    BucketName: "#{DevelopmentVariables.ApplicationBucketName}"
    Extract: true
    CacheControl: max-age=0, no-cache, no-store, must-revalidate

In this example I will use my artifact called application and extract the content in the S3 bucket. It will assume the role that we specify as RoleArn to perform the upload. I will also set the CacheControl so that CloudFront knows that it needs to serve the new content.

Conclusion

It is easy to use the S3 Deploy action to upload your content to a S3 bucket. It removes the need of using a CodeBuild project to upload the content. This will reduce cost and complexity, by not maintaining an extra CodeBuild project.

Cross account deployments using a Customer Managed KMS key

Mon, 27 Feb 2023 00:00:00 +0000

In my previous blog I wrote about application pipelines. These CodePipeline use a S3 bucket. What if you have a need for a customer managed key instead of the Amazon managed key? And you want to deploy the CloudFormation templates cross account? In this blog I will explain how you can achieve this.

The key itself

Lets start with the key itself. When CodePipeline moves artifacts from one stage to another it uses S3 to transfer the data. For this reason we need to allow S3 and CodePipeline to decrypt and encrypt the data:

- Action:
    - kms:Decrypt
    - kms:DescribeKey
    - kms:Encrypt
    - kms:ReEncrypt*
    - kms:GenerateDataKey*
  Effect: Allow
  Principal:
    AWS: "*"
  Resource: "*"
  Condition:
    StringEquals:
      aws:PrincipalOrgID: !Ref OrgId
      kms:ViaService:
        - Fn::Sub: s3.${AWS::Region}.amazonaws.com
        - Fn::Sub: codepipeline.${AWS::Region}.amazonaws.com

Next, CloudFormation will deploy these templates across account. For this reason we need to allow CloudFormation to decrypt the data.

- Action: kms:Decrypt
  Effect: Allow
  Principal:
    AWS: "*"
  Resource: "*"
  Condition:
    StringEquals:
      aws:PrincipalOrgID: !Ref OrgId
      kms:ViaService: !Sub cloudformation.${AWS::Region}.amazonaws.com

The samples shown here will limit the use of the key to the AWS Services within your own AWS organization. The S3 bucket needs to use the KMS key for encryption.

The pipeline

The CodePipeline needs to know that we are using KMS as an encryption method. For this reason we need to tell what key to use.

Pipeline:
  Type: AWS::CodePipeline::Pipeline
  Properties:
    ArtifactStores:
      - Region: eu-west-1
        ArtifactStore:
          Location: !Sub codepipeline-artifacts-${AWS::AccountId}-eu-west-1
          Type: S3
          EncryptionKey:
            Type: KMS
            Id: alias/bucket-key
      - Region: eu-central-1
        ArtifactStore:
          Location: !Sub codepipeline-artifacts-${AWS::AccountId}-eu-central-1
          Type: S3
          EncryptionKey:
            Type: KMS
            Id: alias/bucket-key

We are using the alias so that we do not have to deal with the key id.

Deploy cross account/region

You might have seen it already in the previous code snippet. When you deploy across region you need to configure a bucket and key per region. When you deploy across account you will need 2 roles.

A role to assume in the other account. (cross-account-role, configured on action level)
A role to use for the CloudFormation execution. (cloudformation-execution-role, configured in the configuration)

Actions:
  - Name: Ireland-CreateChangeSet
    Region: eu-west-1
    RunOrder: 1
    RoleArn: !Sub arn:aws:iam::${DevelopmentAccountId}:role/cross-account-role
    InputArtifacts:
      - Name: BuildArtifactAsZip
    ActionTypeId:
      Category: Deploy
      Owner: AWS
      Provider: CloudFormation
      Version: "1"
    Configuration:
      ActionMode: CHANGE_SET_REPLACE
      RoleArn: !Sub arn:aws:iam::${DevelopmentAccountId}:role/cloudformation-execution-role
      StackName: !Sub ${ProjectName}-$development
      ChangeSetName: !Sub ${ProjectName}-development-ChangeSet
      TemplatePath: BuildArtifactAsZip::packaged-template.yaml
      Capabilities: CAPABILITY_NAMED_IAM
      ParameterOverrides: |
        {
          "EnvType": "development"
        }
  - Name: Ireland-ExecuteChangeSet
    Region: eu-west-1
    RunOrder: 2
    RoleArn: !Sub arn:aws:iam::${DevelopmentAccountId}:role/cross-account-role
    ActionTypeId:
      Category: Deploy
      Owner: AWS
      Provider: CloudFormation
      Version: "1"
    Configuration:
      ActionMode: CHANGE_SET_EXECUTE
      RoleArn: !Sub arn:aws:iam::${DevelopmentAccountId}:role/cloudformation-execution-role
      StackName: !Sub ${ProjectName}-development
      ChangeSetName: !Sub ${ProjectName}-development-ChangeSet

The cross-account-role needs the ability to perform crud operations on CloudFormation. Have access to the templates and the ability to pass the cloudformation-execution-role. The BuildAccountId is the account id that holds the KMS keys, S3 buckets and pipeline. This role needs to be able to access the S3 buckets and use the KMS key for decryption.

CrossAccountRole:
  Type: AWS::IAM::Role
  Properties:
    RoleName: cross-account-role
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Action: sts:AssumeRole
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${BuildAccountId}:root
    Policies:
      - PolicyName: AllowCloudFormationDeployments
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action: iam:PassRole
              Resource: !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cloudformation-execution-role
            - Effect: Allow
              Action: cloudformation:DescribeStacks
              Resource:
                - !Sub arn:${AWS::Partition}:cloudformation:eu-west-1:${AWS::AccountId}:stack/*
                - !Sub arn:${AWS::Partition}:cloudformation:eu-central-1:${AWS::AccountId}:stack/*
            - Effect: Allow
              Action:
                - cloudformation:CreateStack
                - cloudformation:DeleteStack
                - cloudformation:UpdateStack
                - cloudformation:CreateChangeSet
                - cloudformation:ExecuteChangeSet
                - cloudformation:DeleteChangeSet
                - cloudformation:DescribeChangeSet
                - cloudformation:SetStackPolicy
                - cloudformation:SetStackPolicy
                - cloudformation:ValidateTemplate
              Resource:
                - !Sub arn:${AWS::Partition}:cloudformation:eu-west-1:${AWS::AccountId}:stack/*
                - !Sub arn:${AWS::Partition}:cloudformation:eu-central-1:${AWS::AccountId}:stack/*
      - PolicyName: AllowArtifactAccess
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - s3:GetObject
                - s3:GetObjectVersion
              Resource:
                - !Sub arn:${AWS::Partition}:s3:::codepipeline-artifacts-${BuildAccountId}-eu-west-1/*
                - !Sub arn:${AWS::Partition}:s3:::codepipeline-artifacts-${BuildAccountId}-eu-central-1/*
            - Effect: Allow
              Action: kms:Decrypt
              Resource:
                - !Sub arn:${AWS::Partition}:kms:eu-west-1:${BuildAccountId}:key/02d730ed-125b-4d2e-a498-7b0187298924
                - !Sub arn:${AWS::Partition}:kms:eu-central-1:${BuildAccountId}:key/e127b617-6951-42b7-83ae-12acc16d12a7

The execution role has all the needed rights to deploy the actual template:

CloudFormationServiceRole:
  Properties:
    RoleName: cloudformation-execution-role
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Action: sts:AssumeRole
          Effect: Allow
          Principal: { Service: cloudformation.amazonaws.com }
    ManagedPolicyArns:
      - !Sub arn:${AWS::Partition}:iam::aws:policy/AdministratorAccess

Conclusion

When using customer managed keys you need some extra configuration. The key is self, configuration where is is used and some logical permissions. Getting all this from the AWS documentation can be challenging as it is somewhat of an edge case.

Building applications with pipelines

Mon, 30 Jan 2023 11:00:00 +0100

In larger organizations a lot of things are in place. Think of things like deployment pipelines for your applications. But how do you set those up using the services AWS provides? In this blog post I will guide you to how I do it for my pet projects.

I am a big fan of CloudFormation so the examples I will show are CloudFormation snippets.

The git repository

I am using AWS CodeCommit as a source repository. The reason for this choice is that it integrates with AWS CodePipeline. We also want the ability to test our infrastructure. So I usually use feature branches to do that. Here the first challange appears. We only need 1 repository and a pipeline to deploy our infrastructure. But we also need a pipeline that deploys our feature branch. In CloudFormation you can do this using conditions.

Parameters:
  FeatureGitBranch:
    Type: String
    Default: ""
  ProjectName:
    Type: String
    Default: ""   
Conditions:
  IsMainBranchPipeline: !Equals [!Ref FeatureGitBranch, ""]
  IsFeatureBranchPipeline: !Not [Condition: IsMainBranchPipeline]

By providing a FeatureGitBranch parameter we can now make some choices. For example we could only create the repository in the production pipeline.

Resources:
  PipelineRepo:
    Condition: IsMainBranchPipeline
    Type: AWS::CodeCommit::Repository
    Properties:
      RepositoryName: !Ref ProjectName
      RepositoryDescription: <repository-description>

The pipeline requirements

The pipeline uses a couple of resources. You can share these resources across your pipelines. So they are in a separate template and I deploy these resources once per region that I am using.

First you will need a S3 bucket to store the artifacts created by each stage.

ArtifactsBucket:
  Type: AWS::S3::Bucket
  DeletionPolicy: Retain
  UpdateReplacePolicy: Retain
  Properties:
    BucketName: !Sub codepipeline-artifacts-${AWS::AccountId}-${AWS::Region}
    VersioningConfiguration:
      Status: Enabled
    BucketEncryption:
      ServerSideEncryptionConfiguration:
        - ServerSideEncryptionByDefault:
            SSEAlgorithm: AES256

I am using a predictable bucket name so I can use it in my pipeline. Because bucket names need to have a globally unique name I also include the account id and region.

You will also need an IAM role. You will use this role in your pipeline to deploy your CloudFormation template.

CloudFormationServiceRole:
 Type: AWS::IAM::Role
 Properties:
   RoleName: !Sub cloudformation-execution-role-${AWS::Region}
   AssumeRolePolicyDocument:
     Version: "2012-10-17"
     Statement:
       - Action: sts:AssumeRole
         Effect: Allow
         Principal: { Service: cloudformation.amazonaws.com }
   ManagedPolicyArns:
     - arn:aws:iam::aws:policy/AdministratorAccess

Again I use a predictable name and I included the region. Since role names are unique in your account you can deploy this in every region. For the sake of simplicity I used the AdministratorAccess managed policy. You might want to limit this for your own scope.

The pipeline

The pipeline itself also needs a role. This role can pass the CloudFormation role and use the buckets created in the previous step.

CodePipelineExecutionRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Action: "sts:AssumeRole"
          Effect: Allow
          Principal:
            Service:
              - codepipeline.amazonaws.com
    Policies:
      - PolicyName: CodePipelineAccess
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Action: "iam:PassRole"
              Resource: !Sub cloudformation-execution-role-${AWS::Region}
      - PolicyName: CodePipelineCodeAndS3Bucket
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Action:
                - s3:GetBucketAcl
                - s3:GetBucketLocation
              Effect: Allow
              Resource:
                - !Sub arn:${AWS::Partition}:s3:::codepipeline-artifacts-${AWS::AccountId}-eu-west-1
                - !Sub arn:${AWS::Partition}:s3:::codepipeline-artifacts-${AWS::AccountId}-eu-central-1
            - Action:
                - "s3:GetObject"
                - "s3:GetObjectVersion"
                - "s3:PutObject"
              Effect: Allow
              Resource:
                - !Sub arn:${AWS::Partition}:s3:::codepipeline-artifacts-${AWS::AccountId}-eu-west-1/*
                - !Sub arn:${AWS::Partition}:s3:::codepipeline-artifacts-${AWS::AccountId}-eu-central-1/*
      - PolicyName: CodeCommitAccess
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Action:
                - codecommit:GitPull
                - codecommit:GetBranch
                - codecommit:GetCommit
                - codecommit:UploadArchive
                - codecommit:GetUploadArchiveStatus
              Effect: Allow
              Resource:
                - !Sub arn:${AWS::Partition}:codecommit:${AWS::Region}:${AWS::AccountId}:${ProjectName}
                - !Sub arn:${AWS::Partition}:codecommit:${AWS::Region}:${AWS::AccountId}:${ProjectName}/*
      - PolicyName: CodePipelineCodeBuildAndCloudformationAccess
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Action:
                - "codebuild:StartBuild"
                - "codebuild:BatchGetBuilds"
              Resource:
                - !GetAtt CodeBuildProjectBuildAndPackage.Arn
            - Effect: Allow
              Action:
                - "cloudformation:DescribeStacks"
              Resource: !Sub "arn:${AWS::Partition}:cloudformation:eu-*:${AWS::AccountId}:stack/*"
            - Effect: Allow
              Action:
                - "cloudformation:CreateStack"
                - "cloudformation:DeleteStack"
                - "cloudformation:UpdateStack"
                - "cloudformation:CreateChangeSet"
                - "cloudformation:ExecuteChangeSet"
                - "cloudformation:DeleteChangeSet"
                - "cloudformation:DescribeChangeSet"
                - "cloudformation:SetStackPolicy"
                - "cloudformation:SetStackPolicy"
                - "cloudformation:ValidateTemplate"
              Resource:
                - !Sub "arn:${AWS::Partition}:cloudformation:eu-*:${AWS::AccountId}:stack/${ProjectName}-*/*"

As you can see in the snippet the role is also allowed to start a CodeBuild project. And it can deploy a CloudFormation template. You might have noticed that I used 2 regions in this role. This is because in this particular example I have a pipeline that uses 2 regions.

Because we use 2 regions we also need 2 artifact stores.

Pipeline:
  Type: AWS::CodePipeline::Pipeline
  Properties:
    ArtifactStores:
      - Region: eu-west-1
        ArtifactStore:
          Location: !Sub codepipeline-artifacts-${AWS::AccountId}-eu-west-1
          Type: S3
      - Region: eu-central-1
        ArtifactStore:
          Location: !Sub codepipeline-artifacts-${AWS::AccountId}-eu-central-1
          Type: S3
    RoleArn: !GetAtt CodePipelineExecutionRole.Arn
    RestartExecutionOnUpdate: true
    Stages:
      - Name: Source
        Actions:
          - Name: SourceCodeRepo
            ActionTypeId:
              Category: Source
              Owner: AWS
              Version: 1
              Provider: CodeCommit
            Configuration:
              PollForSourceChanges: False
              RepositoryName: !Ref ProjectName
              BranchName: !If [IsFeatureBranchPipeline, !Ref FeatureGitBranch, "main"]
            OutputArtifacts:
              - Name: SourceCodeAsZip
            RunOrder: 1

Depending on the IsFeatureBranchPipeline condition we select the correct branch for this pipeline. Depending on your project you might need a build step. I use CodeBuild to run unit tests and build and package any resources needed.

      - Name: BuildAndPackage
        Actions:
          - Name: CodeBuild
            ActionTypeId:
              Category: Build
              Owner: AWS
              Provider: CodeBuild
              Version: "1"
            Configuration:
              ProjectName: !Ref CodeBuildProjectBuildAndPackage
            InputArtifacts:
              - Name: SourceCodeAsZip
            OutputArtifacts:
              - Name: BuildArtifactAsZip

I did not include the CodeBuild project and IAM role snippet for CodeBuild. I will do a blog post on that in the future.

So lets go over to the fun part. The next stage is the actual deployment of the CloudFormation template. I am using 2 regions and I only want to deploy to these 2 regions for my production environment. I also have different parameters for my production environment. Again, I am using conditions to control this behaviour.

          - !If
            - IsMainBranchPipeline
            - Name: Production
              Actions:
                - Name: Ireland-CreateChangeSet
                  RunOrder: 1
                  Region: eu-west-1
                  InputArtifacts:
                    - Name: BuildArtifactAsZip
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Provider: CloudFormation
                    Version: "1"
                  Configuration:
                    ActionMode: CHANGE_SET_REPLACE
                    RoleArn: !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cloudformation-execution-role
                    StackName: !Sub ${ProjectName}-production
                    ChangeSetName: !Sub ${ProjectName}-production-ChangeSet
                    TemplatePath: BuildArtifactAsZip::packaged-template.yaml
                    Capabilities: CAPABILITY_NAMED_IAM
                    ParameterOverrides: |-
                      {
                        "EnvType": "production"
                      }
                - Name: Frankfurt-CreateChangeSet
                  RunOrder: 1
                  Region: eu-central-1
                  InputArtifacts:
                    - Name: BuildArtifactAsZip
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Provider: CloudFormation
                    Version: "1"
                  Configuration:
                    ActionMode: CHANGE_SET_REPLACE
                    RoleArn: !Sub arn:aws:iam::${AWS::AccountId}:role/cloudformation-execution-role
                    StackName: !Sub ${ProjectName}-production
                    ChangeSetName: !Sub ${ProjectName}-production-ChangeSet
                    TemplatePath: BuildArtifactAsZip::packaged-template.yaml
                    Capabilities: CAPABILITY_NAMED_IAM
                    ParameterOverrides: |
                      {
                        "EnvType": "production"
                      }
                - Name: Ireland-ExecuteChangeSet
                  RunOrder: 2
                  Region: eu-west-1
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Provider: CloudFormation
                    Version: "1"
                  Configuration:
                    ActionMode: CHANGE_SET_EXECUTE
                    RoleArn: !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cloudformation-execution-role
                    StackName: !Sub ${ProjectName}-production
                    ChangeSetName: !Sub ${ProjectName}-production-ChangeSet
                - Name: Frankfurt-ExecuteChangeSet
                  RunOrder: 2
                  Region: eu-central-1
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Provider: CloudFormation
                    Version: "1"
                  Configuration:
                    ActionMode: CHANGE_SET_EXECUTE
                    RoleArn: !Sub arn:aws:iam::${AWS::AccountId}:role/cloudformation-execution-role
                    StackName: !Sub ${ProjectName}-production
                    ChangeSetName: !Sub ${ProjectName}-production-ChangeSet
            - !Ref AWS::NoValue

In the 1 action I create a change set. In the second action I execute it. Please note the RoleArn used here. You could also specify a role from a different account. This allows you to use a separate account for different environments. For example you could host the pipeline in a deployment account. And have separate development, testing, acceptance and production accounts.

The feature branch

So we now have a working pipeline for the production environment. How about the feature branch? For this we use again the conditions.

          - !If
            - IsFeatureBranchPipeline
            - Name: FeatureDevelopment
              Actions:
                - Name: Ireland-CreateChangeSet
                  RunOrder: 1
                  Region: eu-west-1
                  InputArtifacts:
                    - Name: BuildArtifactAsZip
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Provider: CloudFormation
                    Version: "1"
                  Configuration:
                    ActionMode: CHANGE_SET_REPLACE
                    RoleArn: !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cloudformation-execution-role
                    StackName: !Sub ${ProjectName}-${FeatureName}
                    ChangeSetName: !Sub ${ProjectName}-feature-ChangeSet
                    TemplatePath: BuildArtifactAsZip::packaged-template.yaml
                    Capabilities: CAPABILITY_NAMED_IAM
                    ParameterOverrides: |
                      {
                        "EnvType": "development"
                      }
                - Name: Ireland-ExecuteChangeSet
                  RunOrder: 2
                  Region: eu-west-1
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Provider: CloudFormation
                    Version: "1"
                  Configuration:
                    ActionMode: CHANGE_SET_EXECUTE
                    RoleArn: !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cloudformation-execution-role
                    StackName: !Sub ${ProjectName}-${FeatureName}
                    ChangeSetName: !Sub ${ProjectName}-feature-ChangeSet
            - !Ref AWS::NoValue

As you can see, the feature branch is only deployed to the eu-west-1 region. And it has set the EnvType parameter to development.

Pickup new commits

When you push your commits to the CodeCommit repository. You can use an event rule to trigger a new pipeline execution. To make this happen we need an IAM role and the rule itself.

CloudWatchEventRole:
   Type: AWS::IAM::Role
   Properties:
     AssumeRolePolicyDocument:
       Version: '2012-10-17'
       Statement:
       - Effect: Allow
         Principal: { Service: events.amazonaws.com }
         Action: sts:AssumeRole
     Policies:
       - PolicyName: root
         PolicyDocument:
           Version: '2012-10-17'
           Statement:
             - Action: codepipeline:StartPipelineExecution
               Resource: !Sub arn:${AWS::Partition}:codepipeline:${AWS::Region}:${AWS::AccountId}:${Pipeline}
               Effect: Allow

CloudWatchEventRule:
  Type: AWS::Events::Rule
  Properties:
    Description: Check CodeCommit Repo Changes
    EventPattern:
      detail-type:
        - "CodeCommit Repository State Change"
      source:
        - aws.codecommit
      resources:
        - !Sub arn:${AWS::Partition}:codecommit:${AWS::Region}:${AWS::AccountId}:${ProjectName}
      detail:
        event:
          anything-but:
            - referenceDeleted
        referenceType:
          - branch
        referenceName:
          - !If [IsFeatureBranchPipeline, !Ref FeatureGitBranch, "main"]
    Targets:
      - Id: codepipeline
        Arn: !Sub arn:${AWS::Partition}:codepipeline:${AWS::Region}:${AWS::AccountId}:${Pipeline}
        RoleArn: !Sub ${CloudWatchEventRole.Arn}

We are triggering on every event except when the branch removal.

Conclusion

You can store the template for the pipeline in your CodeCommit repository. When you create a feature branch you can deploy that template specifying the branch name. The pipeline can then deploy the project CloudFormation template. And with every commit that you push your feature environment will receive an update.

With this setup you have everything in a single repository. (Except for the shared S3 bucket and IAM role.) Having everything in a single repository makes it easier to maintain.

Use EventBridge to handle API requests

Tue, 03 Jan 2023 12:00:00 +0100

At AWS re:Invent I attended SVS312 Building Serverlesspresso. This talk was about how you can create event-driven architectures. I am a fan of event-driven architectures. So I wanted to highlight a interesting technique using EventBridge shown in this talk.

In the past I used a similar technique that I talked about in an AWS Meetup. Here I used an API Gateway to post the request to a SQS Queue. This is very useful for fire and forget patterns. Because you do not need to invoke a Lambda function to handle the request. The response to the user is faster and you do not need to pay for the Lambda invocation.

But James Beswick showed a iteration on that pattern. Instead of posting your payload to a SQS Queue you can also create an event within EventBridge. The initial request can still be a fire and forget call, but James did something interesting here.

Because the API request itself became an event, you are able to process the API request as an event.

A closer look

All examples are coming from the actual serverless-coffee repository from AWS

When you order a coffee a POST call is executed. The API Gateway uses the x-amazon-apigateway-integration extension. This enables you to pass the body of the request as the details of an EventBridge event.

From here you can do all sorts of actions. For example, the serverless-coffee project used IOT Core. With IOT Core you can notify the end-user with status updates. And notify the barista that what kind of coffee needs to be created.

The advantage of this is that these two actions are completely decoupled from each other. In fact the initial request is decoupled from these two actions. This makes this pattern very extendable! You do not need to take into account what other services are there.

Conclusion

Using EventBridge to handle your API requests can be very powerful. But it does need you to poll for updates or use streamed updates using IOT Core.

When you see what kind of flexibility you get extendability it is worth it!

Photo by Josh Sorenson

Don't let security slow you down

Fri, 16 Dec 2022 10:00:00 +0100

In my previous blog I wrote on how AWS embeds security in their teams. Jorge Liauw Calo challenged me to tell me a bit more on my security vision. So why not do another blog post ;-).

Some background, I have been working with AWS since 2009. So most of my opinions are formed by the input from AWS. This is not a bad thing because in general AWS is doing a great job developing safe and secure services. At AWS re:Invent 2022 I also heard the following quote:

Trust but verify!

I think that sums up my personal take on security as well. I mixed the input from AWS with my own experience as a developer and topped it off with my consulting experience.

Getting blocked

Traditional security departments tent to oppose al sorts of rules and frameworks on the development teams. The problem with this approach is that there are more developers than security engineers. This means that the security engineers do not have the time to guide the developers. The end result is that your workload cannot go into production because security would block you. And to be fair, they have good reasons to do so!

It is in the interest of the developer to embed security in your requirements. Based on those requirements you make sure you incorporate the security aspect in your design. Afterwards you can build your solution still with security in mind. By doing this you are not only lowering the risk of being blocked by a security team. You are also preventing the need of fixing the security issues after the fact. The effort that is needed to fix certain security issues is higher when you already build your solution than during the requirement and design phase of that same solution.

You should not rely on the control mechanisms opposed by the security team. You need to think for your self. Your team knows what your solution is doing and therefor you are the perfect candidate to secure it. In general those security team controls will work but in specific cases they are not enough.

You can use a lock, but you need to use it correctly to have effect!

Dependencies

When developing your solutions you will have to deal with dependencies. These dependencies might have vulnerabilities in them. Often you see that a vulnerability becomes a hot topic and the urgency goes up. It becomes something that needs to be fixed NOW!

Having tools in place to detect and notify, will reduce problems in the future. This allows you to prioritize, plan and test the mitigation. Instead of rushing a fix and hoping nothing will break. When these things happen it’s not only the time that you spend on the actual fix. You also need to mobilize your teams on the spot. This will pull them out of there current task wasting more and more time.

By enabling the developers to proactively update their dependencies. They will gain control over this process and they are less likely to get distracted by urgent required fixes. This is good for the team morale and the development velocity.

By showing developers what the added value is for them. It is more likely that they will start adopting these security principles. At the same time you will improve your security posture, and increase your development velocity!

Conclusion

When you also focus on security during the requirement and design phase, you will safe time in the future! This time can be used to develop new features improving your developing velocity while improving your security posture.

Embed security in your teams

Wed, 07 Dec 2022 20:00:01 +0000

During AWS re:Invent I attended the SEC322: Transform builder velocity with security chalk talk. I would have pointed you to the recording on YouTube. But unfortunately, chalk talks are not recorded! So instead you have to do it with my interpretation of the session.

The title of the session was Transform builder velocity with security. Most of you might recognize the feeling of being stopped by security when you want to go-live with your application. We tent to blame the security team at this point saying that they are the reason for the delay. The problem with this is that it has a huge impact on your development velocity. You are ready to deploy, but you can’t. And there is that potential risk that you need to go back to the drawing board because you made a fundamental mistake in your design.

The solution seems strait forward. Include security earlier in the process. And yes, this will work in the beginning. But when your organization starts to grow you will face scaling issues in your security team.

So, when you talk about scale you could use AWS itself as an example. They have a lot of teams and they seem to do a good job. And AWS always stated that there is a security person on each team. But I never heard how they actually do this. This all changed for me in this session.

Guardians

In this session Collin Pace and Anya Khlobystina explained that each team should have a guardian. The guardian will ask all relevant security questions to the team. So they will play the role of the security person on the team. So where is this guardian coming from? Well, from the team! In theory anyone can become a guardian.

AWS relies on the ambitions of the individuals to fulfill this role. The reason behind this is simple. When you have someone how wants to do the job from ambition they are more invested versus an appointed person. Ambition persons bring that spark that drives them, that motivates them to be the best version of them selves.

These guardians are supported by other more senior guardians. And there are training courses available to support the guardians in their journey.

The whole idea of having these guardians in the actual teams is to identify potential security issues as soon as possible. In the design phase, but also in all the steps that follow. They know the workload that they are building so they have the best insights on why and how the workload is build.

Conflicting interests

So like in any organization you might have a conflict between the security team and a product owner that wants to ship.

When working with guardians this shifts to the guardian. But a guardian can always ask the security team to confirm that the concern is legit. The only difference is that you are still in the design phase, or in the implementation phase. You are not ready for production yet, this is a big difference!

The team is not stopped, only a specific area is being discussed. So the rest of the team can still continue on making sure you are ready for a release.

Conclusion

With the use of guardians you are embedding the security role in your teams. This will allow you to act faster on potential security risks. And prevent them from being build in the first time.

The time that you save, will be used to develop your workload. And this will increase your building velocity!

Photo by cottonbro studio

Write AWS Config rules

Fri, 25 Nov 2022 11:00:00 +0100

AWS Config rules allow you to determine if a resource is compliant or not. Previously when you wanted to do custom checks you needed to write AWS Lambda functions to validate the configuration of a resource. Since Aug 2, 2022 you have the ability to use cfn-guard rules to achieve the same.

Why should I use cfn-guard

Writing a Lambda function is not difficult. But it becomes harder when you add unit testing. When you determine your compliancy status based on Lambda function, you want to ensure that it works as intended. This is the advantage of cfn-guard, you write a rule and you write some tests. These tests are a couple of snippets of yaml and per snippet you define if your rule should: PASS, FAIL or SKIP.

The other advantage of cfn-guard is that you have a single way of writing rules. On Nov 8, 2022 I gave a talk on this topic on the AWS User Group NL. In this talk I highlighted the differences between detective and preventive controls. Using cfn-guard rules in AWS Config is a detective control. But the same language can be used to build preventive controls!

Yes, I said language and not rules

It turns out that the rules written for AWS Config cannot be used to validate AWS CloudFormation templates. They are similar but different! Where an AWS Config rule has the following structure:

{
  "version": "1.3",
  "arn": "arn:aws:s3:::aws-meetup-2022-11-08-noncompliantbucket-xxxxxxxxxxxxx",
  "resourceType": "AWS::S3::Bucket",
  "resourceId": "aws-meetup-2022-11-08-noncompliantbucket-xxxxxxxxxxxxx",
  "resourceName": "aws-meetup-2022-11-08-noncompliantbucket-xxxxxxxxxxxxx",
  "awsRegion": "eu-west-1",
  "supplementaryConfiguration": {
    "ServerSideEncryptionConfiguration": {
        "rules": [
            { "applyServerSideEncryptionByDefault": {"sseAlgorithm": "aws:kms"} }
        ]
    }
  }
}

A CloudFormation template has the following structure:

{
    "Resources": {
        "NonCompliantBucket": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
                "BucketEncryption": {
                    "ServerSideEncryptionConfiguration": [
                        {
                            "ServerSideEncryptionByDefault": {
                                "SSEAlgorithm": "aws:kms"
                            }
                        }
                    ]
                }
            }
        },
        "OtherResource": { ... }
    }
}

The first thing that you can see is that the AWS Config rule only has the context of the resource it validates. While the CloudFormation template has one or more resources. So you can compare the to locations as followed:

supplementaryConfiguration vs Resources > NonCompliantBucket > Properties

But looking closer you also notice that the structure and naming varies. Meaning you are doomed to maintain 2 types of rules. 1 for AWS Config (Detective) and 1 for CloudFormation (Preventive).

Why should you write a detective and a preventive rule?

The answer is to decrease your feedback cycle. For example, when you only have a detective rule you need to wait until the everything is deployed. Then you need to wait until a compliance issue pops up. When you use a preventive rule you can check this on different levels:

You can run the cfn-guard rules at any time on your local development environment.
You can use a pre-commit hook, this will prevent you from committing non-compliant code.
You can run it during your Continuous Integration step. This prevents merging non-compliant code in your develop/main branches.
You can run it before deployment to testing, acceptance and/or production.

Conclusion

cfn-guard is a great tool to write compliancy checks for both detective and preventive controls. Interested in a working sample? I published my demo material to the Nr18/aws-meetup-2022-11-08 GitHub repository.

Photo by Joshua Miranda

Speedup python cli projects

Thu, 27 Oct 2022 11:00:00 +0200

Use our python-cli-tool cookiecutter project. Get an python cli project up and running in a few steps. In my previous speedup-event-driven-projects blog. I wrote how you could use cookiecutter to spin up an event driven project for AWS. In this blog post I will show you how you could do this for python cli projects.

For this blog post I assume that you have the following installed:

python3
cookiecutter
poetry

Lets create an cli project

We will begin with executing the cookiecutter command:

cookiecutter gh:binxio/python-cli-tool

Cookiecutter will create a folder based on the project_name. In this folder you now only need to install the dependencies. You can use the following command:

make install

The install target of the Makefile will:

Perform a poetry init, this will create the pyproject.toml file.
Create git repository and install the pre-commit hooks.
Perform a poetry install. This will install all project dependencies based on the pyproject.toml file.
Adds extra configuration to the pyproject.toml file.
Commits everything to the git repository.

You will end up with something like this:

The project itself has unit testing in-place:

And you can test your cli tool as well by executing the commands through poetry:

Now it’s time to configure your remote GitHub repository:

git remote add origin git@github.com:binxio/my-example-cli.git
git branch -M main
git push -u origin main
git checkout -b develop
git push --set-upstream origin develop

We pushed the main branch to the remote and created a develop branch as a working branch.

The project itself contains the following workflows:

Continuous Integration, checks when you push to main or open a pull request to develop.
Create Release, creates an release for you when you merge to main and uploads it to PyPi.org. (You need to configure a PYPI_API_TOKEN secret in GitHub for this to work)
Release Notes, creates a proposal for your release notes. This is based on your merged pull requests. So make sure you use correct messages.

Conclusion

By using the python-cli-tool cookiecutter project you will speedup your python cli project!

And you can focus on what matters! Your business logic!

Photo by Ekaterina Belinskaya

Speedup event driven projects

Tue, 11 Oct 2022 11:00:00 +0200

Use our sam-eventbridge-project cookiecutter project. Get an event driven project up and running in a few steps. On the 3rd of October the AWS Community Day NL was held in Amersfoort. I gave a talk about CodeCommit and how you could use events to make your life easier.

The “hidden” message of the presentation was that you need to find a trigger. This trigger could be an event send to EventBridge.

Most of the times when I start something new. I copy an other project and then change it to my likings. After I gave my presentation at the Community Day. I realized that that was a “problem”! So, I automated the wanted result.

Now, when I want to start a new project I execute: (as a trigger)

cookiecutter gh:binxio/sam-eventbridge-project

Cookiecutter will then create the project structure for me. It contains the following things:

AWS SAM, used to build and deploy the CloudFormation stack.
Sample implementation of a CloudFormation Stack event’s.
Unit tests, validates the sample implementation.
Linting and formatting, keeps your code clean and follows style guides.
Complexity scanning, prevents your code to become to complex.
Makefile to simplify the usage of the project.

By using the sam-eventbridge-project cookiecutter project you will speedup your event driven projects!

Read speedup python cli projects to learn more…

Photo by Pixabay

How to work with different browser profiles

Thu, 22 Sep 2022 12:00:00 +0000

Finicky will help you when you use different browsers, and/or different profiles. As a consultant I often use browser profiles to split my clients. I have a “Binx.io” profile for all Binx related work. For each client I have corresponding profiles. Let’s call them “Client A” and “Client B”.

Browser profiles let you manage bookmarks per profile. So you can keep all your related bookmarks together making them easier to find. The biggest advantage that I love is that the sessions are also separated. This means you can login with different users into the AWS Console. Allowing you to use 2 console sessions at the same time.

How does it work?

First you need to install Finicky. Then you configure your default browser to Finicky.

The last thing you need to do is to configure some logic. For this you need to create a ~/.finicky.js file. This file contains the logic when to select what browser and profile.

module.exports = {
  defaultBrowser: "Safari",
  handlers: [
    {
      match: /(binx.io|xebia.com)/,
      browser: {
        name: "Google Chrome",
        profile: "Profile 1"
     }
    },
    {
      match: /(clientA.com)/,
      browser: {
        name: "Google Chrome",
        profile: "Profile 2"
     }
    },
    {
      match: /(clientB.com)/,
      browser: {
        name: "Google Chrome",
        profile: "Profile 3"
     }
    }
  ]
};

If you look at the code snippet. You will notice that the default browser is set to Safari.

When I click on a link or I open a webpage on my machine. This logic kicks in. If the url matches binx.io or xebia.com it will use Chrome and Profile 1. If the url contains clientA.com it will use Chrome and Profile 2.

The profiles mentioned earlier Client A and Client B are the names. Unfortunately, Finicky only works with numbered profiles. So you need to do some testing what profile number is what profile name. But when you figured it out you are good to go!

Conclusion

Finicky gives you the flexibility to choice your browser and profile. All based on logic that you control.

And that combination is really powerful.

Photo by Exelsius Adam

Use a help target in your Makefile

Thu, 08 Sep 2022 16:30:00 +0200

In one of my previous blog. I wrote how you could make your life easier when you start using Makefile. But when you start using Makefile in many projects. The targets that you use may vary from project to project.

In this blog post I will address a trick how you could write a help target. Lets imagine that we have a project that allow you to build, start and stop a docker container. These names are pretty straightforward. But without looking in the Makefile you never know for sure.

By calling the help target you could list all available targets:

make help

When you have a look at the Makefile used for this blog post. It looks like this:

.DEFAULT_GOAL:=help
.PHONY: help
help:  ## Display this help
	$(info Example Makefile for my blog post)
	awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n  make \033[36m<target>\033[0m\n"} /^[a-zA-Z0-9_-]+:.*?##/ { printf "  \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)

.PHONY: build
build: ## Build the container image
	docker build -t my-container:latest .

.PHONY: stop
stop: ## Stop the container
	docker stop my-named-container > /dev/null 2>&1 || True
	docker rm my-named-container > /dev/null 2>&1 || True

.PHONY: start
start: stop ## Start the container
	docker run --name my-named-container my-container:latest > /dev/null

.PHONY: shell
shell: ## Start a shell session on the container
	docker run -it my-named-container:latest bash

.PHONY: logs
logs: ## Tail the logs of the running container
	docker logs my-named-container -f

$(VERBOSE).SILENT:

You need to add the help target. And for each target you need to supply a help text. And each target needs a prefix with ##.

By adding a help target to your Makefile you make it easier to use for others. But also for yourself because you don’t need to remember each target in the Makefile.

Photo by lalesh aldarwish

Learning HashiCorp Terraform on AWS

Mon, 22 Aug 2022 00:00:00 +0200

I have been working as a consultant for over a year now. And I noticed that a lot of our clients are asking for experience on HashiCorp Terraform and AWS. So I figured it might be time to learn something new!

So when I started to look at Terraform I never used it. What I did have is extensive knowledge of AWS. And like any other infrastructure as code. You need to know the cloud provider to actually build proper infrastructure. Tools like Terraform, CloudFormation and CDK are tools to build infrastructure. Those can be learned and you can switch between them depending on the project needs.

Terraform introduction

So I started out with reaching out to my colleagues Dean Shanahan and Bruno Schaatsbergen. They explained the basics on how Terraform works. Let me recap that for you in a short and simple story:

In Terraform you declare your infrastructure. The declarations are translated in API calls towards the cloud provider. So, if you know how to build infrastructure in your cloud provider. You know what is needed in Terraform, you only need to lookup the syntax.

So what is the difference between Terraform and CloudFormation then? When you remove the actual DSL (Domain Specific Language). Then the only difference is that CloudFormation manages the state for you. While in Terraform the state is managed by Terraform. Meaning you have the responsibility of storing it somewhere.

So what is this state that I mentioned?

Imagine that we want to build a S3 bucket. When you do this using the AWS Console you follow the wizard. The console uses the CreateBucket API call to create the bucket. When you create a bucket using CloudFormation. You hand over a template to the CloudFormation service. The service reads the template. It checks if the bucket we added is already in the current version. Since we added it it is not, so CloudFormation now knows it needs to perform a CreateBucket. This lookup action is done against the state.

The same principle applies to Terraform. When you add a resource that is not known in the state it will create it. When the resource is known in the state it will update it.

When you are just playing around with Terraform you can have the state in a local file. But when you are dealing with teams and pipelines you do want your state to be stored somewhere central. You can do this using a S3 Bucket and a DynamoDB table. The S3 bucket will contain the state file. The DynamoDB table is used for state locking. The state locking prevents 2 deployments from happening at the same time.

Learning Terraform

I followed the Terraform Associate Certification Exam Preparation on Cloud Academy. But HashiCorp Certified Terraform Associate from A Cloud Guru also works. Or you can just play around reading the documentation.

Within a couple of hours I already became comfortable enough to state. I know terraform! There are so many providers available nobody knows them all. You shouldn’t learn them by memory because the providers receive updates. So it’s always better to rely on the documentation than your memory.

In the end when you join an existing team you need to learn what they are actually building. Once you know that you can have a look on how they did that in Terraform.

Using Terraform

So one thing that I really liked from Terraform. You can do pre and post actions when you create resources. This is super useful when for example you create an Elastic Container Registry. After you created the repository you can build a Docker image and upload it into the repository.

But also when you delete resources. For example, when you delete a S3 Bucket. The bucket needs to be empty. With Terraform you can first remove all the objects and then let Terraform remove the bucket.

Terraform also has native drift detection. For example, when you add tags to a resource using the console. And if you provision an update using Terraform you will get a notification that that tag will be removed.

That being said, I usually use CodePipeline for my deployments. CodePipeline has a good integration with CloudFormation. If you want to deploy your Terraform code in a CodePipeline. You will need to use CodeBuild to perform the Terraform deployment. And there is some additional work needed to manage the state properly.

Conclusion

CloudFormation and Terraform are really comparable products in what they do. If you know how to build your infrastructure in your cloud provider. Learning Terraform and/or CloudFormation is easy. It’s just declaring what you want to build.

The more you use it the easier it gets because you gain experience in the tool. But in the end it’s just a tool and if a better one comes along. You can easily switch because you know the underlying cloud provider.

Photo by Pixabay

IAM User credentials with MFA

Tue, 26 Jul 2022 09:00:00 +0000

When you use IAM user for programmatic access. You have an ACCESS_KEY and a SECRET_ACCESS_KEY. With these credentials you are able to interact with AWS. A common pattern is that you will use the IAM User to assume roles in the same or even in different AWS accounts. The CLI configuration would look something like this:

[profile my-role]
source_profile = default
role_arn = arn:aws:iam::111122223333:role/Administrator
mfa_serial = arn:aws:iam::111122223333:mfa/my-iam-user

But when you use this role for the first time. Or when the cached credentials are expired it will prompt for MFA token. When you use many accounts and profiles this becomes annoying. You could configure and use a longer session time on your roles. But you still need to enter your MFA token per role. In this blog post I will explain why this is and what you can do to avoid it!

How does it work?

So what happens under the hood? When you perform an cli command like:

aws s3 ls --profile=my-role

It will first check in the ~/.aws/cli/cache/ folder if there are credentials cached. If not or if the credentials are expired an assume-role call is executed. Because we configured the default source profile. It will use the default credentials found in the ~/.aws/credentials file.

[default]
aws_access_key_id = AKIAXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Because we configured an mfa_serial under our profile in our ~/.aws/config file. The AWS cli will prompt for the token and once given it will do the assume-role API call. If everything is fine temporary credentials are returned and these are cached. The problem with this approach is that it will be executed each time you need to assume a role. Therefor triggering an MFA prompt each time you use a role. For the first time or if the cached credentials are expired.

How can I avoid it?

You can avoid the MFA prompts by first fetching the temporary credentials of the IAM User using MFA. This will give you credentials that are MFA authenticated. And because they are already MFA authenticated. There is no need to provide the token during the assume-role API call. As you can see we split the process into 2 parts:

Fetch the temporary credentials for the IAM User using MFA.
Use the temporary credentials to assume the my-role role. The magic behind this approach is because we have split them. You are able to repeat step 2 with other roles. This removes the MFA token prompt because we are already authenticated using MFA. To make this easier I wrote a small cli tool called aws-iam-login that helps you with this! It only needs to know the mfa_serial and username. You can retrieve these if you execute:

aws-iam-login default init

This command will fetch the ARN of the caller identity. Based on this identity we will determine the username and mfa_serial of the IAM User. These will then be stored in the ~/.aws/credentials file. For example:

[default]
aws_access_key_id = AKIAXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
mfa_serial = arn:aws:iam::111122223333:mfa/my-iam-user
username = my-iam-user

Now you can use the tool as followed:

aws-iam-login default

This authenticates against the AWS API. And request temporary credentials from AWS using your MFA Token. These credentials are then stored as <profile-name>-sts. For example:

[default-sts]
aws_access_key_id = ASIAXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
aws_session_token = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
expiration = 2022-07-25 19:00:00+00:00

So if you now use the following in your ~/.aws/config file:

[profile my-role]
source_profile = default-sts
role_arn = arn:aws:iam::111122223333:role/Administrator

You will now use the temporary credentials defined in the ~/.aws/credentials file. When the temporary credentials become invalid. Re-run the aws-iam-login default command and new credentials will be fetched.

Rotating your credentials

Next to helping you assume roles aws-iam-login can help you with more! It can rotate your credentials for you! You can do this with the following command:

aws-iam-login default rotate

And the tool will:

List all available keys for the user, when 1 key is active rotation is possible!
Create a new AccessKey and SecretAccessKey.
Use the created keys to deactivate the old keys.
Write the new keys to the ~/.aws/configuration file.
Delete the old keys.

Conclusion

By using the temporary credentials of the IAM user. You are not prompted to enter your MFA token every time. And aws-iam-login can help you with that!

Working with AWS Permission Policies

Wed, 13 Jul 2022 15:01:51 +0000

Policies and Permission Controls

Access management is a crucial part of managing your workloads. It is also one of the primary areas where debugging can be problematic. In AWS, there are multiple ways to control access to the platform at various levels. These permission policies make it possible to optimize control of applications, programs, and resources, so day-to-day operations run as smoothly as possible. AWS has many options to create secure policies on its platform. Deciding which policies and permission controls to use for different organizations can seem daunting. Organizations rely on AWS Identity and Access Management (IAM) for access management. But there are more options, particularly for larger, more complex organizations with many accounts. Here, we tackle the best use case scenarios for AWS access management according to the needs of different users and organizations, small and large. We also address new developments and guidance in debugging permission errors.

Identity and Access Management (IAM)

AWS identity access management is a web service that securely enables user or application access to AWS resources and services. It allows users to control and define what users or applications are allowed to do within an account by authenticating and authorizing the user. It also limits what users can do. Within IAM, you have users and roles; unlike a role, a user can be organized into groups. Within IAM, these entities are referred to as principals. Principals can only perform actions when granted permission. IAM roles are not unique to a single individual, they are like a hat that any users or resources can wear if they have permission to do so, and it can give you a temporary “superpower”. Roles permit actions executed by AWS services, and users can assume a role. For example, an AWS service can request temporary access by assuming a role. The most commonly used services are Amazon Simple Storage Service (Amazon S3) buckets, Amazon Elastic Compute Cloud (Amazon EC2), AWS Lambda, and Amazon DynamoDB. Roles are used either within the web console or through a program such as AWS CLI. IAM policies are attached to users, groups, roles and resources. This makes managing permissions in your workload more manageable, particularly for smaller organizations with a small number of accounts. In the case of broader permissions, with multiple accounts. In these cases, you typically have a platform team managing the account vending process. And you probably have some standardized resources that you deploy. The workload teams should not alter these resources because they are created by the workload team. You can protect these resources on an organizational level using Service Control Policies (SCPs).

AWS Permissions

Policies are created based on a user’s request and authorization. They should follow the principle of least privilege. A policy defines permissions and determines access or denial when an IAM principal (user or role) requests it. A policy is nothing more than a JSON definition. You can write them from scratch, copy from existing ones, or use the visual editor, you define the policy, but AWS does the enforcement. AWS also offers managed policies that are managed and maintained by AWS’s team. You can use them directly and let AWS maintain them or, you can copy and modify them according to your needs. Each user and role can have policies attached to it. Users can inherit policies based upon the groups they are a member of. At the time of writing, there are a maximum of 10 policies that you can attach to a user or role. This is a soft limit and you can request an increase with a maximum of 20. There are six types of policies AWS supports which are mainly composed in the JSON policy document format:

Organization SCP: Service Control Policy is used to define the maximum permissions in an organization. SCPs are applied to organization units, and policies are inherited from the parent OU. SCPs are not for allowing actions, they are only used to deny actions. For example, you can use SCPs to prevent the development team from deploying expensive resources such as EC2 “u-12tb1.112xlarge” which costs $3,648 per day. Or you can use SCPs to limit which regions can be used in your organization by denying the rest of the regions.
Identity-Based Policies: Permissions granted by IAM identities (users, groups, or roles). This controls what actions an identity can take. In policy, you can deny or allow actions related to each service involving all or specific resources with certain conditions. For example, you can have a policy to allow listing objects of S3 buckets whose name starts with app1-acc-* with the condition that the user used MFA during authentication.
Resource-based Policies: You can deny or allow specific actions to users or groups by attaching inline policies to resources. Resource-based policies are available for several services; you can find the list here.
Permission Boundaries: The maximum permissions an identity-based policy can grant to an entity. Permission boundaries are the perfect solution to delegate more tasks to developers to remove overhead from the platform team without compromising security.
Session Policies: Limits the permission the user or role can grant to a session when you are programmatically assuming a role. By passing a session policy, you can ensure this specific session has only the least privileges needed and not the entire role permission.

AWS Organizations

When your organization grows you will face multiple challenges. One of them is that you onboard more people to your organization. You can use your existing SSO (Single Sign On) solution or use AWS Single Sign On to authenticate these users. Allowing them to assume a role in one of the AWS Accounts. Another challenge would be that you separate your workloads across multiple AWS Accounts. This would lead to more and more AWS Accounts while you grow. A common practice is also to split environments across multiple accounts. So for a single workload you could easily have development, testing, acceptance and production accounts. These challenges can be addressed using AWS Organizations. It allows you to centrally manage the accounts itself and access to them. It would also allow you to manage SCP (Service Control Policies).

Managing Permissions in AWS Organizations: Service Control Policies

Service Control Policies are a policy type found in AWS. SCPs work well in large organizations because they allow the organization to split the responsibilities. With an organizational SCP you can deny complete services, or only a handful of actions. These SCPs have the same policy structure as the IAM policies with one difference. They apply to the whole AWS Account. For example, when you have a certain blueprint of how an AWS Account needs to look like when you hand it over to a workload team. You can protect these resources from being altered or deleted by the users and roles used by the workload team. An SCP is placed on an Organization Unit (OU) and is applied to all accounts in the OU including any nested OUs. One of its primary functions is to ensure that all accounts stay within an organization’s access control guidelines. This means that while SCPs do not grant permissions, they do, however, define the limits and actions of IAM users and roles. SCPs effectively act as the guardrail of all accounts an administrator controls. An SCP can override an IAM policy. In effect, an SCP helps organizations resolve problems that may arise in an evolving cloud infrastructure. Because organizations tend to have many moving parts, with different teams, departments, hierarchies, etc., SCPs help define the boundaries of what is allowed within the account. In the account itself you still need to define the access rights for each principal. If the principal is granted access that is denied by the SCP. The principal is not allowed to perform that action.

Working with an SCP Example

Assume we have a platform team and a workload team. The workload team is the only team allowed to create S3 buckets that start with xebia-workload. The platform team is the only one that can create buckets that start with xebia-platform. You can achieve this with the following SCP:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPlatformPrefixBuckets",
      "Effect": "Deny",
      "Action": [
        "s3:CreateBucket"
      ],
      "Resource": [
        "arn:aws:s3:::xebia-platform-*"
      ],
      "Condition": {
        "ForAnyValue:StringNotLike": {
          "aws:PrincipalARN": [
            "arn:aws:iam::*:role/Administrator*"
          ]
        }
      }
    },
    {
      "Sid": "AllowWorkloadPrefixBuckets",
      "Effect": "Deny",
      "Action": [
        "s3:CreateBucket"
      ],
      "Resource": [
        "arn:aws:s3:::xebia-workload-*"
      ],
      "Condition": {
        "ForAnyValue:StringNotLike": {
          "aws:PrincipalARN": [
            "arn:aws:iam::*:role/Developer*"
          ]
        }
      }
    }
  ]
}

When you try to create a xebia-platform-myworkload S3 bucket using the “Developer” role. You will get an error like this:

When you create a xebia-workload-myworkload S3 bucket using the “Developer” role. You will be able to create the bucket. When you have a closer look at the policy you will see that we set a Deny on the s3:CreateBucket action. For buckets that match the name xebia-workload-* when the ARN of the principal does not match the Developer role. You can translate this to only the Developer role is not denied to create a bucket with the xebia-workload- prefix. This is a simple example but it illustrates the power of a SCP. From an organizational point you can prevent developers from creating platform buckets. And you prevent administrators from creating workload buckets. This also creates a problem for the developer role. An action is denied but from the account perspective the developer role should be able to create a xebia-platform-workload bucket. The solution to this is using the IAM Policy Simulator. You can select a role, action and the resource and simulate what would happen.

As you can see the action was denied. But it also states that it is denied by AWS Organizations. Now the developer knows that the action is blocked due to a SCP.

Permission Boundaries: Why do you need to start using them?

Imagine this scenario, you are the platform admin, and you need to create a policy for a developer that works on serverless applications. The developer needs a role for each application created, which means you need to create new roles every time there is a new application. To reduce the overhead, you want to permit the developer to independently create the roles, but you are worried if the developer decides to create a role that can do more than what is needed! You can create a policy that allows the developer to create roles only if they attach permission boundaries to the role. Then you can define the maximum permission in the permission boundaries. For our above scenario, permission boundaries can only allow AWS S3 and AWS Lambda function actions. If the developer decides to create a role with administrator access the policy boundary will limit this to only what is allowed within the boundary.

This is what AWS calls the “Effective permissions”. From the organizational point of view, you manage the permission boundaries. The developers can create their own roles but only when they attach one of the boundaries you made available.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowRolesWithBoundary",
      "Action": [
        "iam:CreateRole",
        "iam:AttachRolePolicy",
        "iam:PutRolePolicy",
        "iam:PutRolePermissionsBoundary"
      ],
      "Effect": "Deny",
      "Condition": {
        "StringNotEquals": {
          "iam:PermissionsBoundary": [
            "arn:aws:iam::111122223333:policy/xebia-default-boundary"
          ]
        }
      },
      "Resource": "*"
    }
  ]
}

As you can see in this example policy, we deny the actions when there is no permission boundary called xebia-default-boundary.

How to Debug Permission Errors

As a developer, you might run into permission errors. These errors might not make sense because from the developers perspective all policies are set correctly. In these cases, you might be running into a SCP that has blocked you from performing the action. For this reason, it’s useful to use the policy simulator. The simulator will allow you to select a role that you created. Followed by the service and action you want to simulate. This will tell you if the action is allowed or not. But more importantly, it will tell you why it is denied.

Conclusion

When you are working in larger organizations it is important to have a good vision of how you are handling your permission management. As you could read in this paper there are different places where you can apply your policies. Write down how you are handling your permission management and look at it before you make future changes to your policies. The best practice is to document functions for each role, define permission according to a function with the least privileges using the “allow” action, and avoid using the “deny” action.

Migrate resources across CDK stacks

Thu, 30 Jun 2022 12:15:00 +0200

When you start building infrastructure as code. You will run into the situation that, you want to split a stack into separate ones. Or, you deployed a resource in the wrong stack. This is all part of the development lifecycle. You try things, some succeed, some will fail. But deployed production resources are hard to rebuild. They either have a complex data migration path or you can’t afford the downtime.

If you accept that some resources or located in unexpected stacks. Your complexity of your infrastructure will increase. And complexity will lead to incidents in the future.

In this blog post I will explain how you can move a deployed resource from one CDK stack to another.

Introduction

First of all, it is important to realize that CDK generates CloudFormation. CloudFormation offers functionality to bring existing resources under it’s control. This means you can import an already deployed resource into a CloudFormation stack. Now how does that relate to CDK?

Lets assume you created a CDK stack that created a DynamoDB table. This table contains information or is in use. In these scenarios you cannot delete the resource and recreate it. That would lead to data loss or downtime. So in this scenario we could migrate the resource from one stack to another.

Prerequisites

You need to have the following command line tools in place.

AWS CLI
jq

Make sure you have valid credentials in the correct account.

Step 1 – Remove the resource from the existing stack

Download the existing CloudFormation template:

aws cloudformation get-template \
    --output json \
    --stack-name <my-cloudformation-stack-name> | jq -r '.TemplateBody' > original-template.yml
cp original-template.yml modified-template.yml

In the template modified-template.yml you need to locate the resource you want to move to your “new” stack. Once you found it you need to add the following DeletionPolicy: Retain to the resource.

Resources:
  MyTable794EDED1:
    Type: AWS::DynamoDB::Table
    Properties:
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      ProvisionedThroughput:
        ReadCapacityUnits: 5
        WriteCapacityUnits: 5
      TableName: Games
    UpdateReplacePolicy: Retain
    DeletionPolicy: Retain

This will preserve the resource when you delete it from the stack. Now we will update the CloudFormation stack:

aws cloudformation update-stack \
    --stack-name <my-cloudformation-stack-name> \
    --template-body file://modified-template.yml
cp modified-template.yml final-template.yml

The next step depends a bit on your situation. In some cases you can remove the complete stack because, there are no other resources in the stack. Or because they are not needed anymore. But in other cases you need to move out a resource. (A DynamoDB Table in this example)

When you don’t need the other resources anymore you can delete the complete stack. If you want to preserve the rest of the resources. You will need to remove the resource from the template that you want to move. So we remove MyTable794EDED1 from the final-template.yml file.

By re-running the update-stack command again. We will instruct CloudFormation to remove the resource. But, because we have the Retain policy set. The resource will only be removed from the stack.

aws cloudformation update-stack \
    --stack-name <my-cloudformation-stack-name> \
    --template-body file://final-template.yml

Step 2 – Import the resource

When you move resources around you either move them to a new or an existing stack. When you deal with an existing stack you need the existing template first. If you are updating an existing stack, continue with “Download the existing stack”. Otherwise you can skip that section.

Download the existing stack

Download the existing CloudFormation template:

aws cloudformation get-template \
    --output json \
    --stack-name <my-other-cloudformation-stack-name> | jq -r '.TemplateBody' > template.yml

Import the existing resource

When you want to move the resource to a new stack. You need to create a new template.yml file. If you want to move the resource in an existing stack you already have the existing template ready.

To import the resource we need the resource definition. You can copy that from the modified-template.yml file from step 1. I could recommend this because the CloudFormation definition should match the deployed resource. The second thing is it needs to have the DeletionPolicy: Retain definition. So our example will look like this:

Resources:
  MyTable794EDED1:
    Type: AWS::DynamoDB::Table
    Properties:
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      ProvisionedThroughput:
        ReadCapacityUnits: 5
        WriteCapacityUnits: 5
      TableName: Games
    UpdateReplacePolicy: Retain
    DeletionPolicy: Retain

Next we need to create a file, lets call it import.json. In this file we will define the following for each resource what we import:

AWS Resource type, the type of the resource itself.
LogicalResourceId, the identifier in the template. In this example that would be Games.
ResourceIdentifier, the identifier of the existing resource.

As you can see this file contains the relation between the stack and what is already deployed. The file looks as followed:

[
    {
        "ResourceType":"AWS::DynamoDB::Table",
        "LogicalResourceId":"MyTable794EDED1",
        "ResourceIdentifier": {
            "TableName":"Games"
        }
    }
]

Next, you will need deploy the stack. You can do this with the following commands:

aws cloudformation create-change-set \
    --stack-name <my-other-cloudformation-stack-name> \
    --change-set-name ImportChangeSet \
    --change-set-type IMPORT \
    --template-body file://template.yml \
    --resources-to-import file://import.json
aws cloudformation wait change-set-create-complete --change-set-name ImportChangeSet --stack-name <my-other-cloudformation-stack-name>
aws cloudformation execute-change-set --change-set-name ImportChangeSet --stack-name <my-other-cloudformation-stack-name>

Step 3 – Redeploy using CDK

At this point the DynamoDB table is part of the new stack. But when you redeploy the CDK stack it will not render the DynamoDB resource definition. So for this we will need to add the definition to the CDK stack. For this example I used the following CDK snippet. But when you move from one CDK stack to another you can just move the relevant code.

dynamodb.Table(self, "MyTable",
    table_name="GamesTable",
    partition_key=dynamodb.Attribute(name="id", type=dynamodb.AttributeType.STRING),
    billing_mode=dynamodb.BillingMode.PROVISIONED
)

Now in theory this should render the same template you used during the import. But if there are changes CloudFormation will thread them as updates. It tries to change the resource to its new state. After you deployed using CDK its business as usual.

Conclusion

By reducing complexity in your infrastructure. You reduce the chance on failures and incidents. This reduction can be achieved by moving resources to more logical stacks. The process itself does contain manual steps. But when your resources contain data or cannot afford downtime. This might be your only solution.

Diagrams as code

Fri, 10 Jun 2022 08:45:00 +0000

Most of you have heard of “infrastructure as code” by now. But building cloud infrastructures is more than coding infrastructure. In this blog I will show you how I make my own live easier by generating infrastructure diagrams from code. When you have read my previous blogs about Commit messages done the right way. You are not surprised that I like to make things easy for the reader. When you are new to a team, or you are not familiar with a specific repository that your team maintains. It helps when the README.md of the repository helps you understand what the repository does.

A picture is worth a thousand words!

Nice diagrams will help you a lot! But often they are either created in a program that you don’t master. Or you don’t have a access or a license to it. This makes it harder to keep the images up to date. And easier not to update them at all. By managing the diagrams the same way as we manage our infrastructure. We also get the same benefits! Not only that when you make a change to your infrastructure you can update the image in the same commit. This makes it easier for the reviewer. Because, next to the code change he also will see a visual presentation of the change.

How does it work?

Enough about the theory, how do you do that? Within Python there is a package named diagrams. The website has a lot of examples I copied a few to this article. First we will configure a poetry environment. Afterwards we will install the dependency.

mkdir diagrams-as-code
cd diagrams-as-code
poetry init
poetry add diagrams

Once you have done that, we can create a file called grouped_workers.py:

from diagrams import Diagram
from diagrams.aws.compute import EC2
from diagrams.aws.database import RDS
from diagrams.aws.network import ELB

with Diagram("Grouped Workers", show=False, direction="TB", filename="images/grouped_workers"):
    ELB("lb") >> [EC2("worker1"),
                  EC2("worker2"),
                  EC2("worker3"),
                  EC2("worker4"),
                  EC2("worker5")] >> RDS("events")

As you can see the syntax is pretty straight forward. We have 5 workers that are behind a load balancer. The workers can connect to an RDS database named events. When you then run the following command:

poetry run python grouped_workers.py

The output looks like this: Pretty cool right? Let’s do another one, we will create a file called clustered_web_services.py:

from diagrams import Cluster, Diagram
from diagrams.aws.compute import ECS
from diagrams.aws.database import ElastiCache, RDS
from diagrams.aws.network import ELB
from diagrams.aws.network import Route53

with Diagram("Clustered Web Services", show=False, filename="images/clustered_web_services"):
    dns = Route53("dns")
    lb = ELB("lb")

    with Cluster("Services"):
        svc_group = [ECS("web1"),
                     ECS("web2"),
                     ECS("web3")]

    with Cluster("DB Cluster"):
        db_primary = RDS("userdb")
        db_primary - [RDS("userdb ro")]

    memcached = ElastiCache("memcached")

    dns >> lb >> svc_group
    svc_group >> db_primary
    svc_group >> memcached

So this example is a bit more complex. We now introduced Route53, and we added a caching mechanism. To clarify the grouping we made use of the Cluster option. And when we run the following command:

poetry run python clustered_web_services.py

We will get the following output:

Conclusion

You will receive the same benefits from “infrastructure as code”. When you manage your “diagrams as code”. Version control comes out of the box. Because you combine the infrastructure code change. With the diagram code change you make the life of you and your teammates a lot easier. Photo by Luca Bravo on Unsplash.

Sync AWS CodeCommit repositories

Tue, 12 Apr 2022 11:13:35 +0200

In some scenarios you might have the need to replicate an AWS CodeCommit repository. When I was setting up a test organization using AWS Deployment Framework (ADF) I ran into this myself. Because I want to test the deployment of my landing zone I needed to have a close replica. This includes the CodeCommit setup.
But at the same time I did not want to change the development workflow. The workflow is pretty straight forward. You create a feature branch to work in. When you are ready you merge it to a development branch. When it needs to go to production you merge it into the main branch.
So we will use the development branch to deploy to the test organization. But, because the test organization is a replica of production. Merging to the development branch would not have effect on the test organization. For this we need to synchronize the development branch to the test organization.

How does it work

We will use an AWS Lambda function with a git client to perform the following actions:

Clone the CodeCommit repository.
Checkout the development branch.
Assume the an IAM role in the target account.
Perform a git push.
The name of the CodeCommit repositories in both organization are identical. By assuming a role in the AWS Account that hosts the CodeCommit repositories in the test organization. You can perform a git push assuming you have the correct permissions. The changes are then pushed to the repository in the test organization.
There is a difference between the test and production organization. The test organization is configured to listen to the development branch. The production organization listens to the main branch.
With this in place you have the following workflow:
Create a pull request for your feature branch into development.
When reviewed and approved you can merge the pull request.
(Automatic) an event is triggered, we use an event rule to trigger the Lambda function.
(Automatic) the Lambda function will then:
1. Clone and pull the code from the production repository.
2. Assume a role in the test organization.
3. Push the code to the remote repository in the test organization.
(Automatic) the CodePipeline for the repository is triggered.
Now you only need to confirm whether your changes are correct in the test organization. Once satisfied you can merge the change to the main branch. This will deploy the same changes in production.

Conclusion

Synchronizing of git repositories can help you with automated testing. See the aws-lambda-git repository for more information.

Running a git binary in AWS Lambda

Thu, 24 Mar 2022 20:28:51 +0100

Running a git binary in AWS Lambda

In some scenarios it can be easier to use a binary in a lambda function. In this blog post I will walk you through how you can run a git binary in an AWS Lambda.

For this blog post I will use the AWS Serverless Application Model(AWS SAM). AWS SAM makes it easy to build, package and deploy lambda functions. The used template can be found on GitHub. AWS SAM will look at the Runtime of the AWS::Serverless::Function resource. In this example I used python so it will look at the requirements.txt. It will look at the packages defined in the file. AWS SAM installs them in .aws-sam/build/<ResourceName>. This folder is compressed and uploaded to Amazon S3 when you package and deploy the template.

You can also invoke the function on your own computer. This makes it easier to develop and troubleshoot. To invoke the function we first need to build it. Afterwards you need to invoke it. I created a Makefile, for more information on this you can read one of my previous blogs.

make invoke

You will see that you will see the following error:

{
    "errorMessage": "[Errno 2] No such file or directory: 'git'",
    "errorType": "FileNotFoundError"
}

And that makes sense right? We do not have the git client installed. You cannot install it using the requirements.txt file. This is because it is not in the PyPi package manager.

Install the git client

If you look at the Makefile you can see that it will:

Build a docker container.
- Install git on the container.
- Move all the dependencies into the /var/task structure.
- Zip the content of the /var/task folder.
Extract the zip file from the container.
Unzip the contents into the git_client folder.
sam build will copy the content to .aws-sam/build/GitClient.
sam build installs the dependencies from the requirements.txt.
sam invoke will use the .aws-sam/build/GitClient content to invoke the function.

The Git binary is located in /var/task/bin. Because when we invoke the git command we also supply the /var/task/bin as PATH. This allows us to call git and the container knows it needs to invoke /var/task/bin/git. The libraries are located in /var/task/lib. Because we have set the LD_LIBRARY_PATH to /var/task/lib the libraries can be found.

I created a Dockerfile that uses public.ecr.aws/lambda/python:3.8 as the base. This is the same runtime of the Lambda function. I prepare 2 folders /var/task/bin and /var/task/lib. Then I install the git and zip packages within the docker context.

I then copy the /usr/bin/git and the /usr/libexec/git-core/git-remote-http executables to the /var/task/bin folder. When you try to build the Lambda function and run it it fails on a missing library.

{
    "errorMessage": "git: error while loading shared libraries: libpcre2-8.so.0: cannot open shared object file: No such file or directory",
    "errorType": "Exception"
}

By using the ldd command you can find all the shared libraries. We will use this to identify the needed libraries and copy them into the /var/task/lib/ folder. Once you have all dependencies in place. You can fetch the git --version running inside of the Lambda function.

Git version: git version 2.32.0

If you combine this with the awscli and git-remote-codecommit packages. You can clone a CodeCommit repository and perform your actions and commit it again.

Conclusion

It is possible to embed binaries in AWS Lambda and use them. It might take you some effort. But it opens new possibilities that you might not considered before. In my next blog you will read how you can use this to synchronize two CodeCommit repositories.

Photo by Emile Perron on Unsplash

Hosting a single page application or website on S3

Wed, 09 Mar 2022 16:13:58 +0100

In this blog post I will walk you through how to configure a single page application or website. In a previous blog Best practices for S3 web hosting and explaining why. Tibor Hercz explained the best practices for hosting a static website on S3. As Tibor explained, there are some downsides on using a Origin Access Identity (OAI). But in some Single Page Applications (SPA) you will not experience this downside.

What is a single page application?

A single page application loads a single web document. Based upon your interaction with this page the content is updated. The pages are rendered on your web browser. And more API calls are performed to fetch extra input. Due to this fact an SPA can be fast! You can use the framework of your choice: angular, reactjs, vue, etc, etc.

Setting up the infrastructure

For simplicity, I am providing CloudFormation snippets.

First we will need an S3 bucket to host the application in. We need an OAI. And we need to grant it access yo the bucket:

  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private

  MyOriginAccessIdentity:
    Type: AWS::CloudFront::CloudFrontOriginAccessIdentity
    Properties:
      CloudFrontOriginAccessIdentityConfig:
        Comment: Allows CloudFront to reach the bucket

  MyBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref MyBucket
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Action:
              - s3:GetObject*
              - s3:GetBucket*
              - s3:List*
            Effect: Allow
            Principal:
              CanonicalUser: !GetAtt MyOriginAccessIdentity.S3CanonicalUserId
            Resource:
              - !GetAtt MyBucket.Arn
              - !Sub ${MyBucket.Arn}/*

Next we will need the CloudFront distribution:

  MyDistribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        CustomErrorResponses:
          - ErrorCode: 404
            ResponseCode: 200
            ResponsePagePath: /index.html
        DefaultCacheBehavior:
          AllowedMethods:
            - GET
            - HEAD
          CachedMethods:
            - GET
            - HEAD
          Compress: true
          ForwardedValues:
            Cookies:
              Forward: none
            QueryString: false
          TargetOriginId: origin
          ViewerProtocolPolicy: redirect-to-https
        DefaultRootObject: index.html
        Enabled: true
        HttpVersion: http2
        IPV6Enabled: true
        Origins:
          - ConnectionAttempts: 3
            ConnectionTimeout: 10
            DomainName: !GetAtt MyBucket.RegionalDomainName
            Id: origin
            S3OriginConfig:
              OriginAccessIdentity:
                Fn::Join:
                  - ""
                  - - origin-access-identity/cloudfront/
                    - Ref: MyOriginAccessIdentity

Once you have set up the infrastructure. You need to build your SPA. And upload the build to the S3 bucket. My assumption here is that you created an app with various routes/paths.

How does it work?

When you navigate to the hostname of the created CloudFront distribution. The distribution will fall back on the index.html file. This is because we have set the DefaultRootObject to index.html. When you navigate in the application it looks like the browser is changing paths. But in fact it is only updating the address in the address bar. Let say you created a /blog route. You navigate to this route. And you do a hard refresh! You expect to get a 404 Not Found error because there is no folder called blog. But because we configured a CustomErrorResponses. That will redirect all 404 errors to the index.html file with a 200 response code. This will make sure that the Single Page Application gets loaded again. It will see the location in the address bar and renders the correct content. This does mean that you will need to handle unknown routes in your application.

Conclusion

When you want to host a Single Page Application you can use CloudFront with private buckets. You will get all the benefits of CloudFront while keeping the bucket private.

Improve your pull request workflow using pull-request-codecommit

Wed, 23 Feb 2022 21:21:45 +0100

In my previous blog post, I wrote why it is important to write good commit messages. In this post I will highlight an example how this works in your advantage. Using an AWS CodeCommit repository.

Working example

Alright, lets assume we have a project that uses a main branch. In combination with feature branches. We want to add a feature to our project, so we create a branch called: feat/my-feature. Let say we manage to get the feature working in a single commit. The commit message would look something like this:

feat: add support for x

By adding support for x we allow x users to adopt this tool in their workflow.

Issue: #123

Now it is time to create a pull request! I have created a tool called pull-request-codecommit. This tool looks at the differences between the current branch. And the target branch and proposes a pull request message. So if you want to create a pull request you type:

git pr

The default editor is opened. And in this editor you will see:

feat: add support for x (#123)

By adding support for x we allow x users to adopt this tool in their workflow.

Issue: #123

The first line will be used as the title and the rest as a description. You can alter the message to your liking, but if you use a proper commit message this is spot on already. When you save the proposal the pull request is created for you:

For convenience the link is displayed and the eventual message. When you open the link the pull request will look like this:

What about many commits?

When many commits are used the tool tries to propose a good pull request message. When you split your commits. And you used proper messages it looks something like this:

As you can see, the first commit is used as the title. The rest of the commit titles are shown as a list in the description.

Conclusion

When you write proper commit messages, you help your self. It makes it easier for you to draft pull requests. And it allows you to start using tools because you use standards.

Commit messages done the right way

Thu, 10 Feb 2022 15:08:49 +0100

We have all seen it. You see something in your source code. You do not know why it was added. And you look at the commit history. And then you see: “fix bug” or “add feature name”. How helpful is that message? In this blog post I will address how you can write commit messages that are helpful. And optimized for the reader.

So, why is this important?

Most agile teams need peer reviews. The most common or well known way is via pull requests. When you introduce a change. For a feature or fixing a bug. The question your peer reviewer will have is: Why this change was made. And then continues to look at the changes. You create a pull request on one or more commit messages. A commit describes the why of a change. When you create a pull request you can use these commit messages when you create your pull request.

Example commit

fix input

Or:

fix: white spaces on input

Remove the leading and trailing spaces. This will clean the input before we save it to the database.

Issue: #123

This is the difference between writing a commit message. And writing a useful commit message. Useful commit messages explain why the change was made. When you create a pull request you can use the commit messages as input. And draft a useful pull request description for your reviewer.

Commit format

I am a big fan of the conventional commits format. But the most important thing is that you write down why the change was made. Not what the change is. The changes are already displayed when you do a diff between commits/branches. So describing what has changed does not make sense. The reason why that is in the authors head. And unless you add it in a commit message that information gets lost. By doing this you create a better historical overview. And it makes it easier to create pull requests.

Conclusion

When you optimize your commit messages for the reader. The reader does not need to go on a scavenger hunt. And it creates a better historical overview of why changes are made in your repository.

How to set up AWS Session Manager logging cross account

Wed, 26 Jan 2022 22:47:30 +0100

Setting up and managing access to your EC2 instances can be challenging. There are a lot of things you need to consider. How do you rotate your keys? Who logged in using that key? How do you limit the risk of thread actors? And what did they do? What if you could log the complete session in a different AWS Account? And encrypt it using a customer managed KMS key? That would give you complete visibility and auditability on your shell sessions. In this blog post I will address those questions using AWS Session Manager.

What is the AWS Session Manager?

Taken from the documentation:

Session Manager is a fully managed AWS Systems Manager capability. With Session Manager, you can manage your Amazon Elastic Compute Cloud (Amazon EC2) instances, edge devices, and on-premises servers and virtual machines (VMs).

So, what does that mean? It means that you can open an SSH shell using the AWS Console in your browser. Select the EC2 instance. Click the connect button and a shell session loads in the browser. You can also use CLI commands to do this. The interesting part is that you grant access based on your IAM credentials. So you do not have to share private SSH keys across the team. No need to rotate them. No need to rotate them when a team member leaves. Because you don’t need a private key at all. But this solution comes with another advantage. You do not need to open port 22 to connect. In fact, you do not need to open any inbound ports.

The SSM Agent comes pre-installed with the Amazon Linux 2 AMI.

Instead, it uses an SSM Agent. And this agent will register itself to the AWS Systems Manager. It will use the attached instance profile to do that. From that point on when you request a shell from the Session Manager. The agent will react on this response and establish the connection.

What are the other advantages?

Yes, there is more! Session Manager has some extra features! You can track session activity using Amazon EventBridge. This means that you can react on these events. This creates the possibility to proactively react on these events. You could for instance: terminate the session if no access is allowed. And that brings me to the main topic of this blog post. What if you suspect that some commands where executed on a server using a shell session? How do you prove it happened or did not happen?

Logging your session activity

The Session Manager gives you the ability to log the session to S3. Setting this up is described on the documentation pages. But, if you want to use a centralized logging account. And you need to use a customer managed KMS key, the documentation is not that helpful.

Depending on your use-case, you might want to limit the access. You can do this per account. Or within an AWS Organization. For this example, I will limit the scope within the same organization.

First, you will need to create a KMS key. This key will be used to encrypt the data stored on S3. S3 needs to be able to use the KMS key add following statements to the key policy:

{
    "Effect": "Allow",
    "Principal": { "AWS": "*" },
    "Action": [ "kms:Decrypt", "kms:Encrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*" ],
    "Resource": "*",
    "Condition": {
        "StringEquals": {
            "kms:ViaService": "s3.eu-central-1.amazonaws.com",
            "aws:PrincipalOrgID": "o-xxxxxxx"
        }
    }
}

Next, we allow kms:Decrypt and kms:GenerateDataKey within the organization. These actions are needed so that the Session Manager can encrypt the session. And it can confirm that it can log to the given S3 bucket.

{
    "Effect": "Allow",
    "Principal": {
        "AWS": "*"
    },
    "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
    ],
    "Resource": "*",
    "Condition": {
        "StringEquals": {
            "aws:PrincipalOrgID": "o-xxxxxxx"
        }
    }
}

We then need to create a S3 Bucket. You need to enable default encryption using the KMS key you created. You need to grant other accounts to write to this bucket. So, you need a bucket policy. Here is an example of a bucket policy that you need:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetEncryptionConfiguration",
            "Resource": "arn:aws:s3:::my-bucket-name",
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "o-xxxxxxx"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::my-bucket-name/*",
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "o-xxxxxxx"
                }
            }
        }
    ]
}

Alright, the logging account has been prepared. It can now receive logs from all accounts within your organization. Now it is time to set up the account that will use the Session Manager. Create a JSON file locally with the following content:

Replace the name of the bucket and the KMS key! You could use the account id as a prefix.

{
    "schemaVersion": "1.0",
    "description": "Document to hold regional settings for Session Manager",
    "sessionType": "Standard_Stream",
    "inputs": {
        "s3BucketName": "[NAME OF THE BUCKET]",
        "s3KeyPrefix": "",
        "s3EncryptionEnabled": true,
        "cloudWatchLogGroupName": "",
        "cloudWatchEncryptionEnabled": false,
        "cloudWatchStreamingEnabled": false,
        "kmsKeyId": "[FULL ARN OF THE KMS KEY]",
        "runAsEnabled": true,
        "runAsDefaultUser": "",
        "idleSessionTimeout": "20",
        "maxSessionDuration": "",
        "shellProfile": {"windows": "", "linux": ""}
    }
}

Apply this SSM Document to the account using the Session Manager:

aws ssm update-document \
    --name "SSM-SessionManagerRunShell" \
    --content "file://SessionManagerRunShell.json" \
    --document-version "\$LATEST"

At this point everything is inplace to make this work. Except for the IAM permissions on the instance profile. So to get this to work you will need the following permissions:

Attach the arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore managed policy.
Allow the “s3:GetEncryptionConfiguration” action on the arn:aws:s3:::[NAME OF THE BUCKET] resource.
Allow the “s3:PutObject” and “s3:PutObjectAcl” action on the arn:aws:s3:::[NAME OF THE BUCKET]/* resource.
Allow the “kms:Decrypt” and “kms:GenerateDataKey” action on the [FULL ARN OF THE KMS KEY] resource.

With these permissions in place you are able to start an SSH shell.

Conclusion

When you use Session Manager you not only remove the need to expose port 22. Or don’t need to rotate keys when a team-mate leaves. You get a lot more! Your SSH session is encrypted using a customer encrypted KMS key. That you manage and maintain! Next to that, once the session is closed the complete session is stored on S3. And again encrypted using a customer encrypted KMS key. That you manage and maintain! This allows you to archive shell sessions for later use. Giving you full auditability and improving your security posture. And because you don’t need to bother your team-mates with new keys for this week. Easier for your team to use as well.

Testing your Landing Zone when using AWS Deployment Framework

Wed, 19 Jan 2022 21:06:15 +0100

If you are thinking of setting up a landing zone using ADF in AWS you will need to read this. And why? Because I will tell you why you need to set up two landing zones. And how you can use them to test your landing zones.

What is a landing zone?

A landing zone is a solution that help you to set up a secure, multi-account AWS environment. There are a couple of ways how you can do this, for example:

AWS Landing Zone (predecessor of AWS Control Tower)
AWS Control Tower
AWS Deployment Framework
Build your own…

In this post I am focussing on ADF. The benefit of ADF over AWS Control Tower is that you have more control over the framework. All resources run in your accounts and are under your control. AWS Control Tower is a managed service by AWS. Giving less flexibility but remove maintenance burden.

What is ADF?

From the ADF documentation:

The AWS Deployment Framework (ADF) is an extensive and flexible framework to manage and deploy resources across multiple AWS accounts and regions within an AWS Organization.

So a simple example of this would be, if you need to provide a new DTAP environment for a new workload. You will add some details to a YAML file and commit/push/review/merge it. Then you ADF does its magic. It will create the AWS Accounts on your behalf and bootstrap them. And you end up with a couple of AWS Accounts that have:

A pre-provisioned network setup.
Security controls
etc etc Please note, you need to build what it bootstraps into these accounts your self. ADF only makes it easy to manage what templates are deployed where.

So what is there to test?

Within AWS Organizations you can apply Service control policies (SCPs). All AWS Accounts under the OU (Organization Unit) with the SCP will be subjective to this SCP. What if you need to make a change in this SCP? How can you test this change? SCPs are not the only things you might want to test. Remember that I mentioned that ADF is also bootstrapping the accounts? That could be a VPC with subnets for networking. How do you ensure that the change that you made works as intended? When merging to your main branch. It will trigger a rollout process depending on your configuration.

The solution

You can solve this by rolling out a “test” organization. It will mirror your production organization. But you can scope it down to a smaller set. As long as you cover the cases that are important to your organization. The idea is that when you make a change you first deploy it on the “test” organization. Once deployed you do some tests and roll it out in production. Setting up a “test” organization when you already have a full-fledged landing zone. That is harder than doing it from the beginning. It also helps with the mindset of the team maintaining of the landing zone that there is a “test” environment. And depending on your own use-case you can make a lot of different choices how to set this up. For example, moving the dev and test environments to the “test” organization.

Conclusion

Start thinking about how you can test your landing zone. Setting up 2 landing zones will help you with that. Use a test driven approach when building your landing zone. It will be much harder to add testing afterwards.

This is how you can test your cfn-guard rules

Wed, 29 Dec 2021 21:20:51 +0100

In my previous blog, How do you prove that your infrastructure is compliant. I explained how you can prove your infrastructure is compliant using CloudFormation Guard. But, how do you write those rules? And even more important, how do you test your rules? If you look at the repository CloudFormation Guard. You will notice that the project itself offers a testing framework. Alright! Let’s build a ruleset and write some tests for it!

Let’s build a ruleset for S3

We will start with a sample for S3, create a file called: rules/s3.guard.

let buckets = Resources.*[ Type == 'AWS::S3::Bucket' ]

rule BucketEncryption when %buckets !empty {
  %buckets.Properties {
    BucketEncryption.ServerSideEncryptionConfiguration[*] {
      ServerSideEncryptionByDefault.SSEAlgorithm IN ["AES-256", "aws:kms"] <<Ensure all S3 buckets use encryption-at-rest>>
    }
  }
}

Now we will write the test, called: rules/s3_tests.yaml.

---
- name: "Skip if not present"
  input:
     Resources: {}
  expectations:
    rules:
      BucketEncryption: SKIP
- name: "No encryption used"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
  expectations:
    rules:
      BucketEncryption: FAIL

- name: "Bucket with KMS Encryption"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
         Properties:
           BucketEncryption:
             ServerSideEncryptionConfiguration:
               - ServerSideEncryptionByDefault:
                   SSEAlgorithm: aws:kms
  expectations:
    rules:
      BucketEncryption: PASS

- name: "Bucket with AES-256 Encryption"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
         Properties:
           BucketEncryption:
             ServerSideEncryptionConfiguration:
               - ServerSideEncryptionByDefault:
                   SSEAlgorithm: AES-256
  expectations:
    rules:
      BucketEncryption: PASS

- name: "Bucket with UNKNOWN Encryption"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
         Properties:
           BucketEncryption:
             ServerSideEncryptionConfiguration:
               - ServerSideEncryptionByDefault:
                   SSEAlgorithm: UNKNOWN
  expectations:
    rules:
      BucketEncryption: FAIL

Now it’s time to run the tests:

cfn-guard test --rules-file rules/s3.guard --test-data rules/s3_tests.yaml

This will give the following output:

Test Case #1
Name: "Skip if not present"
  PASS Rules:
    BucketEncryption: Expected = SKIP, Evaluated = SKIP

Test Case #2
Name: "No encryption used"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL

Test Case #3
Name: "Bucket with KMS Encryption"
  PASS Rules:
    BucketEncryption: Expected = PASS, Evaluated = PASS

Test Case #4
Name: "Bucket with AES-256 Encryption"
  PASS Rules:
    BucketEncryption: Expected = PASS, Evaluated = PASS

Test Case #5
Name: "Bucket with UNKNOWN Encryption"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL

You might be thinking, that looks nice! But the output is quite verbose, and you need to execute a command per rule/test set. It gets even worse when a test does fail. For this example I altered the rules/s3.guard file to use aws:kmz instead of aws:kms. Then it looks like this:

Test Case #1
Name: "Skip if not present"
  PASS Rules:
    BucketEncryption: Expected = SKIP, Evaluated = SKIP

Test Case #2
Name: "No encryption used"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL

Test Case #3
Name: "Bucket with KMS Encryption"
  FAILED Rules:
    BucketEncryption: Expected = PASS, Evaluated = FAIL

Test Case #4
Name: "Bucket with AES-256 Encryption"
  PASS Rules:
    BucketEncryption: Expected = PASS, Evaluated = PASS

Test Case #5
Name: "Bucket with UNKNOWN Encryption"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL

If you look at Test Case #3 it says: BucketEncryption: Expected = PASS, Evaluated = FAIL. That makes it hard to spot what test failed.

cfn-guard-test

For this reason I created a python package called cfn-guard-test. This package makes it easier to run many rule/test sets. This is especially useful in the CI/CD pipelines.

Please read the repository for the installation instructions.

Let’s run the tests again!

cfn-guard-test --verbose

The output now looks something like:

===== Analyzing Results =====

Running rules/s3_tests.yaml

Passed 5
Failed 0

That is a nice and clean overview. It lists the number of passed and failed tests. For the next example I altered the

rules/s3.guard file to use aws:kmz instead of aws:kms. This will again trigger a failure:

===== Analyzing Results =====

Running rules/s3_tests.yaml

Passed 4
Failed 1

Rule BucketEncryption failed on #3 "Bucket with KMS Encryption" in rules/s3.guard

Well you have to agree. That immediately sums up how many passed and failed. And more it shows that the BucketEncryption rule fails. And it failed on the test case “Bucket with KMS Encryption”. You don’t want to spend time figuring out what is wrong. This is why we use tooling, to make our life easier. This tool also has the ability to generate JUnit report.

cfn-guard-test --verbose \
  --junit-path "reports/cfn-guard.xml"

Conclusion

cfn-guard-test can help you run many tests. And get an aggregated report of the results. This tool will help you detect bugs in your rules easier and quicker.

How do you prove that your infrastructure is compliant

Tue, 23 Nov 2021 10:30:25 +0100

When you are building your cloud infrastructure. And you have to meet certain compliance standards. You can achieve this with preventive and detective controls. In this blog post I want to focus on the preventive control. How you can stop the deployment pipeline when the infrastructure is non-compliant. And how you can visualize this using CloudWatch Reports. Let’s say that our company has the following rule:

S3 buckets must be KMS encrypted

How can you block a deployment that would deploy a S3 bucket without KMS encryption? The answer is using CloudFormation Guard. With CloudFormation Guard you can write preventive compliance rules. You can then use to check your CloudFormation templates. Create a template.yaml file with the following content:

Resources:
  MyBucket:
    Type: AWS::S3::Bucket

Install CloudFormation Guard using the provided instructions. Then you need to create a s3.guard file with the following content:

# Select all resources of the type AWS::S3::Bucket
let buckets = Resources.*[ Type == 'AWS::S3::Bucket' ]

# Only when there are reources of the type AWS::S3::Bucket
rule S3_bucket_encryption_at_rest when %buckets !empty {
  # Check that each resource has the SSEAlgorithm set to aws:kms
  %buckets.Properties {
    BucketEncryption.ServerSideEncryptionConfiguration[*] {
      ServerSideEncryptionByDefault.SSEAlgorithm == 'aws:kms' <<S3 Bucket must use KMS encryption.>>
    }
  }
}

For more information on the the rules and the format. I recommend reading Introducing AWS CloudFormation Guard 2.0.

Alright, now we can validate the template with the following command:

cfn-guard validate --rules s3.guard --data template.yml

If you now change the template.yml to the following:

Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: aws:kms

And you execute the command again it will now pass.

If you execute this step in your deployment pipeline. It will stop and fail your CodeBuild project. Preventing the non-compliant infrastructure to be deployed. But when you have a deployment that is blocked. You don’t want to look up the CloudWatch logs of the build. And you might have many guard files that you want to check. To make it easier for yourself you can use CodeBuild Reports. With these reports you get a visual overview of what rule passed and what role failed. But by default cfn-guard does not generate compatible reports. And when you execute it, and has failures it will exit with an exit code of 1. I have written a conversion tool to help you with this called report2junit. It merges and coverts the JSON reports into the JUnit format used by CodeBuild Reports. By using the || true postfix the CodeBuild execution will continue. Instead, the report2junit tool will return an exit code of 1 after it generated the report. And because the report is there when it fails you can use that to see what rule failed.

This removes the need to look at the CodeBuild Logs.

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.8
    commands:
      - pip install -Ur requirements.txt
      - mkdir -p reports
  build:
    commands:
      - cfn-guard validate --rules s3.guard --data template.yml --output-format json --show-summary none > reports/s3-guard.json || true
      - cfn-guard validate --rules dynamodb.guard --data template.yml --output-format json --show-summary none > reports/dynamodb-guard.json || true
  post_build:
    commands:
      - report2junit reports/s3-guard.json reports/dynamodb-guard.json --destination-file reports/cfn-guard.xml

reports:
  Tests:
    base-directory: ./reports
    file-format: JUNITXML
    files:
      - cfn-guard.xml

Conclusion

When you use CloudFormation Guard in combination with CodeBuild Reports it makes it easier to see what rules have failed and keeps a history. When you have a solid set of compliance rules. It gives you a report that you can use to prove that the build of the infrastructure was compliant. You are also able to prevent non-compliant code rollout in production.

Python and relative imports in AWS Lambda Functions

Mon, 25 Oct 2021 12:11:42 +0200

When writing an AWS Lambda function, it’s quite possible that you get to the point where the file becomes too big. So, what do you do? You create a second file and refactor your code to do a relative import. Then, the only step left is to deploy your code and run it. And then it fails… Have you been here before? Then I might have some sound advice for you. If you look at the PEP 8 — Style Guide for Python Code.
You will read that absolute imports are recommended. But, explicit relative imports are an acceptable alternative to absolute imports.
This means that you should favor absolute imports. And only use relative imports within a package.

Sample code

Let’s create a hello world lambda function to show this. So we have the following code in a file called hello_world/app.py:

import json

def lambda_handler(event, context):
    return {
        "statusCode": 200,
        "body": json.dumps({"message": "hello world"}),
    }

For simplicity reasons I used the AWS Serverless Application Model.
The templates contain the following resource:

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.9
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get

The most important thing is the Handler definition. The value app.lambda_handler translates to:

Use the app.py file and invoke the lambda_handler method.

We will now move the payload to a file called hello_world/response.py:

def get_response() -> dict:
    return {"message": "hello world"}

And update the hello_world/app.py:

import json
from .response import get_response

def lambda_handler(event, context):

    return {
        "statusCode": 200,
        "body": json.dumps(get_response()),
    }

You run your test and it works, nice! So we will now deploy our code and that it also works. And it fails…

[ERROR] Runtime.ImportModuleError: Unable to import module 'app': attempted relative import with no known parent package
Traceback (most recent call last):

The content of the hello_world folder ends up in the /var/task/ folder. And not the folder itself.
So it might look like that the code is in a package but from the Lambda runtime it’s not.

The solution

You can solve this by creating a package. How you might ask? Create a hello_world folder in the hello_world folder.
Move all files except the requirements.txt into the created folder. Next you need to update the Handler in the template:

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: hello_world.app.lambda_handler
      Runtime: python3.9
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get

Now the value of the Handler translates to:

In the hello_world package you use the app.py file and invoke the lambda_handler method.

When you deploy you will now notice that the relative import works.

Conclusion

I would always tell you to deploy your python code in a package. It makes it easier to use many files and organize your code better.
It also allows you to separate the Lambda invocation logic from your business logic. Making it easier to test and maintain. Image by Gerd Altmann from Pixabay

Make your life easier using Makefiles

Mon, 27 Sep 2021 12:19:04 +0200

In this blog post I will show how you can use a Makefile to make your life easier. In this blog I will use CDK with python. But the principle of the Makefile will work on almost any project. When you start a new PoC you do not have a full CI/CD environment in place yet. So how can you deploy your CDK application? And make it easy to add the CI/CD pipeline at a later point?

Let’s begin!

I used CDK 1.124.0 (build 65761fe) for this blog post.

First we will need a CDK project. So we will create a working folder named: makefile_cdk_blog:

mkdir makefile_cdk_blog
cd makefile_cdk_blog
cdk init --language python
mkdir tests

At this point you will have an empty CDK project. Most projects have dependencies. You can divide them into 2 categories: * Project dependencies, dependencies needed for the project, located in setup.py. * Development dependencies, dependencies needed to develop on the project, located in requirements.txt. Let’s start with adding some development dependencies in the requirements.txt file:

-e .
black
pytest
pytest-cov
pytest-black
setuptools

Now we need to configure pytest. Add the following to a new file setup.cfg:

[tool:pytest]
testpaths = tests

[coverage:run]
branch = True
source = makefile_cdk_blog

[coverage:report]
show_missing = true
fail_under = 100
exclude_lines =
    pragma: no cover
    if __name__ == .__main__.:

The only thing left is the Makefile itself. Create a file called Makefile in the root of your project. Make sure it has the following content:

SHELL = /bin/bash -c
VIRTUAL_ENV = $(PWD)/.venv
export BASH_ENV=$(VIRTUAL_ENV)/bin/activate

$(VIRTUAL_ENV):
    python3 -m venv $(VIRTUAL_ENV)

.PHONY: install clean synth diff deploy test lint

install: $(VIRTUAL_ENV)
    pip install -r requirements.txt

clean:
    [[ -d $(VIRTUAL_ENV) ]] && rm -rf $(VIRTUAL_ENV) || true
    [[ -d .pytest_cache ]] && rm -rf .pytest_cache || true
    [[ -d cdk.out ]] && rm -rf cdk.out || true
    [[ -f .coverage ]] && rm .coverage || true

synth:
    cdk synth

diff:
    cdk diff

deploy: test
    cdk deploy

test: lint
    pytest --cov --cov-report term-missing

lint:
    black .

 $(VERBOSE).SILENT:

Alright we are good to go! First we want to commit our changes, but before we can do that we should test them. We added dependencies, so we need to install those first! But since we have a make target for that now we can use it.

make install

I will walk you through what is happening on the background:

First the Makefile will select bash as a shell.
Define the location of the virtual environment.
The BASH_ENV variable has a dependency on the $(VIRTUAL_ENV) target. It will create a virtual environment on the defined location.
The virtual environment is then exported in the BASH_ENV variable.
Then we will execute the install target. All commands are now executed in the context of the virtual environment.
pip install will install all dependencies in the requirements.txt and in the setup.py.

We can now run the test target to see if everything works as expected.

make test

You would see something like:

So what happened? We invoked the test target. This target has a dependency on the lint target. The lint target contains black . and this command formats all the python code. Afterwards, it will continue and execute the commands in the test target. So the test is failing because we did not meet the configured code coverage threshold. That is fine for now so let’s commit!

git add .
git commit -m "chore: setup project using a Makefile"

Deployment time!

Before we can deploy I assume that you configured your AWS credentials properly.

export AWS_PROFILE=my-profile-name
make deploy

And it failed? Because the deploy target has a dependency on the test target. So the test target is being executed first but that fails. This is the real power of using a Makefile. You have simple names for actions that you want to do. For example: test, deploy, etc. And in this case you should not deploy if your test fail. You can extend these targets with different checks/commands. For example, you could synthesize the template and then use the following tools: * cfn_nag * cloudformation-guard.

Conclusion

Makefiles allow you to standardize actions across projects. They can execute commands in virtual environments for you. And if kept simple they are easy to read!

Update: Improve your Makefile even more by adding a help target!

Increasing your development speed with AWS Lambda Powertools

Tue, 24 Aug 2021 21:04:05 +0200

It is not possible to remember all methods and properties from all classes that you write and use. That is the reason why I like to have type hinting in my IDE. In fact I will spend time to make sure it works. I even wrote a blog post about how to set it up when I started experimenting with a new language. In this post I will provide you with a couple of tips that help you develop faster. By leveraging type hinting using the AWS Lambda Powertools

LambdaContext

When you are developing AWS Lambda functions you receive an event and a context property. If you need metadata from the context you could read the documentation or use type hinting. As you can see in the example, the IDE will help you find the correct method or property. But what about the event? The example tells you that we expect a Dict[str, Any] but this depends on how you invoke the Lambda function.

Event Source Data Classes

Most of the time you know how you invoke your Lambda function. For example an API Gateway proxy integration could trigger the function: Next to using the data class you could also use a decorator. Then use the type hint in the method signature.

from aws_lambda_powertools.utilities.data_classes import event_source, APIGatewayProxyEvent

@event_source(data_class=APIGatewayProxyEvent)
def lambda_handler(event: APIGatewayProxyEvent, context):
    if 'helloworld' in event.path and event.http_method == 'GET':
        do_something_with(event.body, user)

For a full list see the supported event sources page.

What about my custom events?

You could also invoke a Lambda function with a custom payload. This event is not supported as a data class. Let say we receive an order with one or more items. By using the parser you can define the event in a model. And you can use that model for type hinting.

from typing import Optional
from aws_lambda_powertools.utilities.parser import event_parser, BaseModel, ValidationError
from aws_lambda_powertools.utilities.typing import LambdaContext

import json

class OrderItem(BaseModel):
    id: int
    quantity: int
    description: str

class Order(BaseModel):
    id: int
    description: str
    items: List[OrderItem] # use nesting models
    optional_field: Optional[str] # this field may or may not be available when parsing

@event_parser(model=Order)
def handler(event: Order, context: LambdaContext):
    print(event.id)
    print(event.description)
    print(event.items)

    order_items = [items for item in event.items]
    ...

But my custom event is in an existing event source

Well in that case you need to supply the event source as an envelope.

from aws_lambda_powertools.utilities.parser import event_parser, parse, BaseModel, envelopes
from aws_lambda_powertools.utilities.typing import LambdaContext

class UserModel(BaseModel):
    username: str
    password1: str
    password2: str

# Same behavior but using our decorator
@event_parser(model=UserModel, envelope=envelopes.EventBridgeEnvelope)
def handler(event: UserModel, context: LambdaContext):
    assert event.password1 == event.password2

Conclusion

If you found this blog useful I recommend reading the official documentation. It’s a small investment that will pay out in nice and clean code. No need to remember all the methods and properties and what types the expect and return. An IDE can do that for you so that you can focus on the actual business logic.

How to use AWS SAM with Go in Visual Studio Code

Tue, 27 Jul 2021 20:25:47 +0200

Setup Visual Studio Code as your IDE to build and deploy an AWS SAM stack using Go. I recently started to experiment with Go and the best way to learn is to play with it and build something. So I decided to build a simple application. During this process I ran into the problem that my IDE did not like having many go modules within one project.

Project layout

With AWS SAM you can build and deploy your stack or via a CI/CD pipeline, the process is the same. First you perform a sam build. This will compile the source code for each AWS::Serverless::Function in the template.yaml. Followed by sam deploy. This will upload the artifacts and deploy the template using AWS CloudFormation. The problem that I faced was that each function needs a go.mod in the source folder. Visual Studio Code expects a single go.mod file in the project. As a result imports in the unit test do not get recognized:

The RequestEvent struct.
The handler method.
The ResponseEvent struct. The reason for this is that the IDE has set the GOPATH to the root of the project. Therefor the imports are not correct. By using Workspace Folders you can change this behavior so that each folder has it’s own scope.

From Go 1.16+ the default build mode has switched from GOPATH to Go modules.

To do this you need to create a workspace.code-workspace file. Place it in the root of your project with the following content:

{
    "folders": [
        {
            "path": "functions/api"
        },
        {
            "path": "functions/confirm"
        },
        {
            "path": "functions/ship"
        },
        {
            "path": "functions/stock"
        },
        {
            "path": "."
        }
    ]
}

Now because each Go module lives within it’s own scope the imports are correct. As a result you are able to navigate through the code using the ⌘ click action.

Conclusion

When you deal with many Go modules within one project it is important to configure your IDE. This will improve readability, reduces errors and increases your development speed!

How to search and find related logs in AWS Log Groups

Fri, 09 Jul 2021 15:22:09 +0200

You can search across log groups by setting up a correlation id on your log lines. You can then use CloudWatch Logs Insights to query all related log messages. If you are dealing with different lambda functions in a project, you might not have a logging system yet. Finding and troubleshooting using Cloudwatch Logs could become cumbersome. Especially if you try to track a flow through the different log groups. In CloudWatch Logs Insights you can execute queries across log groups. In our case we want to collect related log lines across log groups. To achieve this we need to annotate the log lines with a correlation_id that ties them together.

Sample application

We will use the following sample application to show how to setup a correlation id on your log lines. The application looks as followed:

API Gateway that is invoking an AWS Lambda function
A api Lambda function places the body in a SQS Queue
A callback Lambda function is receiving messages from the SQS Queue Every time that an API call executes, the apifunction is invoked. We can annotate the log lines by using the Lambda Powertools for Python.

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.utilities.data_classes import (
    event_source,
    APIGatewayProxyEvent,
)
logger = Logger(service="api")

@event_source(data_class=APIGatewayProxyEvent)
@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
def lambda_handler(event: APIGatewayProxyEvent, context: LambdaContext) -> dict:
    logger.info("This message now has a correlation_id set")

We take the requestContext.requestId from the incoming event. We can use that as a correlation_id. (This is also returned to the caller as a x-amzn-requestid header by default by API Gateway.) To be able to search across log groups using the same correlation id. We need to pass it along with the message as an attribute when we send it to SQS.

    response = client.send_message(
        QueueUrl=os.environ.get("QUEUE_URL"),
        MessageBody=json.dumps(event.body),
        MessageAttributes={
            "correlation_id": {
                "DataType": "String",
                "StringValue": event.request_context.request_id,
            }
        },
    )

I used a correlation_id attribute to pass the correlation id with the SQS message. The queue will trigger the callback function with a batch of messages. You need to loop through the messages in the batch. Process them and remove them from the queue. Or you use the sqs_batch_processor from the powertools.

from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord
logger = Logger(service="callback")

def record_handler(record: dict) -> None:
    record = SQSRecord(record)
    # Get the correlation id from the message attributes
    correlation_id = record.message_attributes["correlation_id"].string_value
    logger.set_correlation_id(correlation_id)
    logger.info(f"Processing message with {correlation_id} as correlation_id")

@sqs_batch_processor(record_handler=record_handler)
def lambda_handler(event: dict, context: LambdaContext) -> None:
    logger.set_correlation_id(None)
    logger.info(f"Received a SQSEvent with {len(list(event.records))} records")

The sqs_batch_processor decorator will call the record_handler method for each record received. No need for you to handle the deletion of messages. In the end we do set the correlation id to None. We do this because the rest of the log lines are not related to the message received.

Show me!

If you call the API using a simple curl command, you can see the correlation_id (Note: this is also in the headers) So imagine that something went wrong. We know the correlation_id so we will use CloudWatch Logs Insights:

fields level, service, message
| filter correlation_id="263021ca-fb82-41ff-96ad-2e933aadbe0a"
| sort @timestamp desc

As you can see there are 3 results 2 lines originated from the api service and 1 from the callback service. If have an interest in how to create queries you can read the CloudWatch Logs Insights query syntax.

Conclusion

The Lambda Powertools for Python make it easy to add a correlation_id. By using the structured logs from the powertools you can use each property in the log line to query on. It has a lot more utilities, so I encourage go check it out! Not a python developer? There is a Java version and there are some initiatives on the roadmap to add other languages as well.