Migrating from AWS Lambda to AWS ECS: Benefits and Challenges

AWS Lambda is a powerful service that provides a hassle-free way to run our code in the cloud without needing to concern ourselves with the underlying infrastructure.
At Taxdoo, we rely on Lambda for a vast array of services, as it streamlines our development process and ensures our focus remains on delivering innovative solutions.

However, while Lambda is an ideal choice for many use cases, it’s not always the perfect fit. There are times when a more flexible and customizable approach is necessary, such as when dealing with long-running tasks or more complex workloads. In these situations, containerized solution shines as a robust and scalable container orchestration service that can accommodate a wide range of needs. By moving from Lambda to the containerized solution, we were able to achieve greater control and flexibility while maintaining the efficiency and convenience we’ve come to expect from AWS services.

Recently, we encountered a well-known limit of Lambda functions, which is why they have a 15-minute execution limit. Over the years, as Taxdoo has grown so has the amount of clients we need to prepare invoices for. This task used to take a few minutes to complete, but not too long ago, we quickly encroached on the 15-minute time limit.

We needed to move our workload to an AWS service which didn’t impose a 15-minute execution time limit. We needed something that could be set up fast and required very little maintenance.

Furthermore, we considered moving the workload to an EC2 instance but quickly dismissed the idea as the workload only runs a few times monthly and also runs for around a maximum of 30 minutes. While we could have spawned an instance as needed, doing so would have burdened our team unnecessarily. Furthermore, we preferred to continue using CloudWatch rules for scheduling and invoking our workloads, which can be complex to manage with EC2.

Moving from AWS Lambda to a containerized solution, we chose to use AWS ECS, a fully managed container orchestration service that simplifies the deployment, management, and scaling of Docker containers on the AWS cloud using EC2 instances. Here are some of the key benefits of using ECS over EKS that we found:

Simplified management: With ECS, we were able to simplify the management of our containerized applications. ECS is easier to set up and manage compared to EKS, especially for those who are new to Kubernetes. EKS requires more investment from a technical perspective and base cost perspective.
Better integration with AWS services: As an AWS customer, we found that ECS integrates more seamlessly with other AWS services like CloudWatch or CloudFormation This made it easy for us to build, test, and deploy our containerized applications using familiar AWS tools.
Focus on our application code: With ECS, we were able to focus on writing our business logic instead of worrying about the underlying infrastructure. ECS provides a managed Docker environment for our containers, which meant we didn’t need to spend time setting up and maintaining our own cluster inside AWS.
Cost-effective solution: We found ECS to be a cost-effective solution for our container orchestration needs. We didn’t need the same level of infrastructure or any expertise in Kubernetes as it would be needed for EKS, which helped to keep our costs down.
Highly scalable: ECS is highly scalable, allowing us to quickly scale up or down based on the demand of our application. This ensured that we could deliver the best performance and experience to our customers.

Overall, we found ECS to be a great choice for our company when moving from AWS Lambda to a containerized solution. It simplified our management, integrated better with AWS services, allowed us to focus on our application code, and provided a cost-effective and highly scalable solution.

ECS allows us to run our tasks on EC2 instances which ECS provisions for us as and when required, but it also allows us to run our workloads using Fargate. Fargate allows us to run our ECS tasks and services on AWS-managed servers, similar to Lambda.

When comparing an ECS Fargate task to Lambda, it’s important to note that the primary distinction is that an ECS task requires a Docker image, whereas Lambda can run a variety of programming languages natively (as well as Docker images, if desired).

Initially, our service was intended to operate within a Java lambda function. In order to transition to a containerized solution, our first step was to create a Docker image capable of running our task. To achieve this, we made only a minor modification to our task code by implementing a main method that could be linked to a Docker image. Fortunately, we found that a simple and straightforward vanilla Java Docker image in which we simply referenced our output jar file was more than sufficient for our needs.

				
					FROM openjdk:19-jre-slim
COPY project/path/code.jar /docker/path/code.jar

ENTRYPOINT ["java","-jar","/docker/path/code.jar"]

In order to run ECS tasks within an ECS cluster, we need to create a task definition. We did this using AWS CDK. One particularly useful method offered by the CDK was `ecs.ContainerImage.fromAsset` which did three things for us:

Build the docker image for us, push it to a managed registry for us, and link it to the task definition we’re defining.

				
					const createEcsFargateTask = () => {
  const taskDefinition = new ecs.FargateTaskDefinition(
    this,
    `yourTaskName`,
    {
      cpu: ...,
      memoryLimitMiB: ...,
      taskRole: taskRole, // role which is executing the java code
      executionRole: executionRole, // role which is triggering the ecs fargate task itself
      runtimePlatform: {
        // define runtime here
      },

				
					 }
  );

  taskDefinition.addContainer(`yourContainerInstanceName`, {
    containerName: 'yourContainerName',
    // import docker image here for the container
    image: ecs.ContainerImage.fromAsset('..', {
      exclude: [
        // good for excluding unused modules
      ],
    }),
    environment: {
      // set environment variables here
    },
  });

  return taskDefinition;
}

Code we added inside our CDK configuration

And just like that, with a simple update to the target for our CloudWatch rule, we bid adieu to Lambda and welcomed our application into the world of ECS. Migration complete!

AJ Cole

Software Engineer

Back to blog

28 September 2023,