ECS on bare metal: container scheduling without AWS

ECS is the part of the AWS stack that teams reach for when they want container orchestration without the complexity surface of Kubernetes. A cluster, some task definitions, a service, and a load balancer, and the scheduler handles placement and restarts. The problem is that ECS is deeply tied to AWS. The scheduler is a managed service, the container instances are EC2 instances, and the whole thing assumes a live connection to amazonaws.com. Spinifex implements ECS on the EC2 launch type so the same model runs on bare metal, in disconnected environments, and behind whatever network boundaries your operational context requires.

How ECS on Spinifex works

Spinifex follows the AWS EC2 launch type model directly: you supply the compute. A cluster is a logical grouping. The capacity behind it is container instances, which are ordinary EC2 instances booted from Spinifex's spinifex-ecs-node image, each running the Spinifex ECS agent. The agent registers the instance with the cluster, reports its available CPU and memory, and runs the containers the scheduler places on it.

There is no serverless capacity equivalent and no Fargate-style launch type. RequiresCompatibilities of FARGATE is accepted in a task definition but not honoured. If you're used to Fargate, the shift is that you manage the container instance count yourself rather than letting AWS provision capacity on demand, but the task definition format and service configuration remain the same.

A task definition describes one or more containers with image, CPU and memory limits, port mappings, environment variables, and an optional task IAM role. A task is a running instantiation of that definition, and the scheduler places tasks on instances with sufficient free capacity. A service keeps a desired count of tasks running, replaces any that fail, and registers each task's IP with an ALB target group when load balancer configuration is present.

Before you start

Container instances must boot Spinifex's spinifex-ecs-node image, which ships with the ECS agent pre-installed. Confirm the image is imported:

aws ec2 describe-images \
  --filters 'Name=tag:spinifex:managed-by,Values=ecs' \
  --query 'Images[].[ImageId,Name]' --output text

If no rows come back, the image is not imported and needs to be registered before you can provision capacity.

Container instances also need the ecsInstanceRole instance profile, which is a role trusted by ec2.amazonaws.com with an ecs:* policy, exposed through a profile of the same name. The Spinifex console's provision-capacity action creates it on first use, and the ECS Quickstart Terraform workbook creates it automatically unless you opt out.

Creating a cluster and registering a task definition

export AWS_PROFILE=spinifex-<nodename>

aws ecs create-cluster --cluster-name demo

Then register a task definition. This example uses awsvpc network mode, EC2 launch type, and one nginx container on port 80:

aws ecs register-task-definition \
  --family web \
  --network-mode awsvpc \
  --requires-compatibilities EC2 \
  --cpu 256 --memory 512 \
  --container-definitions '[{
    "name":"web",
    "image":"docker.io/library/nginx:1.27-alpine",
    "portMappings":[{"containerPort":80,"protocol":"tcp"}],
    "essential":true
  }]'

awsvpc network mode means each task gets its own ENI and private IP in your subnet, matching the AWS behaviour. If you're fronting tasks with an ALB target group, it must use target_type = "ip".

Provisioning container instances

Once the cluster exists, add capacity by launching EC2 instances from the ECS node image with the ecsInstanceRole profile attached. The cloud-init user data needs to point the agent at the cluster name. The ECS Quickstart Terraform workbook generates this user data for you, and the Spinifex console wraps the same call in a Provision capacity action.

Once instances boot and the agent registers, verify they appear:

aws ecs list-container-instances --cluster demo

Running tasks and creating services

Run a one-off task:

aws ecs run-task \
  --cluster demo --task-definition web --count 1 \
  --network-configuration 'awsvpcConfiguration={subnets=[subnet-aaaa]}'

Or create a service to keep a desired count running behind a target group:

aws ecs create-service \
  --cluster demo --service-name web --task-definition web \
  --desired-count 2 \
  --network-configuration 'awsvpcConfiguration={subnets=[subnet-aaaa]}' \
  --load-balancers 'targetGroupArn=<tg-arn>,containerName=web,containerPort=80'

aws ecs describe-services --cluster demo --services web \
  --query 'services[0].[runningCount,desiredCount]'

Task IAM roles and credential delivery

If your containers need to call AWS APIs, give the task definition a taskRoleArn. The role must be trusted by ecs-tasks.amazonaws.com:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "ecs-tasks.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}

When a task has a taskRoleArn, the ECS agent injects AWS_CONTAINER_CREDENTIALS_RELATIVE_URI into the container environment and serves short-lived credentials for that role at 169.254.170.2. Any AWS SDK in the container picks these up automatically without static keys. If executionRoleArn is set, the agent uses it to authorise ECR image pulls instead of falling back to the container instance role.

Logging

Spinifex honours the json-file log driver, which is the containerd default. Container stdout and stderr are written on the container instance and are accessible there. There is no CloudWatch Logs path.

To read a task's logs, find the container instance running it and inspect the containerd output on that host:

# On the container instance:
ctr -n default containers ls                # find {taskId}-{containerName}
ctr -n default tasks ls
journalctl -u containerd | grep <taskId>    # container stdout/stderr via the host journal

Task definitions that declare awslogs or another log driver are accepted without error, and Spinifex logs a warning at registration naming the container, so the fallback is not silent. Containers are named {taskId}-{containerName} and carry mulga.ecs.* labels.

Deployment circuit breaker

UpdateService to a new task-definition revision performs a rolling update that honours deploymentConfiguration: minimumHealthyPercent keeps that fraction of the desired count running while maximumPercent bounds how many extra tasks can launch during the roll. Enabling the deployment circuit breaker causes a rollout to fail if its tasks repeatedly fail to start, and with rollback enabled it automatically reverts to the last-good task definition revision.

Current limitations

ECS v1 in Spinifex is deliberately minimal. Known gaps:

No service discovery. serviceRegistries and Cloud Map integration are not implemented. Reach a service through its load balancer rather than a DNS name.
No capacity providers or managed scaling. Capacity is the static total of your registered instances and there is no ASG binding or scale-out. You manage the instance count.
secrets[] are rejected. A task definition that declares container secrets[] fails RegisterTaskDefinition with InvalidParameterException rather than running without the secrets it expects. Use environment variables or fetch secrets at startup instead.
Tags via TagResource are not persisted. Tag the resources at creation time rather than updating tags after the fact.

When this is the right model

ECS on Spinifex fits the same scenarios as ECS on AWS, applied to environments where AWS isn't available or appropriate: disconnected field deployments, regulated environments that require workloads on hardware under direct control, and on-premise data centres where teams have already built tooling around ECS task definitions.

The task definitions and service configuration that work in AWS ECS work in Spinifex without modification, so teams moving workloads from cloud to on-premise hardware don't need to rewrite their container infrastructure to do it.

Get started

The full ECS walkthrough, including the Terraform workbook for provisioning clusters and container instances end-to-end, is in the ECS documentation. Source is at GitHub, AGPL-3.0 licensed and written in Go. Or sign up for our free sandbox to explore ECS on Spinifex before deploying on your own hardware.

Running ECS on Bare Metal With Spinifex