> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ankra.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AWS Clusters

> Create a self-managed kubeadm or k3s cluster on EC2 instances in your AWS account, in a VPC Ankra creates or one you already own, with Ankra installing and operating Kubernetes.

Choose self-managed EC2 over [Amazon EKS](https://docs.ankra.ai/guides/eks-clusters.md) when you want the control plane on instances you own, with no EKS control-plane fee, or when the cluster has to join a VPC you already run. Ankra creates the network (or adopts yours), a bastion, the control plane and worker instances, installs Kubernetes with the AWS cloud controller manager and EBS CSI driver, and then operates the cluster - this is an **Ankra Managed** cluster. The [comparison table](https://docs.ankra.ai/guides/managed-kubernetes.md#cloud-managed-or-ankra-managed) sets the two side by side.

## Before you start

* An **AWS credential** whose role carries the **Provisioning** scope (the one write role for EKS and self-managed clusters) or the EC2-only **Self-managed** scope; access keys with the equivalent policy also work. A provisioning stack launched before the self-managed statements were added must be updated. See [AWS Credentials](https://docs.ankra.ai/platform/credentials/aws.md#provisioning).
* An **SSH key credential** - your own public key, or one Ankra generates. See [SSH Key Credentials](https://docs.ankra.ai/platform/credentials/ssh-key.md).
* A decision on the **network**. Nothing needs to exist in the account beforehand unless you adopt a VPC.

### Created or adopted VPC

* **Created VPC** (the default; leave out `vpc_id`). Ankra creates a VPC from `network_ip_range` (default `10.0.0.0/16`, any `/16` to `/20`) with, per availability zone, a private `/20` for the nodes and a public `/24` for the bastion and NAT gateway, an internet gateway and route tables. It uses one zone when the control plane has fewer than three nodes and three zones otherwise, or the `availability_zones` you name; more than one zone needs at least three control planes, so etcd keeps quorum when a zone is lost. Teardown removes all of it.
* **Adopted VPC** (`vpc_id`). Ankra builds inside a VPC you already run: private `node_subnet_ids` for the control plane and workers (spread them across zones for a multi-AZ cluster) and a public `bastion_subnet_id`. Ankra never creates, modifies or deletes your VPC, subnets, internet gateway, NAT gateways, DHCP options or existing route tables. `network_ip_range`, `availability_zones` and `nat_gateway_single_zone` are refused alongside `vpc_id`, and the subnet fields without it.

Nodes never get a public IP, so they need an egress path (`egress_mode`):

| Mode | Where | What it is | Cost |
| - | - | - | - |
| `nat_gateway` | Created VPC (the default there) | An Ankra-created NAT gateway per zone, so egress survives a zone outage; `nat_gateway_single_zone: true` shares one | About USD 32 a month per gateway plus per-GB processing ([VPC pricing](https://aws.amazon.com/vpc/pricing/)) |
| `bastion_nat` | Either | The bastion forwards node traffic. Egress is capped by the bastion's network performance, and stops while it restarts or resizes | The bastion and its Elastic IP only |
| `existing` | Adopted VPC | Your NAT gateway, NAT instance or transit path already on the node subnets' route table; Ankra changes nothing | What you already pay |

For an adopted VPC, preflight picks the mode when you leave it out: subnets that already have egress get `existing`, empty subnets without it get `bastion_nat`. `bastion_nat` is always refused when a node subnet holds an instance Ankra did not create, because re-pointing that subnet's default route would take over its egress; create a NAT gateway and use `existing` instead. [Egress modes](https://docs.ankra.ai/reference/aws.md#egress-modes) has the full rules.

## Create the cluster

**Dashboard**

**1. Open the create dialog**

Go to **Clusters**, click **Create cluster** and pick **AWS** under **Ankra Managed** (the **AWS** card under **Cloud Managed** creates [EKS](https://docs.ankra.ai/guides/eks-clusters.md)).

**2. Basics**

Name the cluster and pick the AWS credential, the SSH key and the region. A read-only (cost) credential, or a provisioning role whose stack predates the self-managed statements, is listed but refused at preflight with the missing permission named.

**3. Network**

Keep **Create a new VPC (recommended)** - optionally with your own network range and availability zones - or switch to **Use an existing VPC** and pick the VPC, the node subnets and a public bastion subnet; the wizard shows each subnet's zone and route. Choose the egress. **Bastion allowed IPs** starts as `0.0.0.0/0`: SSH is key-only, password login is off and sshd is rate-limited on the host, but replace it with your own CIDRs to close the port to everyone else. Ankra's own addresses stay admitted.

**4. Compute**

Pick the control plane count (1, 3 or 5) and instance type, one or more worker node groups, the bastion instance type (a burstable type such as `t3.small` is enough, unless it is the NAT), the root volume size (an encrypted gp3 volume, 40 GiB by default), the Ubuntu series (24.04 by default) and what happens to EBS volumes on teardown. Instance types load live with prices; only `amd64` types are offered.

**5. Kubernetes**

Keep **kubeadm** (the default, Cilium only) or pick **k3s**, the version, and the etcd topology or CNI. The CNI defaults to Cilium; a k3s cluster can pick Calico or flannel, but only Cilium and Calico enforce the [IMDS guard](https://docs.ankra.ai/reference/aws.md#imds-guard), so flannel gets a preflight warning. See [Choices fixed at create time](https://docs.ankra.ai/guides/operate-ankra-managed-clusters.md#choices-fixed-at-create-time).

**6. GitOps**

Optionally connect a Git repository. Two checkboxes are on by default:

* **Include Networking Stack** - Traefik, cert-manager and a Let's Encrypt ClusterIssuer, with Traefik behind a Network Load Balancer the AWS cloud controller creates.
* **Include DNS Integration** - a delegated subdomain on `ankra.cc` with external-dns wired, so Ingress hostnames publish their DNS records automatically.

**7. Review**

Click **Run preflight**. It reports the network ownership, the zones it resolved, the egress mode, the AMI, and the vCPU, VPC, NAT gateway, Elastic IP and security-group quotas. Once nothing is an error, click **Create cluster** and follow the progress view.

**CLI**

```bash theme={null}
# Provisioning role: print the external ID and CloudFormation link, launch the stack, then store the role
ankra credentials aws onboarding --scope provisioning      # or --scope self_managed for the EC2-only role
ankra credentials aws create-role --name my-aws --role-arn <role-arn> --external-id <external-id> \
  --region eu-north-1 --scope provisioning
# Or access keys (the secret is asked for on a masked prompt)
ankra credentials aws create-keys --name my-aws-keys --access-key-id <access-key-id> --region eu-north-1

ankra credentials hetzner ssh-key create --name my-ssh-key --generate   # any SSH key credential works
```

The external ID must be the one `onboarding` printed and the stack was launched with; a role stored at the `cost` scope is refused at create. Look up what the credential can reach with `ankra cluster aws regions`, `availability-zones`, `instance-types`, `images` and `pricing` (all take `--credential-id` and `--region`), and `vpcs` and `subnets --vpc-id <vpc-id>` when you adopt a VPC.

`preflight` takes the same flags as `create` and creates nothing. A three-zone cluster in a created VPC:

```bash theme={null}
ankra cluster aws preflight \
  --name my-cluster \
  --credential-id <aws-credential-id> \
  --ssh-key-credential-id <ssh-key-credential-id> \
  --region eu-north-1 \
  --bastion-allowed-ips 203.0.113.10/32 \
  --control-plane-count 3 \
  --control-plane-type t3.medium \
  --worker-count 3 \
  --worker-type t3.large
# When it reports no errors, run the same command with create instead of preflight
```

`--network-ip-range`, `--availability-zones`, `--egress-mode nat_gateway|bastion_nat` and `--nat-gateway-single-zone` shape a created network. To adopt a VPC instead:

```bash theme={null}
ankra cluster aws create \
  --name my-cluster \
  --credential-id <aws-credential-id> \
  --ssh-key-credential-id <ssh-key-credential-id> \
  --region eu-north-1 \
  --vpc-id <vpc-id> \
  --node-subnet-ids <subnet-a>,<subnet-b>,<subnet-c> \
  --bastion-subnet-id <public-subnet> \
  --egress-mode existing \
  --bastion-allowed-ips 203.0.113.10/32 \
  --control-plane-count 3 \
  --worker-count 3
```

`--distribution k3s`, `--cni`, `--cni-features`, `--retention-policy retain|delete`, `--environment`, `--criticality` and the `--gitops-*` flags cover the other choices. Every flag is in the [CLI reference](https://docs.ankra.ai/reference/cli/cluster.md).

**API**

Preflight takes the same body as create and answers with findings - `can_proceed`, `network_ownership`, `resolved_availability_zones` and `resolved_egress_mode` - instead of a cluster:

```bash theme={null}
curl -X POST https://platform.ankra.app/api/v1/clusters/aws/preflight \
  -H "Authorization: Bearer $ANKRA_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-cluster",
    "credential_id": "<aws-credential-id>",
    "ssh_key_credential_id": "<ssh-key-credential-id>",
    "region": "eu-north-1",
    "availability_zones": ["eu-north-1a", "eu-north-1b", "eu-north-1c"],
    "egress_mode": "nat_gateway",
    "bastion_allowed_ips": ["203.0.113.10/32"],
    "control_plane_count": 3,
    "control_plane_type": "t3.medium",
    "node_groups": [
      {"name": "default", "instance_type": "t3.large", "count": 3}
    ],
    "retention_policy": "retain"
  }'
```

Send the same body to `POST /api/v1/clusters/aws` to create the cluster. It answers `cluster_id`, `kind: "aws"`, `state: "creating"` and an `operation_id` that tracks provisioning. To adopt a VPC, replace the network fields with `"vpc_id"`, `"node_subnet_ids"`, `"bastion_subnet_id"` and `"egress_mode": "existing"` or `"bastion_nat"`; the created-network fields are then refused with `422`. Every field and default is in the [AWS Reference](https://docs.ankra.ai/reference/aws.md). Ankra's AI can create the cluster in chat too, running the preflight for you.

## Verify

1. The cluster moves from **Provisioning** to **Online** in the clusters list once the Ankra Agent has connected.
2. Point `kubectl` at it through Ankra (this needs a [Cluster Access](https://docs.ankra.ai/guides/cluster-access.md) grant) and check the nodes:

   ```bash theme={null}
   ankra cluster kubeconfig add my-cluster --use
   kubectl get nodes
   ```

   Every control plane and worker should be `Ready`. See [Accessing Clusters with kubectl](https://docs.ankra.ai/guides/kubeconfig.md).

## What Ankra created in your account

* **The network**, in a created VPC: the VPC, subnets, internet gateway, route tables and, in `nat_gateway` mode, NAT gateways with their Elastic IPs. In an adopted VPC, at most one route table (`bastion_nat` mode).
* **Two security groups** (nodes and bastion), an imported **key pair**, one **Elastic IP** for the bastion, and the **bastion**, control plane and worker instances with encrypted gp3 root volumes and no public IPs on the nodes.
* **Two IAM roles with instance profiles**, `ankra-k3s-<cluster>-cp` and `ankra-k3s-<cluster>-node`, that the cloud controller manager and EBS CSI driver use. The credential you gave Ankra is never copied into the cluster.
* **The AWS cloud-provider stack**: the cloud controller manager (load balancers for `LoadBalancer` Services), the EBS CSI driver (encrypted gp3 volumes by default) and the IMDS guard.

Everything carries `ankra.cloud/cluster-id=<id>`, `ankra.cloud/managed=true` and `kubernetes.io/cluster/<id>=owned`, and the credential's write permissions are scoped to those tags. [What Ankra creates](https://docs.ankra.ai/reference/aws.md#what-ankra-creates) has the full list and the network diagram.

## AWS specifics

* **Node groups follow the node subnets.** New nodes are balanced across the cluster's zones - each takes the zone with the fewest instances - unless the group is pinned with `--availability-zone` (API `availability_zone`). Pin a group that runs zonal storage: an EBS volume cannot attach from another zone. Every node carries `topology.kubernetes.io/zone` and `topology.kubernetes.io/region`, so `topologySpreadConstraints` work as they are.
* **An instance type change replaces nodes.** EC2 cannot resize a running instance, so each node of the group is drained, terminated and recreated on the new type, one at a time.
* **Control plane.** A cluster whose nodes span more than one zone needs three control planes and cannot go below three. `ankra cluster aws control-plane get|set-count|set-instance-type` manage it.
* **Restart** is an EC2 reboot; if the instance does not respond it falls back to a stop and start, which keeps the private IP. `ankra cluster aws nodes cloud-init-log` reads a node's cloud-init status over the bastion. OS-level diagnosis and repair over SSH run through Ankra's AI (`ssh_diagnose_node`, `ssh_repair_node`).
* **Bastion.** Resize it from **Nodes → Bastion & VMs** or with `PUT /api/v1/clusters/aws/{cluster_id}/bastion/instance-type`; there is no CLI verb on AWS. The instance is stopped, changed and started, keeping its Elastic IP. SSH as `ubuntu` through it (`ankra cluster aws access-info`); only `bastion_allowed_ips` and Ankra's addresses reach it.
* **Upgrades** move the AWS cloud controller manager with the nodes: its minor version is pinned to the cluster's Kubernetes minor.

> [!WARNING]
>
> In `bastion_nat` mode the bastion is every node's default route, so a bastion resize or restart stops image pulls, the Ankra Agent connection and the cloud controller's AWS calls until it is back.

### Stopping and starting

A stop stops every node and the bastion rather than terminating them, so the instances keep their root volumes and private IPs, and the bastion keeps its Elastic IP. Start brings the same instances back and etcd rejoins on the same addresses, so the cluster returns with its state, not as a rebuild. Ankra records each instance's private IP at create and checks it after every start; a difference is reported as drift. While stopped, compute is not billed, but EBS storage, the Elastic IP, load balancers and - in a created VPC - the NAT gateways are, so a cluster parked for weeks on `nat_gateway` egress is cheaper to terminate. A stop never deletes the volumes the CSI driver provisioned, whatever the `retention_policy`.

### Terminating

A terminate terminates the instances, releases the bastion's Elastic IP, and deletes the security groups, key pair and IAM roles. In a created VPC it then removes the network in reverse order - NAT gateways with their Elastic IPs, route tables, subnets, internet gateway, VPC - by the tags Ankra set. In an adopted VPC it restores each node subnet's previous route table association before deleting the one route table it created, and leaves the rest of your network alone.

EBS volumes the CSI driver provisioned, and load balancers the cloud controller created, follow the cluster's `retention_policy`, chosen at create:

* **`retain`** (the default) keeps them in your account, where they keep billing. The teardown result lists their IDs, and they carry the cluster ID tag. `--force` still deletes the load balancers, never the volumes, and no volume question is asked.
* **`delete`** removes those tagged `ankra.cloud/managed=true` for this cluster - the EBS volumes only once you accept it.

With a created VPC, load balancers are always deleted, because they cannot outlive the VPC; volumes still follow the policy.

```bash theme={null}
ankra cluster aws deprovision <cluster>                            # names the EBS volumes a delete cluster loses, and asks
ankra cluster aws deprovision <cluster> --accept-volume-data-loss  # for scripts
```

## Operate it

Day-2 tasks work the same way on every Ankra Managed provider, with the CLI name `aws`:

* [Node groups](https://docs.ankra.ai/guides/operate-ankra-managed-clusters.md#node-groups) and [legacy worker scaling](https://docs.ankra.ai/guides/operate-ankra-managed-clusters.md#legacy-worker-scaling)
* [Control plane](https://docs.ankra.ai/guides/operate-ankra-managed-clusters.md#control-plane)
* [Restart a node](https://docs.ankra.ai/guides/operate-ankra-managed-clusters.md#restart-a-node) and [the bastion](https://docs.ankra.ai/guides/operate-ankra-managed-clusters.md#bastion)
* [SSH access and keys](https://docs.ankra.ai/guides/operate-ankra-managed-clusters.md#ssh-access-and-keys)
* [Upgrade Kubernetes](https://docs.ankra.ai/guides/operate-ankra-managed-clusters.md#upgrade-kubernetes)
* [Stop and start](https://docs.ankra.ai/guides/operate-ankra-managed-clusters.md#stop-and-start)
* [Terminate a cluster](https://docs.ankra.ai/guides/operate-ankra-managed-clusters.md#terminate-a-cluster)

## Troubleshooting

| Issue | Solution |
| - | - |
| Preflight refuses the credential | It is read-only (cost scope), or its provisioning stack predates the self-managed statements. Add a [provisioning credential](https://docs.ankra.ai/platform/credentials/aws.md#provisioning) or update the stack to the published template |
| The preflight `cidr` item is an error | `network_ip_range` does not hold a private `/20` and a public `/24` per zone - a `/20` VPC fits one zone. Widen the range (up to `/16`) or use fewer zones |
| Preflight refuses the zones | A named zone does not exist or is not available, or several zones were given with fewer than three control planes. Check `ankra cluster aws availability-zones` |
| Preflight reports the VPC or NAT gateway quota | The account is at its VPCs-per-region (`L-F678F1CE`) or NAT-gateways-per-zone (`L-FE5A380F`) limit. Request an increase, remove unused clusters, or use `bastion_nat` or `nat_gateway_single_zone` |
| Preflight refuses `bastion_nat` (adopted VPC) | A node subnet holds an instance Ankra did not create. Pick an empty subnet, or create a NAT gateway and use `existing` |
| Preflight refuses `existing` (adopted VPC) | The node subnets have no `0.0.0.0/0` route to a NAT gateway, NAT instance or transit path. Add one, or use empty subnets with `bastion_nat` |
| `node_subnet_ids` or `network_ip_range` refused with `422` | Subnet fields belong to an adopted VPC and network fields to a created one - send `vpc_id` with subnets, or neither |
| `bastion_allowed_ips` refused | An entry is not an IPv4 CIDR. `0.0.0.0/0` and `::/0` are accepted, and an empty list means `0.0.0.0/0` |
| Cluster stuck in provisioning | Check the vCPU quota for the instance family, the Elastic IP and NAT gateway quotas, and - in an adopted VPC - that the bastion subnet routes to the internet gateway |
| Nodes stay `NotReady` with the `node.cloudprovider.kubernetes.io/uninitialized` taint | The cloud controller cannot match the node to its instance - see below |
| `LoadBalancer` Service stuck in `Pending` | Check the cloud controller pods in `kube-system`. On a fresh account the first load balancer also creates the Elastic Load Balancing service-linked role, which the credential allows |
| PVC stuck in `Pending` | Check the EBS CSI controller pods; a pod pinned to a zone with no volume there needs a node group in that zone |
| Cannot SSH to the bastion | Your address is not in `bastion_allowed_ips`, or the security group was edited outside Ankra |
| Nodes lost egress after a bastion resize | Expected in `bastion_nat` mode while the bastion restarts |
| Create result reports `imds_guard: unavailable` | The cluster runs flannel, which cannot enforce the IMDS guard - see below |

**Cloud controller cluster ID and node names.** The cloud controller finds a cluster's instances by the `kubernetes.io/cluster/<id>=owned` tag and matches each node to an instance by name. Ankra sets both: the tag on every instance, and each node name to the instance's EC2 private DNS name as `DescribeInstances` reports it (k3s gets `--node-name`), so a VPC whose DHCP options set carries a custom `domain-name` does not break the match, and changing the DHCP options later affects no node. Removing or editing the tag in the console makes the controller treat the instance as foreign until the tag is back. If a node was renamed or re-registered by hand, delete its Kubernetes node object and restart the node from **Nodes**; it registers again under the right name.

**Instance metadata (IMDS).** Every instance requires IMDSv2, so tools that use IMDSv1 fail on the nodes. Workers and the bastion have a hop limit of 1, so pods cannot reach the instance role. Control planes have a hop limit of 3 so the EBS CSI controller can use the control-plane role; the IMDS guard, a network policy rendered for Cilium and Calico, blocks every other pod from `169.254.169.254`. On flannel there is no policy engine, so a pod scheduled onto a control plane can read that role's credentials: prefer Cilium or Calico, or keep the control planes tainted. Give a workload that needs AWS access its own credentials; do not raise the hop limits or re-enable IMDSv1 - Ankra restores the hardening on reconciliation.

**AWS service quotas.** Preflight checks the quotas that most often block a create, but a quota can be used up between preflight and create, and the per-cluster quota report answers `404` for AWS. If provisioning fails with a quota error, check the [Service Quotas console](https://console.aws.amazon.com/servicequotas/) for the region - On-Demand vCPUs for the instance family, VPCs per region, NAT gateways per zone, Elastic IPs (one for the bastion plus one per NAT gateway), security groups per interface and rules per group, and Network Load Balancers - then create again. A failed create leaves only tagged objects, which Ankra removes on the next attempt.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.