natkit

Getting started

Mon, 01 Jan 0001 00:00:00 +0000

natkit is a NAT instance AMI for AWS, built for arm64 and x86_64. It has fck-nat’s features on LinuxKit instead of Amazon Linux, systemd and Packer. The OS is an immutable in-memory image:

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

natkit reads its settings from YAML user-data, with the same keys as fck-nat’s /etc/fck-nat.conf. Every key is optional: with none at all, natkit is a plain NAT out of the primary ENI.

 eip_id: eipalloc-0123 # associate this EIP with the primary ENI (after eni_id's ENI is attached)
eni_id: eni-0123 # HA: attach this static ENI (the route tables' target) as device 1,
 # turn off the primary ENI's source/dest check
interface: eth0 # public interface, if not the primary ENI's (ignored with eni_id)
nat64: false # no NAT64 (default: on)
ip_local_port_range: "1024 65535" # tuning, applied only when set
nf_conntrack_max: 262144
nf_conntrack_buckets: 65536
nf_conntrack_tcp_timeout_established: 86400
tcp_keepalive_time: 600
tcp_max_syn_backlog: 4096
cwagent_enabled: true # CloudWatch agent with the JSON config in this SSM parameter
cwagent_cfg_param_name: /natkit/cwagent
console_password: /natkit/root-hash # SSM parameter holding `openssl passwd -6` output: serial console login

An unknown key or a malformed value stops the setup. The console then prints natkit: FAIL <reason>.

High availability

Mon, 01 Jan 0001 00:00:00 +0000

For HA, run natkit in an Auto Scaling group of one, with eni_id and eip_id in its user-data:

Static ENI (eni_id): the private route tables point at this ENI, not at an instance. The instance attaches it as device 1.
EIP (eip_id): the instance associates it with its primary ENI, so egress keeps the same public IP across replacements.

When an instance is replaced:

The new instance keeps retrying the ENI attachment while the old one still holds it, for up to 15 minutes.
Once it has the ENI, it takes the EIP.

fck-nat takes the EIP first. That moves egress off the EIP while the old instance is still routing. natkit waits for the ENI first to avoid that gap.

NAT64

Mon, 01 Jan 0001 00:00:00 +0000

NAT64 is on unless the user-data sets nat64: false. Jool translates 64:ff9b::/96 to IPv4.

To use it:

turn on Route 53 Resolver DNS64 on the IPv6 subnets;
add a 64:ff9b::/96 route to the instance;
give the instance’s ENI an IPv6 address.

Jool’s default IPv4 pool is ports 61001-65535 of the instance’s address. Keep ip_local_port_range below that.

OpenTofu module

Mon, 01 Jan 0001 00:00:00 +0000

tofu/nat sets up the HA layout as a module:

an Auto Scaling group of one;
a static ENI and an EIP;
the private route tables’ routes;
a scoped instance role.

 module "nat" {
 source = "github.com/ulm0/natkit//tofu/nat"
 vpc_id = aws_vpc.main.id
 subnet_id = aws_subnet.public.id
 route_table_ids = [aws_route_table.private.id]
 ami_id = "ami-..." # natkit-<version>-arm64
 instance_type = "t4g.micro"
 settings = { nf_conntrack_max = "262144" }
}

Inputs

Name	Default
`vpc_id`		the VPC
`subnet_id`		public subnet (internet gateway route) for the instance and its static ENI
`route_table_ids`		private route tables that get `0.0.0.0/0` (and `64:ff9b::/96` with `nat64`) to the static ENI
`ami_id`		a natkit AMI of the instance type’s architecture
`name`	`natkit`	name and `natkit` tag of every resource
`instance_type`	`t4g.micro`	see instance size: `t4g.nano` is not supported
`eip`	`true`	allocate an EIP and have the instance take it. Otherwise it egresses from its own public IP, which changes on replacement
`nat64`	`false`	route `64:ff9b::/96` to the ENI and give it an IPv6 address (the subnet needs an IPv6 CIDR)
`ipv6_cidr`	`null`	the VPC’s IPv6 CIDR, admitted by the security group: needed with `nat64`
`settings`	`{}`	more user-data keys
`cwagent_cfg_param_name`	`null`	SSM parameter with a CloudWatch agent JSON config: enables the agent
`console_password_param`	`null`	SSM parameter with root’s crypt hash for the EC2 Serial Console
`tags`	`{}`	extra tags

Outputs

eni_id, public_ip (the EIP, with eip = true), security_group_id, role_name.

Instance role

Mon, 01 Jan 0001 00:00:00 +0000

natkit only needs an instance role for the keys that call AWS:

 {
 "Version": "2012-10-17",
 "Statement": [
 {"Effect": "Allow", "Action": "ec2:AssociateAddress", "Resource": "*"},
 {"Effect": "Allow", "Action": ["ec2:AttachNetworkInterface", "ec2:ModifyNetworkInterfaceAttribute"], "Resource": "*"},
 {"Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws:ssm:*:*:parameter/natkit/*"}
 ]
}

With cwagent_enabled, also attach CloudWatchAgentServerPolicy.

Scope Resource down to your EIP, ENI and instances where you can. The OpenTofu module’s role does this: tag conditions on the ENIs and instances, and the one EIP and SSM parameters by ARN.

CloudWatch agent

Mon, 01 Jan 0001 00:00:00 +0000

To run the CloudWatch agent, set these user-data keys:

 cwagent_enabled: true
cwagent_cfg_param_name: /natkit/cwagent # SSM parameter with the agent's JSON config

The instance role also needs CloudWatchAgentServerPolicy and ssm:GetParameter on the parameter.

The agent image is not in the AMI. natkit pulls it by digest from public.ecr.aws at boot, only when the agent is enabled, through the NAT’s own egress. It is about 150 MB unpacked and stays in memory: use at least a t4g.micro or t3.micro.

Hardening

Mon, 01 Jan 0001 00:00:00 +0000

Login: no SSH, SSM agent or package manager. The only login is the EC2 Serial Console. It stays locked unless console_password is set.
Inbound: the instance accepts no new inbound connections to itself. The nftables input policy is drop: replies, ICMP and DHCP only.
Forwarded traffic: not filtered (use the security group), except packets conntrack calls invalid, which would otherwise leave un-NATed.
Credentials: AWS credentials reach curl on stdin, never on a command line.
Kernel: built from allnoconfig, with no USB, wireless, filesystems beyond tmpfs/overlay, /dev/mem, kexec or 32-bit syscalls. CPU mitigations, KASLR, stack protector, hardened usercopy and zeroed allocations are on. Module loading is disabled once Jool is in.
Network stack: no ICMP redirects or source routing. Unprivileged BPF and kernel log access are off.
Capabilities: natkit-init runs with three capabilities. The CloudWatch agent launcher runs with CAP_SYS_ADMIN only: ctr run mounts the agent image’s rootfs, and its containerd socket is root-equivalent anyway.
Supply chain: images are pinned by digest or content hash, and the Jool source by checksum. IMDSv2 only.

natkit and fck-nat

Mon, 01 Jan 0001 00:00:00 +0000

fck-nat	natkit
Amazon Linux 2023, systemd unit `fck-nat.service`	LinuxKit `onboot` container `natkit-init`
`/etc/fck-nat.conf` written by a cloud-init script	YAML user-data with the same keys
`aws` CLI	`curl --aws-sigv4` with the instance role
`iptables -t nat ... MASQUERADE`	one nftables ruleset, applied atomically
Jool from packages	Jool 4.1.15 modules built against natkit’s kernel
CloudWatch agent installed	CloudWatch agent pulled by digest at boot, only when enabled
SSH / SSM	EC2 Serial Console (root password hash from SSM, optional)

natkit-init follows fck-nat’s service/fck-nat.sh (MIT, Andrew Guenther).

Building

Mon, 01 Jan 0001 00:00:00 +0000

Everything builds natively for the Docker daemon’s architecture (arm64 or amd64):

 make jool # Jool modules for natkit.yml's kernel (lk-kernel); or let linuxkit pull them from GHCR
make # out/natkit-<arch>-efi.img
make test # natkit-init against a mock IMDS/EC2/SSM
make ami # needs AWS credentials, AWS_REGION, NATKIT_BUCKET and the vmimport role

CI

.github/workflows/build.yml builds both architectures on every push. Tags (v*) and manual runs also register the AMIs and run scripts/boot-test.sh, which checks:

natkit

Getting started

Configuration

High availability

NAT64

OpenTofu module

Inputs link

Outputs link

Instance role

CloudWatch agent

Hardening

natkit and fck-nat

Building

CI link

Inputs

Outputs

CI