For a model like Meta's Llama 3.1 405B-Instruct-FP8 weighing in at 454 GB across 109 safetensor shards, the answer can range from 5 minutes to over 30 minutes, depending entirely on how you deliver the model weights to the GPU. In autoscaling scenarios, this cold-start time directly impacts how quickly your infrastructure responds to demand spikes.

In this post, we benchmark eight different strategies for loading LLM weights on Amazon Elastic Kubernetes Service (Amazon EKS) and share our findings on performance, cost, and operational trade-offs.

The Cold-Start Problem​

When vLLM (or any model serving framework) starts up on a GPU node, it goes through several phases:

1. Node provisioning — Karpenter provisions a GPU instance and the node joins the cluster
2. Container image pull — The container runtime pulls the serving framework image from a registry
3. Model weight loading — The framework reads model shards from storage and loads them into GPU memory
4. KV cache initialization — The framework allocates GPU memory for inference caching and runs warmup

For large models, step 3 dominates the cold-start time. A 454 GB model must be read from storage and transferred to GPU memory, and the serving framework (vLLM in our case) reads shards sequentially, one at a time. This means the storage throughput for a single sequential read stream directly determines how long the model takes to load.

The question becomes: what's the fastest, most cost-effective way to get 454 GB of model weights into GPU memory?

Benchmark Environment​

We ran all benchmarks on an Amazon EKS cluster using EKS Auto Mode, which automates node provisioning, scaling, and lifecycle management using Karpenter.

Each strategy was tested with a true cold start, a fresh GPU node with no prior image or data cache, unless otherwise noted. We measured cold-start time as the interval from pod scheduled to pod ready (serving inference).

The Eight Strategies​

Strategy 1: HuggingFace Hub Direct Download​

How it works: vLLM downloads model weights directly from the HuggingFace Hub during startup. The hf_transfer library accelerates the download by using multiple concurrent connections. Files are written to the node's local NVMe storage, and vLLM loads shards into GPU memory from there.

This is the simplest approach — no pre-staging, no external storage to manage. Just point vLLM at a model ID and let it pull the weights at startup.

Results:

What we observed: The download phase ran at approximately 500 MB/s, which is reasonable for a cross-internet download. However, in one of our three runs, the download hung indefinitely on the final shard at 418 GB — requiring a full pod restart. This reliability concern makes HuggingFace direct download risky for production workloads at this model scale.

Strategy 2: Amazon FSx for Lustre​

How it works: We pre-cached the model weights on an Amazon FSx for Lustre filesystem — a high-performance parallel filesystem commonly used for HPC and machine learning workloads. The Lustre volume was mounted into the vLLM pod using the FSx CSI driver. vLLM reads shards directly from the mounted filesystem.

We provisioned the filesystem with the PERSISTENT_2 deployment type on SSD storage at the 1000 MB/s/TiB throughput tier — a 4,800 GiB filesystem providing a theoretical aggregate throughput of 4.8 GB/s.

Results:

What we observed: Despite provisioning a filesystem capable of 4.8 GB/s aggregate throughput, we observed only ~260 MB/s during weight loading. This is because FSx for Lustre is architected for parallel, multi-stream I/O across many clients — the kind of workload you see in distributed training. Our workload is fundamentally different: a single client reading shards sequentially. The 1000 MB/s/TiB throughput tier represents aggregate capacity across multiple concurrent streams, not single-stream performance.

At $1,440/month for the filesystem alone, this was also the most expensive strategy we tested.

With Run:ai Model Streamer: We also tested FSx for Lustre with the Run:ai Model Streamer (--load-format runai_streamer, concurrency=32) to see if concurrent tensor streaming could overcome the sequential read bottleneck. The results were dramatic:

The Run:ai streamer improved Lustre throughput by 4.4x — using 32 concurrent I/O threads per GPU worker (256 total across 8 TP workers) to saturate the filesystem's parallel bandwidth. This confirms that Lustre's low default throughput was caused by the sequential access pattern, not a limitation of the filesystem itself. However, even with this improvement, the $1,440/month storage cost makes this a less compelling option compared to S3-based strategies.

Strategy 3: Amazon EFS (Elastic Throughput)​

How it works: We pre-cached the model weights on Amazon Elastic File System (Amazon EFS) using the Elastic throughput mode. The EFS volume was mounted into the vLLM pod via the EFS CSI driver. Like the Lustre strategy, vLLM reads shards sequentially from the mounted filesystem.

EFS Elastic throughput automatically scales read and write throughput based on demand — no need to pre-provision capacity or throughput. We pre-loaded the model onto EFS using s5cmd to copy from an Amazon Simple Storage Service (Amazon S3) staging bucket.

Results:

What we observed: EFS delivered ~1 GB/s sequential read throughput — nearly 4x faster than FSx for Lustre for this workload. EFS Elastic throughput automatically scaled to meet our read demand without any configuration. This was a surprising result: a general-purpose managed file system outperformed a purpose-built HPC filesystem for our specific access pattern.

The trade-off is cost: EFS Elastic throughput charges $0.03 per GB of data read. At 454 GB per cold start, that's approximately $14 per startup event — which adds up at scale.

Strategy 4: Mountpoint for Amazon S3​

How it works: We mounted the Amazon S3 bucket containing the model as a POSIX filesystem using Mountpoint for Amazon S3, accessed through the Mountpoint for Amazon S3 CSI driver. vLLM reads shards through the FUSE mount layer as if they were local files. No data is staged locally — reads go directly to S3.

Results:

What we observed: Mountpoint for Amazon S3 provides a convenient POSIX interface to S3, but the FUSE layer adds overhead for sequential reads. Per-shard latency was a consistent ~12.4 seconds, resulting in over 22 minutes of sequential weight loading for 109 shards.

The key advantage here is cost: S3 Standard storage for 454 GB costs just ~$10/month, making this the cheapest storage option. But the slow sequential read speed means expensive GPU time is wasted waiting for data.

Strategy 5: Amazon S3 + s5cmd Init Container​

How it works: This strategy takes a fundamentally different approach. Instead of reading model weights from network storage during vLLM's sequential loading phase, we use a Kubernetes init container to download all model shards from S3 in parallel to the node's local NVMe storage before vLLM starts.

The init container runs s5cmd, a high-performance S3 client, with --concurrency 50 to download all 109 shards simultaneously. Once the download completes, vLLM starts and loads shards from local NVMe — which provides ~5 GB/s sequential read throughput.

initContainers:
  - name: s3-download
    image: peakcom/s5cmd:v2.2.2
    command: ["/s5cmd", "--log", "error", "cp",
              "--concurrency", "50", "--part-size", "100",
              "s3://llm-model-cache-usw2/meta-llama/Llama-3.1-405B-Instruct-FP8/*",
              "/scratch/model/"]

This approach decouples the parallel download from vLLM's sequential loading, leveraging S3's massive aggregate bandwidth and the node's high-speed local NVMe storage.

Results:

What we observed: This was the fastest strategy by a significant margin. The p5.48xlarge's 3200 Gbps EFA networking enabled s5cmd to pull 454 GB from S3 at ~3.2 GB/s. Once on local NVMe, vLLM loaded all 109 shards in just 92 seconds at ~5 GB/s.

The insight is simple but powerful: downloading in parallel and reading locally is much faster than reading sequentially from network storage. By using an init container, we exploit S3's massive aggregate bandwidth without being constrained by vLLM's single-stream sequential read pattern.

Note: This benchmark used a warm container image cache (the base vLLM image was already on the node from a prior strategy test). Add approximately 1 minute for a cold image pull of the 9.3 GB base image, bringing the estimated true cold start to ~6 min 30 sec.

Strategy 6: Amazon S3 + Run:ai Model Streamer​

How it works: This strategy uses the Run:ai Model Streamer, an open-source library that replaces vLLM's default sequential weight loader with a concurrent tensor streaming engine. Instead of reading safetensor shards one at a time, the streamer uses multiple C++ threads to read tensors from S3 in parallel, overlapping I/O (S3 → CPU memory) with GPU transfer (CPU → GPU).

The key difference from the s5cmd approach: there is no init container and no local copy step. vLLM reads directly from the S3 path using the --load-format runai_streamer flag. The streamer handles all the parallelism internally.

command:
  - python3
  - -m
  - vllm.entrypoints.openai.api_server
  - --model
  - s3://llm-model-cache-usw2/meta-llama/Llama-3.1-405B-Instruct-FP8
  - --load-format
  - runai_streamer

We built a minimal custom container image that adds runai-model-streamer and runai-model-streamer-s3 packages on top of the base vLLM image.

Results:

What we observed: The Run:ai streamer achieved 1.2 GB/s streaming directly from S3 — nearly 4x faster than Mountpoint for Amazon S3 (340 MB/s) for the same S3 bucket. It works at the tensor level (1509 individual tensors across 109 shards), reading and transferring them concurrently rather than waiting for each shard to complete before starting the next.

This approach is the simplest S3-based strategy — no init container, no local NVMe storage, no volume mounts. Just point vLLM at an S3 path. The trade-off is that it's ~3 minutes slower than s5cmd because S3 streaming at 1.2 GB/s can't match the NVMe local read speed of 5 GB/s after a parallel download.

Strategy 7: Amazon ECR Container Image​

How it works: We baked the model weights directly into the container image and stored it in Amazon Elastic Container Registry (Amazon ECR). The 454 GB model was split across 11 OCI image layers (~44 GB each) to comply with Amazon ECR's 49 GB per-layer limit. When the pod starts, containerd pulls all layers in parallel from ECR. vLLM then reads model weights from the local overlay filesystem, which is backed by the node's NVMe storage.

# Each batch downloads 10 shards as a separate layer (~44 GB)
RUN for i in $(seq -w 1 10); do \
      echo "cp ${S3}/model-000${i}-of-00109.safetensors ${D}/"; \
    done | s5cmd --log error run
# ... repeated for batches 2-11

This approach treats the model as an immutable part of the container image — no runtime downloads, no external storage mounts.

Results:

What we observed: Containerd on EKS Auto Mode (Bottlerocket) pulled all 11 model layers in parallel from ECR, downloading 369 GB (compressed) in just over 8 minutes. Once unpacked to the local overlay filesystem, weight loading completed in 67 seconds — the fastest weight loading of any strategy.

The trade-off is operational complexity: you need to build and maintain a ~460 GB container image, the build process requires a machine with 600+ GB of disk, and the ECR layer size limit necessitates splitting the model across multiple Dockerfile RUN commands. Any model update requires rebuilding and re-pushing the entire image.

Strategy 8: Amazon EBS Snapshot​

How it works: We pre-loaded the model weights onto an Amazon EBS volume, created a snapshot, and used Karpenter's blockDeviceMappings to attach the snapshot as a secondary volume on every GPU node at launch. The node's userData script mounts the snapshot volume at /mnt/model-data, and vLLM reads model weights from the host-mounted path via a hostPath volume.

We enabled EBS Fast Snapshot Restore (FSR) on the snapshot in the target Availability Zone to eliminate the lazy-loading penalty that EBS snapshots incur on first read. Without FSR, the first read of each block must be fetched from S3, adding over 100 seconds per shard — making cold starts impractical.

The volume was provisioned as gp3 with 1000 MB/s throughput and 16,000 IOPS — the maximum for gp3. We used the Run:ai Model Streamer (--load-format runai_streamer, concurrency=32) to read tensors concurrently from the mounted volume.

blockDeviceMappings:
  - deviceName: /dev/xvdb
    ebs:
      snapshotID: snap-0db355e5da8895b94
      volumeSize: 750Gi
      volumeType: gp3
      throughput: 1000
      iops: 16000
      encrypted: true
      deleteOnTermination: true

Results:

What we observed: The Run:ai streamer fully saturated the gp3 throughput ceiling at 1003 MiB/s — reading 453.8 GiB across all 8 tensor-parallel workers in 462.9 seconds. For comparison, the default sequential loader on the same gp3 volume achieved only ~367 MB/s (20 min 36 sec), meaning the concurrent streamer delivered a 2.7x speedup by better utilizing the available bandwidth.

Without the Run:ai streamer, vLLM's default sequential loader on the same gp3 volume took 20 min 36 sec at only ~367 MB/s — leaving over 60% of the gp3 bandwidth unused. Without FSR enabled, the first read of each snapshot block triggers a lazy fetch from S3, inflating per-shard load times to over 130 seconds and making cold starts completely impractical.

The key advantage of EBS snapshots is zero download time — model data is available the moment the node boots, with no init container or network transfer required. The trade-off is operational: you need to manage snapshots, enable FSR in each target AZ ($0.75/hr per AZ), and pin your NodePool to snapshot-enabled zones.

To go faster with EBS, you could use io2 Block Express volumes (up to 4000 MB/s throughput), though at significantly higher cost. The gp3 throughput ceiling is the bottleneck here, not the loader.

Results at a Glance​

* Estimated cold start based on warm Lustre cache with Run:ai Model Streamer (concurrency=32). See Strategy 2 for details.

Why the s5cmd Approach Wins​

The results highlight a fundamental architectural insight: the bottleneck isn't storage throughput — it's the sequential access pattern.

vLLM's default loader reads safetensor shards one at a time. When reading from network-attached storage (EFS, Lustre, S3 Mountpoint), each shard becomes a blocking I/O operation. The per-shard latency ranges from 4.4 seconds on EFS to 16.2 seconds on Lustre — and with 109 shards to load, these delays compound to 8–30 minutes of GPU idle time.

The s5cmd, Run:ai streamer, and concurrent loading approaches break this bottleneck in different ways:

s5cmd init container (fastest — 5 min 31 sec): Separates the problem into two phases — parallel download all 109 shards from S3 at ~3.2 GB/s, then vLLM reads sequentially from local NVMe at ~5 GB/s. This two-phase approach turns a 30-minute network I/O problem into a 2-minute download plus a 90-second local read.

Run:ai Model Streamer (simplest — 8 min 28 sec): Replaces vLLM's sequential loader with a concurrent tensor streaming engine that reads directly from S3 at ~1.2 GB/s. No init container, no local storage — just a single flag change. It's 3 minutes slower because S3 streaming can't match NVMe local read speeds, but the operational simplicity is compelling.

Run:ai streamer on filesystems: The concurrent streaming approach also dramatically improved FSx for Lustre — from 260 MB/s to 1.1 GiB/s (4.4x faster). Lustre's parallel architecture was being underutilized by vLLM's sequential reader, and the Run:ai streamer's 256 concurrent threads (32 per GPU worker × 8 workers) were able to saturate the filesystem bandwidth. However, we found that Run:ai streamer did not improve EFS performance (~859 MiB/s vs ~1 GB/s with the default loader), likely because EFS Elastic throughput already delivers optimal single-client bandwidth for sequential reads.

Cost Analysis​

Faster cold starts aren't just about user experience — they directly reduce costs. Every minute a GPU instance spends loading a model is a minute it's not serving inference. On a p5.48xlarge Spot Instance at ~$32/hour, that's $0.53 per minute of wasted compute.

Monthly Cost Comparison (1 cold start/day, Spot pricing)​

The s5cmd strategy delivers the lowest total cost at $98 — combining the cheapest storage option (S3 at $10/month) with the least GPU idle time ($88). FSx for Lustre, despite being the most expensive storage at $1,440/month, also incurs the highest GPU idle cost due to its slow sequential read performance.

A notable observation: Amazon EFS appears cost-competitive at first glance ($136/month storage + $173 GPU idle), but the Elastic throughput read charges of $0.03/GB add approximately $14 per cold start — totaling $409/month in read fees alone at one cold start per day.

Amazon EBS Snapshot storage itself is cheap ($0.05/GB/month = ~$38 for 750 GiB), but the Fast Snapshot Restore (FSR) charge of $0.75/hr per AZ adds $540/month if left enabled continuously. Without FSR, lazy-loading penalties make cold starts impractical — so you must either keep FSR enabled or implement automation to enable it only before scaling events and disable it afterward.

Choosing the Right Strategy​

There is no one-size-fits-all answer. The best strategy depends on your operational priorities:

Optimize for cold-start speed and cost → Amazon S3 + s5cmd Init Container

This came out as top strategy balancing both speed and cost. It delivers the fastest cold start (5 min 31 sec), the lowest monthly cost ($98), and relies only on Amazon S3 — a service you're likely already using. The init container pattern is straightforward to implement and doesn't require any specialized storage infrastructure.

Best practices for this approach:

• Use s5cmd's --concurrency 50 --part-size 100 flags to maximize parallelism
• Ensure your pod's service account has IAM permissions to read from the S3 bucket

Optimize for simplicity → Amazon S3 + Run:ai Model Streamer

If you want the simplest possible setup with no init containers, no volume mounts, and no local storage management, the Run:ai Model Streamer is an excellent choice. At 8 min 28 sec, it's faster than EFS and ECR while requiring nothing beyond a custom vLLM image with the streamer packages installed. Just point --model at an S3 path and set --load-format runai_streamer.

Optimize for zero-download startup → Amazon EBS Snapshot

If eliminating the download phase entirely is a priority, EBS snapshots deliver model data at node boot time — no init container, no network transfer. Combined with the Run:ai Model Streamer, the snapshot volume saturates gp3 bandwidth at ~1003 MiB/s for an 11-minute cold start. However, this approach carries the most operational overhead of any strategy:

• Snapshot lifecycle management: You must create and maintain EBS snapshots for each model version. Updating a model means creating a new snapshot and updating the Karpenter EC2NodeClass with the new snapshot ID.
• Fast Snapshot Restore (FSR): FSR must be enabled in every target Availability Zone before nodes launch. At $0.75/hr per AZ, this costs $540/month if left on continuously. Consider automating FSR enable/disable around scaling events to reduce cost.
• AZ pinning: Since FSR is per-AZ, your NodePool must be pinned to specific zones where FSR is enabled, reducing scheduling flexibility and Spot availability.
• Non-standard AMI requirements: EBS snapshots require userData scripting to mount the volume, which means using AL2023 (not Bottlerocket) and managing NVIDIA device plugin installation separately.
• NVMe device detection: On Nitro instances, NVMe device ordering is non-deterministic. The userData script must detect the snapshot volume by filesystem type (e.g., blkid for ext4) rather than hardcoding device names.

This strategy is best suited for environments where download-free startup is a hard requirement and you have the operational maturity to manage snapshot lifecycles and FSR automation.

Optimize for deployment immutability → Amazon ECR Container Image

If you value the container-as-deployment-unit model — where the container image is the single artifact that contains everything needed to run — baking the model into the image is compelling. There are no runtime dependencies on external storage, and rollbacks are as simple as reverting to a prior image tag.

Consider this when:

• You need strict reproducibility across environments
• Your CI/CD pipeline can handle building and pushing ~460 GB images
• Model updates are infrequent

Optimize for shared access across pods → Amazon EFS

When multiple pods or services need to read the same model weights — for example, running different model versions side by side, or sharing a model across development and staging environments — EFS provides a familiar ReadWriteMany filesystem that simplifies the architecture. Just be aware of the Elastic throughput read charges at scale.

Conclusion​

Cold-start time is a critical but often overlooked factor in LLM serving architectures. Our benchmarks show a 6x difference between the fastest and slowest strategies — from 5 minutes with the s5cmd init container approach to 32 minutes with FSx for Lustre. The Run:ai Model Streamer emerged as a compelling middle ground, delivering 8-minute cold starts with the simplest possible configuration.

The key insight is that how you deliver model weights matters more than the raw throughput of your storage system. A high-throughput parallel filesystem like FSx for Lustre can underperform a simple S3 download when the access pattern is single-client sequential reads — but applying concurrent loading (via the Run:ai Model Streamer) to that same Lustre filesystem improved throughput by 4.4x. EBS snapshots with the Run:ai streamer similarly saturated the gp3 bandwidth ceiling (1003 MiB/s) where the default loader achieved only 367 MB/s — a 2.7x improvement. By rethinking the loading architecture — whether downloading in parallel to fast local storage or using concurrent tensor streaming — we achieved cold-start times that make GPU autoscaling practical for even the largest open-source models.

All Kubernetes manifests, Dockerfiles, and detailed benchmark data from this post are available in our GitHub repository.

Ashok Srirama is a Principal Solutions Architect at Amazon Web Services, based in Washington Crossing, PA. He specializes in serverless applications, containers, and architecting distributed systems. When he's not spending time with his family, he enjoys watching cricket, and driving his bimmer.

In this approach, the entire workflow is executed in the spoke cluster. argo-workflow-controller deployed to the hub cluster watches for workflow CRD changes in the spoke cluster and launches k8s pods to execute the various steps.

Prerequisites​

• Install the eksctl
• 2 AWS Accounts
• AWS CLI profiles setup for both AWS Accounts
• Setup the env variables

export AWS_REGION=us-west-2
export EKS_HUB_CLUSTER=ekshub
export EKS_SPOKE_CLUSTER=eksSPOKE
export EKS_HUB_ACCOUNT=112233445566
export EKS_SPOKE_ACCOUNT=998877665544

Step 1: Create EKS Cluster in the hub AWS Account​

Create a hub EKS cluster with default configuration

eksctl create cluster --name $EKS_HUB_CLUSTER --region $AWS_REGION --profile hub-profile
.......
[✔]  EKS cluster "ekshub" in "us-west-2" region is ready

aws eks update-kubeconfig --name $EKS_HUB_CLUSTER --region $AWS_REGION --alias $EKS_HUB_CLUSTER --profile hub-profile

Step 2: Create EKS Cluster in the spoke AWS Account​

Create a spoke EKS cluster with default configuration in same or different AWS Account

eksctl create cluster --name $EKS_SPOKE_CLUSTER --region $AWS_REGION --profile spoke-profile
.......
[✔]  EKS cluster "eksspoke" in "us-west-2" region is ready

aws eks update-kubeconfig --name $EKS_SPOKE_CLUSTER --region $AWS_REGION --alias $EKS_SPOKE_CLUSTER --profile spoke-profile

Wait for the cluster messages as shown above and proceed with next steps.

Step 3: Create custom images of Argo Workflows and Argo Server as they require AWS CLI to interact with spoke EKS clusters​

Create a new Argo Workflow container image with AWS CLI

cat << EOF > DockerfileArgow
FROM quay.io/argoproj/workflow-controller:v3.5.5 as argow

FROM public.ecr.aws/aws-cli/aws-cli:2.15.33

COPY --from=argow /etc/ssh/ssh_known_hosts /etc/ssh/
COPY --from=argow /etc/nsswitch.conf /etc/

COPY --from=argow /bin/workflow-controller /bin/

ENTRYPOINT [ "workflow-controller" ]
EOF

docker build -t sriram430/workflow-controller-aws:v3.5.5 -f DockerfileArgow .

docker push sriram430/workflow-controller-aws:v3.5.5

Create a new Argo Server container image with AWS CLI

cat << EOF > DockerfileArgoServer
FROM quay.io/argoproj/argocli:v3.5.5 as argocli

FROM public.ecr.aws/aws-cli/aws-cli:2.15.33

COPY --from=argocli /etc/ssh/ssh_known_hosts /etc/ssh/
COPY --from=argocli /etc/nsswitch.conf /etc/

COPY --from=argocli /bin/argo /bin/

ENTRYPOINT [ "argo" ]
EOF

docker build -t sriram430/argocli-aws:v3.5.5 -f DockerfileArgoServer .

docker push sriram430/argocli-aws:v3.5.5

Step 4: Create an IAM role for the hub cluster to authenticate with spoke cluster​

aws iam create-policy \
    --policy-name AmazonEKSReadPolicy \
    --profile hub-profile \
    --policy-document \
'{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "eks:AccessKubernetesApi",
                "eks:DescribeCluster"
            ],
            "Resource": "*"
        }
    ]
}'

eksctl create addon --name eks-pod-identity-agent \
--cluster $EKS_HUB_CLUSTER --region $AWS_REGION \
--profile hub-profile

eksctl create podidentityassociation \
--cluster $EKS_HUB_CLUSTER \
--namespace argo \
--service-account-name argo \
--role-name eks-hub-cluster-role \
--permission-policy-arns arn:aws:iam::${EKS_HUB_ACCOUNT}:policy/AmazonEKSReadPolicy \
--region $AWS_REGION \
--profile hub-profile

eksctl create podidentityassociation \
--cluster $EKS_HUB_CLUSTER \
--namespace argo \
--service-account-name argo-server \
--role-arn arn:aws:iam::${EKS_HUB_ACCOUNT}:role/eks-hub-cluster-role \
--region $AWS_REGION \
--profile hub-profile

Step 5: Create a cluster access entry for the Hub Cluster's IAM role in the spoke EKS cluster​

For testing purposes we are using AmazonEKSClusterAdminPolicy, scope down these permissions based on argo-workflow requirements.

aws eks create-access-entry --cluster-name $EKS_SPOKE_CLUSTER --principal-arn arn:aws:iam::${EKS_HUB_ACCOUNT}:role/eks-hub-cluster-role \
    --profile spoke-profile --region $AWS_REGION

aws eks associate-access-policy --cluster-name $EKS_SPOKE_CLUSTER --principal-arn arn:aws:iam::${EKS_HUB_ACCOUNT}:role/eks-hub-cluster-role \
    --access-scope type=cluster --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
    --profile spoke-profile --region $AWS_REGION

Step 6: Create the Argo Workflows CRDs, Service Account, Roles, Role Bindings in spoke EKS cluster​

kubectl apply -f https://raw.githubusercontent.com/ashoksrirama/argo-workflows-demo/main/argo-workflows-spoke.yaml --context $EKS_SPOKE_CLUSTER

Step 7: Create a k8s secret in hub cluster with kubeconfig file of spoke cluster​

# Create the kubeconfig file of the spoke cluster
aws eks update-kubeconfig --name $EKS_SPOKE_CLUSTER --region $AWS_REGION --profile spoke-profile --kubeconfig spoke-kubeconfig

export KUBECONFIG_SPOKE=$(head -n -3 spoke-kubeconfig) #If fails, replace head with ghead

# Create namespace
kubectl create namespace argo --context $EKS_HUB_CLUSTER

# Create k8s secret with spoke cluster's kubeconfig
kubectl create secret generic spoke-kubeconfig --from-literal kubeconfig=$KUBECONFIG_SPOKE -n argo --context $EKS_HUB_CLUSTER

Step 8: Install the Argo Workflow Controller and Argo Server in the Hub EKS cluster​

# Replace ARGO_WORKFLOWS_VERSION with latest Argo Workflows version
kubectl apply -f https://github.com/argoproj/argo-workflows/releases/download/v<<ARGO_WORKFLOWS_VERSION>>/install.yaml --context $EKS_HUB_CLUSTER

Step 9: Patch argo-server authentication​

cat << EOF > argo-server-patch.yaml
spec:
  template:
    spec:
      containers:
      - name: argo-server
        image: sriram430/argocli-aws:v3.5.5 #replace with your image
        args:
        - server
        - '--auth-mode=server'
        - '--loglevel=debug' #setup loglevel
        - '--kubeconfig=/kube/config/kubeconfig'
        volumeMounts:
        - name: spoke-kubeconfig
          mountPath: /kube/config
        securityContext:
          capabilities:
          runAsNonRoot: false
          readOnlyRootFilesystem: false
          allowPrivilegeEscalation: false
      volumes:
      - name: spoke-kubeconfig
        secret:
          secretName: spoke-kubeconfig
EOF

kubectl patch deployment -n argo argo-server --patch-file argo-server-patch.yaml --context $EKS_HUB_CLUSTER

Step 10: Patch argo-workflow controller​

cat << EOF > workflow-controller-patch.yaml
spec:
  template:
    spec:
      containers:
      - name: workflow-controller
        image: sriram430/workflow-controller-aws:v3.5.5 #replace with your image
        command:
        - workflow-controller
        - '--loglevel=debug' #setup loglevel
        - '--kubeconfig=/kube/config/kubeconfig'
        volumeMounts:
        - name: spoke-kubeconfig
          mountPath: /kube/config
        securityContext:
          capabilities:
          runAsNonRoot: false
          readOnlyRootFilesystem: false
          allowPrivilegeEscalation: false
      volumes:
      - name: spoke-kubeconfig
        secret:
          secretName: spoke-kubeconfig
EOF

kubectl patch deployment -n argo workflow-controller --patch-file workflow-controller-patch.yaml --context $EKS_HUB_CLUSTER

Step 11: Confirm the pods are up and running​

kubectl get pods -n argo --context $EKS_HUB_CLUSTER

NAME                                  READY   STATUS    RESTARTS   AGE
argo-server-56547dc758-842v5          1/1     Running   0          4m13s
workflow-controller-c699d6f49-9vc5j   1/1     Running   0          108s

Step 12: Submit a sample workflow in Spoke EKS Cluster​

cat << EOF > hello-workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow                  # new type of k8s spec
metadata:
  generateName: hello-world-    # name of the workflow spec
spec:
  entrypoint: whalesay          # invoke the whalesay template
  templates:
    - name: whalesay              # name of the template
      container:
        image: docker/whalesay
        command: [ cowsay ]
        args: [ "hello world" ]
        resources: # limit the resources
          limits:
            memory: 32Mi
            cpu: 100m
EOF

kubectl create -f hello-workflow.yaml --context $EKS_SPOKE_CLUSTER

Step 13: Verify the workflow execution​

kubectl get pods --context $EKS_SPOKE_CLUSTER

NAME                READY   STATUS      RESTARTS   AGE
hello-world-klhjq   0/2     Completed   0          97s

We completed the walkthrough of running argo-workflow controller in Hub EKS cluster and actual workflow executions in a spoke EKS cluster.

Prerequisites​

• Install the eksctl

Create a basic cluster with default configuration

eksctl create cluster --name eksdemo --region us-west-2
.......
[✔]  EKS cluster "eksdemo" in "us-west-2" region is ready

Wait for the cluster message as shown above and export the following env variables.

export AWS_REGION=us-west-2
export EKS_CLUSTER=eksdemo

Step 1: Create MostAllocated kube-scheduler configuration​

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  my-scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    profiles:
    - pluginConfig:
      - args:
          scoringStrategy:
            resources:
            # Update these weights according to your requirements
            - name: cpu
              weight: 1
            - name: memory
              weight: 1
            # Considering GPU resources in utilization calculation
            - name: nvidia.com/gpu
              weight: 1
            type: MostAllocated
        name: NodeResourcesFit
      plugins:
        score:
          enabled:
          - name: NodeResourcesFit
            weight: 1
          disabled:
          # Disabling these plugins to avoid spreading the pods
          - name: NodeResourcesBalancedAllocation
          - name: PodTopologySpread
      schedulerName: my-scheduler
    # Enabling leaderElection require additional permissions to the SA
    leaderElection:
      leaderElect: true
      resourceNamespace: kube-system    
      resourceName: my-scheduler
kind: ConfigMap
metadata:
  name: most-allocated-scheduler-policy
  namespace: kube-system
EOF

configmap/most-allocated-scheduler-policy created

Step 2: Create ServiceAccount and ClusterRolebindings for the custom scheduler​

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-scheduler
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-scheduler-as-kube-scheduler
subjects:
- kind: ServiceAccount
  name: my-scheduler
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:kube-scheduler
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-scheduler-as-volume-scheduler
subjects:
- kind: ServiceAccount
  name: my-scheduler
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:volume-scheduler
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-scheduler-extension-apiserver-authentication-reader
  namespace: kube-system
roleRef:
  kind: Role
  name: extension-apiserver-authentication-reader
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: my-scheduler
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: my-scheduler
rules:
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
  - get
  - list
  - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: my-scheduler
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: my-scheduler
subjects:
- kind: ServiceAccount
  name: my-scheduler
  namespace: kube-system
EOF

serviceaccount/my-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-kube-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-volume-scheduler created
rolebinding.rbac.authorization.k8s.io/my-scheduler-extension-apiserver-authentication-reader created
clusterrole.rbac.authorization.k8s.io/my-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler created

Step 3: Deploy the custom scheduler using above configuration​

Make sure your kube-scheduler image matches your EKS cluster version

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: scheduler
    tier: control-plane
  name: my-scheduler
  namespace: kube-system
spec:
  selector:
    matchLabels:
      component: scheduler
      tier: control-plane
  replicas: 1
  template:
    metadata:
      labels:
        component: scheduler
        tier: control-plane
        version: second
    spec:
      serviceAccountName: my-scheduler
      containers:
      - command:
        - /usr/local/bin/kube-scheduler
        args:
        - --config=/etc/kubernetes/my-scheduler/my-scheduler-config.yaml
        - --leader-elect=true
        # log verbose level
        - --v=4
        # update the image based on your EKS version
        image: registry.k8s.io/kube-scheduler:v1.28.5
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10259
            scheme: HTTPS
          initialDelaySeconds: 15
        name: kube-second-scheduler
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10259
            scheme: HTTPS
        resources:
          requests:
            cpu: '0.1'
        securityContext:
          privileged: false
        volumeMounts:
          - name: config-volume
            mountPath: /etc/kubernetes/my-scheduler
      hostNetwork: false
      hostPID: false
      volumes:
        - name: config-volume
          configMap:
            name: most-allocated-scheduler-policy
EOF

deployment.apps/my-scheduler created

Step 4: Deploy a sample application​

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  strategy: {}
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
      schedulerName: my-scheduler
EOF

deployment.apps/nginx created

nginx pods will be scheduled to a node with higher utilization based on the scoring strategy defined in the Scheduler Configuration.

Prerequisites​

• Install the eksctl

Create a basic cluster with default configuration

eksctl create cluster --name eksdemo --region us-west-2
.......
[✔]  EKS cluster "eksdemo" in "us-west-2" region is ready

Wait for the cluster message as shown above and export the following env variables.

export AWS_REGION=us-west-2
export EKS_CLUSTER=eksdemo

Step 1: Install AWS LB Controller to create AWS ELB resources for k8s ServiceType LoadBalancer and Ingress resources​

Follow the instructions here to install the AWS LB controller addon on the EKS cluster.

Validate the deployment after the installation

kubectl get deployment -n kube-system aws-load-balancer-controller

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
aws-load-balancer-controller   2/2     2            2           84s

Step 2: Install the nginx-ingress controller​

We are installing the nginx-ingress controller on the cluster and exposing it internally via ClusterIP service.

helm install my-nginx ingress-nginx/ingress-nginx \
--namespace nginx-ingress \
--set controller.service.type=ClusterIP \
--create-namespace

Step 3: Expose the nginx-ingress controller via AWS Application LoadBalancer​

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: nginx-ingress
  name: ingress-nginx-controller
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=10
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: my-nginx-ingress-nginx-controller
              port:
                number: 80

EOF

wait for the ALB creation, nginx-ingress pod registration to complete. And grab the ALB dns name from the below command:

kubectl get ingress -n nginx-ingress

NAME                       CLASS   HOSTS   ADDRESS                                                                   PORTS   AGE
ingress-nginx-controller   alb     *       k8s-nginxing-ingressn-af3887fb52-1426628659.us-west-2.elb.amazonaws.com   80      2m

Step 4: Deploy sample application and exposed via nginx ingress​

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1 
kind: Deployment 
metadata:
  name: httpd-app
  namespace: default 
spec:
  replicas: 2
  selector:
    matchLabels:
      app: httpd-app
  template:
    metadata:
      labels:
        app: httpd-app
    spec:
      containers:
      - name: httpd-app
        image: httpd:latest
        ports:
        - containerPort: 80 

--- 
apiVersion: v1 
kind: Service 
metadata:
  name: httpd-app-svc
  namespace: default
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: httpd-app

---
apiVersion: networking.k8s.io/v1 
kind: Ingress 
metadata:
  name: httpd-app-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - http:
        paths:
        - backend:
            service:
              name: httpd-app-svc
              port:
                number: 80
          path: /httpd
          pathType: Prefix
EOF

deployment.apps/httpd-app created
service/httpd-app-svc created
ingress.networking.k8s.io/httpd-app-ingress created

Step 5: Test the setup

Access the sample httpd application using ALB Ingress url (from Step 3)

curl http://k8s-nginxing-ingressn-af3887fb52-1426628659.us-west-2.elb.amazonaws.com/httpd

<html><body><h1>It works!</h1></body></html>

We successfully validated two ingress controllers setup in our Amazon EKS cluster.