For on-premises node should we buy a VPS or a dedicated server that provides us with a private IP CIDR? Do VPS providers even offer the private IP feature?

[arjunthazhath2001]

I'm trying to set up a hybrid EKS cluster using a VPS I purchased from SSD Nodes, but I'm facing critical issues related to private IP networking and CNI stability. Here's a detailed breakdown:

---

📌 Setup Context

• I am building a hybrid EKS setup, where my on-prem node (VPS) should join an Amazon EKS control plane.

• As part of the cluster creation, EKS requires two non-overlapping private CIDRs:

  • remoteNodeNetworkCIDR
  • remotePodNetworkCIDR

• I purchased a VPS from SSD Nodes, which provided a dedicated public IP, but no private IP.

---

🚨 Issue Faced

• When I run nodeadm init -c file://nodeConfig.yaml, I receive the error:

  The IP address "X.X.X.X" is not within the configured remoteNodeNetworkCIDR.
  

  (Where X.X.X.X is the public IP of the VPS.)

• To bypass this, I ran:

  nodeadm init --skip-node-validation
  

  This did register the node with the EKS control plane successfully.

---

🧪 What I Tried

1. Manually assigned a private IP (from the remoteNodeNetworkCIDR range) to the VPS using ip addr add, but the system still uses the public IP for outbound traffic and cluster communication.
2. Set up a site-to-site VPN using StrongSwan; both tunnel endpoints show "UP" status.
3. Installed various CNIs (Cilium, Calico, Flannel). However, the pods keep crashing and restarting.
4. Only kube-proxy is in a READY state; CoreDNS fails to come up due to CNI failures.

---

🤔 Root Cause Suspected

It appears the issue is that the VPS lacks proper private IP support. Even after assigning one manually, the node continues to use its public IP, which is incompatible with the expectations of hybrid EKS networking (specifically the VPN and pod routing logic). This is likely why the CNIs fail.

---

❓ My Questions

1. Did I make the wrong choice by purchasing a VPS instead of a true dedicated server that provides private IP ranges?

2. Is there any way to configure a VPS to use custom private IPs properly for internal routing and communication?

3. If not, should I purchase from a provider that offers true private IP allocation (RFC1918 range) and allows proper control over networking?

   • If yes, could you recommend such providers?

4. Despite VPN tunnel status being "UP", is it possible that the lack of private IP support on this VPS is the core reason why CNI pods are failing?

---

📌 Summary

Although I could register the node using --skip-node-validation, CNI pods crash due to networking/routing issues, likely caused by the node’s reliance on its public IP in a setup that expects private CIDR-based routing. I'm trying to understand whether this is a fundamental limitation of the VPS I’m using, or if it can be resolved via configuration.

Would appreciate your guidance.

TIRED OF HAVING MULTIPLE CALLS WITH AWS SUPPORT TEAM. They themselves are not aware of how this thing works.

[abhinavmpandey08]

Hi @arjunthazhath2001, thanks for reaching out! Just to confirm, does your node not have any private IPs and is publicly routable through the internet?

[abhinavmpandey08]

Based on Prerequisite setup for hybrid nodes

To use Amazon EKS Hybrid Nodes, you must have private connectivity from your on-premises environment to/from AWS, bare metal servers or virtual machines with a supported operating system, and AWS IAM Roles Anywhere or AWS Systems Manager (SSM) hybrid activations configured. You are responsible for managing these prerequisites throughout the hybrid nodes lifecycle.

I would recommend setting up private network connectivity between your VM and your cluster's VPC.
Once you configure this networking requirement, you can specify the IP for the node to join the cluster with using kubelet flags in the nodeadm config.
Here's an example of how you can achieve that:

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  kubelet:
    flags:
    - "--node-ip=<node-ip>"

[arjunthazhath2001]

NO @abhinavmpandey08 this doesnt work. I tried it out.

BTW for your previous question, our VPS (SSD nodes) has been assigned a public IP by the seller. So yes its routable thru internet. But the provider has not assigned a private IP. I tried to assign a private IP myself for my VPS but still my node is communicating with the control plane using its public IP and ignoring the pirvate IP I [set.](

)

1.I have confirmed that private IP of our on-prem server is the issue. A on-prem VPS/ dedicateD server provider should provide a private CIDR for their clients, some providers even allow us to set a private CIDR ourselves.

2. I have mailed VULTR, DIGITAL OCEAN, LINODE sales teams and have scheduled a couple of calls on Monday to confirm the private IP CIDR feature and their pricing.

3. Although I was able to get the on-prem node registered to the control plane using --skip-node-validation (flag), its just a temporary fix to make the control plane to ignore what the private IP of the on premises node is, but I am getting caught when its time for the kubelet(on premises) to communicate with the API server on AWS control plane. The prime reason why CILIUM/CALICO/FLANNEL pods are crashing thus not being able to setup CNI.