To NAT or Not to NAT? Multus CNI for Telco Workloads on Kubernetes

sequenceDiagram
    participant ClientA as Client A<br/>(Behind NAT-A)
    participant NAT_A as NAT-A
    participant Internet as Internet
    participant NAT_B as NAT-B
    participant ClientB as Client B<br/>(Behind NAT-B)
    
    Note over ClientA,ClientB: Both clients have private IPs.<br/>Neither can receive unsolicited inbound traffic.
    
    ClientA->>NAT_A: UDP from 10.0.0.5:5000
    NAT_A->>Internet: UDP from 203.0.113.10:53801 (mapped)
    Note over Internet: NAT-A opened port 53801,<br/>but NAT-B doesn't know about it
    
    ClientB->>NAT_B: UDP from 192.168.1.5:5000  
    NAT_B->>Internet: UDP from 198.51.100.20:49152 (mapped)
    Note over Internet: NAT-B opened port 49152,<br/>but NAT-A doesn't know about it
    
    Note over ClientA,ClientB: ❌ Neither NAT will accept<br/>unsolicited inbound packets!

sequenceDiagram
    participant P as Pod (10.0.1.5:5060)
    participant N as K8s Node / NAT
    participant I as Server on Internet
    
    Note over P: App binds UDP 5060<br/>SDP says: "Reach me at 10.0.1.5:5060"
    
    P->>N: SIP INVITE (Src: 10.0.1.5:5060)
    Note over N: SNAT to Node IP + Random Port
    N->>I: SIP INVITE (Src: 1.2.3.4:34982)
    
    Note over I: Server reads SDP body.<br/>Tries to send media to 10.0.1.5:5060<br/>(Private IP - Unreachable!)
    I--xP: RTP Media Dropped ❌
    
    Note over I: Server reads packet header.<br/>Replies to 1.2.3.4:34982
    I->>N: SIP 200 OK
    N->>P: SIP 200 OK (mapped back)
    
    Note over P,I: Signaling works, but media fails.<br/>Result: "1-way audio" or complete silence.

sequenceDiagram
    participant CA as Client A
    participant STUN as STUN Server
    participant TURN as TURN Server
    participant CB as Client B

    Note over CA,CB: Step 1: STUN Discovery
    
    CA->>STUN: What's my public IP:port?
    STUN-->>CA: You appear as 203.0.113.10:53801
    
    CB->>STUN: What's my public IP:port?
    STUN-->>CB: You appear as 198.51.100.5:49152
    
    Note over CA,CB: Step 2: Direct Connection Attempt
    
    CA->>CB: Try P2P using discovered addresses
    CB->>CA: Try P2P using discovered addresses
    
    Note over CA,CB: ❌ Often fails with Symmetric NAT<br/>(port changes per destination)
    
    Note over CA,CB: Step 3: TURN Relay (Fallback)
    
    CA->>TURN: Allocate relay address
    TURN-->>CA: Your relay: 54.1.2.3:12345
    
    CB->>TURN: Allocate relay address
    TURN-->>CB: Your relay: 54.1.2.3:12346
    
    rect rgb(255, 230, 230)
        Note over CA,CB: All media now relayed through TURN
        CA->>TURN: Media traffic
        TURN->>CB: Relayed media
        CB->>TURN: Media traffic
        TURN->>CA: Relayed media
    end

sequenceDiagram
    participant Pod as Pod<br/>10.0.1.50
    participant KP as kube-proxy<br/>(MASQUERADE)
    participant Node as Node<br/>10.0.1.10
    participant NAT as NAT Gateway<br/>54.1.2.3
    participant IGW as Internet Gateway
    participant Peer as Remote Peer

    Note over Pod,Peer: Outbound Connection from EKS Pod (Private Subnet)
    
    Pod->>KP: src: 10.0.1.50:40000<br/>dst: Remote Peer
    
    Note over KP: SNAT (MASQUERADE)<br/>Port randomized!
    
    KP->>Node: src: 10.0.1.10:53801<br/>dst: Remote Peer
    
    Node->>NAT: src: 10.0.1.10:53801<br/>dst: Remote Peer
    
    Note over NAT: SNAT<br/>Port randomized again!
    
    NAT->>IGW: src: 54.1.2.3:49152<br/>dst: Remote Peer
    
    IGW->>Peer: src: 54.1.2.3:49152<br/>dst: Remote Peer
    
    Note over Pod,Peer: ⚠️ Original port 40000 is lost after double NAT

flowchart LR
    subgraph "VPC"
        subgraph "Private Subnet"
            Node[Worker Node<br/>Primary ENI]
        end
        subgraph "Public Subnet"
            SecENI[Secondary ENI]
        end
        IGW[Internet Gateway]
        RT[Route Table<br/>0.0.0.0/0 → IGW]
    end
    
    Node -- "eth0<br/>(K8s traffic)" --> PrivateRT[Private Route Table]
    SecENI -- "net1<br/>(Telecom traffic)" --> RT
    RT --> IGW
    
    Pod[Pod] -- "eth0" --> Node
    Pod -- "net1" --> SecENI
    
    SecENI <--> EIP[Elastic IP]
    EIP <--> IGW
    IGW <--> Internet[Internet]

flowchart TB
    subgraph "CNI Plugin Chain"
        Kubelet[Kubelet] --> Multus[Multus<br/>Meta-Plugin]
        Multus -->|"Primary: eth0"| VPCCNI[Amazon VPC CNI]
        Multus -->|"Secondary: net1"| IPVLAN[ipvlan Plugin]
        IPVLAN --> Whereabouts[Whereabouts IPAM]
    end
    
    subgraph "Pod Network Namespace"
        eth0_result[eth0<br/>K8s control plane]
        net1_result[net1<br/>Telecom data plane]
    end
    
    VPCCNI --> eth0_result
    IPVLAN --> net1_result

flowchart TB
    subgraph "Same Subnet: 10.0.2.0/24"
        DHCP[VPC DHCP<br/>assigns to ENIs] 
        WA[Whereabouts<br/>assigns to Pods]
        
        DHCP -->|"Could assign"| IP1[10.0.2.50]
        WA -->|"Could assign"| IP1
        
        Conflict[💥 IP Conflict!<br/>Two entities, same IP]
    end

flowchart LR
    subgraph "Public Subnet: 10.0.2.0/24"
        subgraph "DHCP Managed: .1 - .127"
            ENI1[Node ENI<br/>10.0.2.10]
            ENI2[Node ENI<br/>10.0.2.11]
        end
        
        subgraph "Reserved for Whereabouts: .128 - .191"
            Pod1[Pod net1<br/>10.0.2.128]
            Pod2[Pod net1<br/>10.0.2.129]
            Pod3[Pod net1<br/>10.0.2.130]
        end
        
        subgraph "Future Use: .192 - .254"
            Reserved[Available]
        end
    end

sequenceDiagram
    participant Peer as Remote Peer
    participant IGW as Internet Gateway
    participant ENI as Secondary ENI
    participant Pod as Pod (net1)
    
    Peer->>IGW: UDP to 203.0.113.50:5060
    Note over IGW: DNAT: 203.0.113.50 → 10.0.2.128
    IGW->>ENI: UDP to 10.0.2.128:5060
    ENI->>Pod: Delivered to net1
    Note over Pod: App sees dst=10.0.2.128:5060<br/>(Private IP after DNAT)

sequenceDiagram
    participant Pod as Pod (net1)
    participant ENI as Secondary ENI
    participant IGW as Internet Gateway
    participant Peer as Remote Peer
    
    Pod->>ENI: UDP from 10.0.2.128:5060
    Note over ENI: Source-based routing<br/>directs to secondary ENI
    ENI->>IGW: UDP from 10.0.2.128:5060
    Note over IGW: SNAT: 10.0.2.128 → 203.0.113.50
    IGW->>Peer: UDP from 203.0.113.50:5060
    Note over Peer: Sees src=203.0.113.50:5060 ✓<br/>Matches advertised address!

flowchart TB
    subgraph "Pod with Two Interfaces"
        App[App receives packet<br/>on net1, sends reply]
        eth0[eth0<br/>10.0.1.50<br/>default route]
        net1[net1<br/>10.0.2.128]
    end
    
    subgraph "Without Source Routing"
        App -->|"Reply from 10.0.2.128"| Decision{Routing<br/>Decision}
        Decision -->|"Default route<br/>says use eth0"| eth0
        eth0 -->|"Packet exits eth0<br/>with src=10.0.2.128"| Wrong[❌ Wrong interface!<br/>Packet dropped or<br/>asymmetric routing breaks NAT]
    end

flowchart TB
    Packet[Outbound Packet<br/>src: 10.0.2.128]
    
    Rule1{ip rule check:<br/>from 10.0.2.128?}
    Packet --> Rule1
    
    Rule1 -->|"Yes"| Table200[Use table 'telecom']
    Rule1 -->|"No"| MainTable[Use table 'main']
    
    Table200 --> Route200[default via 10.0.2.1<br/>dev net1]
    Route200 --> net1_out[net1 → Secondary ENI]
    net1_out --> Correct[✓ Correct path!<br/>IGW applies EIP NAT]
    
    MainTable --> RouteMain[default via 10.0.1.1<br/>dev eth0]
    RouteMain --> eth0_out[eth0 → Primary ENI]
    eth0_out --> Wrong[❌ Internal network]
    
    style Table200 fill:#9f9
    style Correct fill:#9f9
    style Wrong fill:#f66

flowchart TB
    subgraph "Without Floating IP"
        InternalClient[Internal Service<br/>10.0.1.100]
        Pod[Pod<br/>net1: 10.0.2.128<br/>EIP: 203.0.113.50]
        
        InternalClient -->|"Connect to<br/>203.0.113.50:5060"| VPCRouter{VPC Router}
        VPCRouter -->|"No route to<br/>203.0.113.50/32"| Dropped[❌ Dropped or<br/>sent to IGW unnecessarily]
    end

flowchart TB
    subgraph "With Floating IP Route"
        InternalClient[Internal Service<br/>10.0.1.100]
        Pod[Pod<br/>net1: 10.0.2.128<br/>+ 203.0.113.50/32 local]
        
        InternalClient -->|"Connect to<br/>203.0.113.50:5060"| VPCRouter{VPC Router}
        VPCRouter -->|"/32 route → ENI"| ENI[Secondary ENI]
        ENI -->|"Direct delivery"| Pod
        
        Note[✓ Pod receives packet<br/>dst still = 203.0.113.50]
    end
    
    style Pod fill:#9f9

#!/usr/bin/env python3
"""
Floating IP Manager Sidecar

Manages EIP association and VPC route table entries for pod lifetime.
Cleanup is guaranteed by Kubernetes native sidecar lifecycle.
"""

import boto3
import signal
import sys
import subprocess
import os
import time

ec2 = boto3.client('ec2')

# State for cleanup
state = {
    'public_ip': None,
    'allocation_id': None,
    'association_id': None,
    'eni_id': None,
    'route_table_ids': [],
}


def get_interface_ip(interface: str) -> str:
    """Get IP address assigned to interface by Whereabouts."""
    result = subprocess.run(
        ['ip', '-4', '-j', 'addr', 'show', interface],
        capture_output=True, text=True
    )
    import json
    data = json.loads(result.stdout)
    return data[0]['addr_info'][0]['local']


def find_eni_by_ip(private_ip: str) -> str:
    """Find ENI ID that has this private IP."""
    resp = ec2.describe_network_interfaces(
        Filters=[{'Name': 'addresses.private-ip-address', 'Values': [private_ip]}]
    )
    return resp['NetworkInterfaces'][0]['NetworkInterfaceId']


def allocate_eip(pool_tag: str) -> dict:
    """Find available EIP from tagged pool or allocate new one."""
    # Try to find unassociated EIP with matching tag
    resp = ec2.describe_addresses(
        Filters=[
            {'Name': 'tag:Pool', 'Values': [pool_tag]},
            {'Name': 'association-id', 'Values': ['']}  # Unassociated
        ]
    )
    if resp['Addresses']:
        return resp['Addresses'][0]
    
    # Allocate new EIP if pool is empty
    resp = ec2.allocate_address(Domain='vpc', TagSpecifications=[{
        'ResourceType': 'elastic-ip',
        'Tags': [{'Key': 'Pool', 'Value': pool_tag}]
    }])
    return resp


def setup(interface: str, route_table_ids: list, pool_tag: str):
    """Setup EIP association and floating IP routes."""
    
    # 1. Get pod's net1 IP (assigned by Whereabouts)
    private_ip = get_interface_ip(interface)
    print(f"Pod private IP: {private_ip}")
    
    # 2. Find the ENI
    eni_id = find_eni_by_ip(private_ip)
    state['eni_id'] = eni_id
    print(f"ENI ID: {eni_id}")
    
    # 3. Get EIP from pool
    eip = allocate_eip(pool_tag)
    state['public_ip'] = eip['PublicIp']
    state['allocation_id'] = eip['AllocationId']
    print(f"Public IP: {state['public_ip']}")
    
    # 4. Associate EIP with private IP on ENI
    resp = ec2.associate_address(
        AllocationId=state['allocation_id'],
        NetworkInterfaceId=eni_id,
        PrivateIpAddress=private_ip,
        AllowReassociation=False
    )
    state['association_id'] = resp['AssociationId']
    
    # 5. Add /32 routes to VPC route tables (floating IP)
    for rtb_id in route_table_ids:
        try:
            ec2.create_route(
                RouteTableId=rtb_id,
                DestinationCidrBlock=f"{state['public_ip']}/32",
                NetworkInterfaceId=eni_id
            )
            state['route_table_ids'].append(rtb_id)
            print(f"Added route in {rtb_id}")
        except ec2.exceptions.ClientError as e:
            if 'RouteAlreadyExists' in str(e):
                ec2.replace_route(
                    RouteTableId=rtb_id,
                    DestinationCidrBlock=f"{state['public_ip']}/32",
                    NetworkInterfaceId=eni_id
                )
                state['route_table_ids'].append(rtb_id)
    
    # 6. Add public IP as local address on interface
    subprocess.run([
        'ip', 'addr', 'add', f"{state['public_ip']}/32", 'dev', interface
    ], check=True)
    
    # 7. Add source routing rule for public IP
    subprocess.run([
        'ip', 'rule', 'add', 'from', f"{state['public_ip']}/32", 
        'table', 'telecom', 'priority', '100'
    ], check=True)
    
    # 8. Write public IP for main container
    with open('/shared/public-ip', 'w') as f:
        f.write(state['public_ip'])
    
    print(f"Setup complete: {state['public_ip']}")


def cleanup():
    """Remove routes and disassociate EIP."""
    print("Cleaning up...")
    
    # Remove VPC routes
    for rtb_id in state['route_table_ids']:
        try:
            ec2.delete_route(
                RouteTableId=rtb_id,
                DestinationCidrBlock=f"{state['public_ip']}/32"
            )
            print(f"Removed route from {rtb_id}")
        except Exception as e:
            print(f"Failed to remove route: {e}")
    
    # Disassociate EIP (returns it to pool)
    if state['association_id']:
        try:
            ec2.disassociate_address(AssociationId=state['association_id'])
            print(f"Disassociated EIP {state['public_ip']}")
        except Exception as e:
            print(f"Failed to disassociate: {e}")
    
    print("Cleanup complete")


def main():
    interface = os.environ.get('INTERFACE', 'net1')
    route_table_ids = os.environ.get('ROUTE_TABLE_IDS', '').split(',')
    pool_tag = os.environ.get('EIP_POOL_TAG', 'telecom-pool')
    
    # Handle termination signals
    signal.signal(signal.SIGTERM, lambda *_: (cleanup(), sys.exit(0)))
    signal.signal(signal.SIGINT, lambda *_: (cleanup(), sys.exit(0)))
    
    try:
        setup(interface, route_table_ids, pool_tag)
        
        # Keep running until terminated
        # Native sidecar lifecycle ensures we run until main container exits
        while True:
            time.sleep(60)
            
    except Exception as e:
        print(f"Setup failed: {e}")
        cleanup()
        sys.exit(1)


if __name__ == '__main__':
    main()

sequenceDiagram
    participant K as Kubelet
    participant S as Floating IP Sidecar
    participant M as Main Container (SIP)
    
    K->>S: Start sidecar
    S->>S: setup() - EIP + routes
    S-->>K: Running (stays running)
    
    K->>M: Start main container
    M->>M: SIP server running...
    
    Note over M: Pod termination triggered
    
    K->>M: SIGTERM
    M->>M: Graceful shutdown
    M-->>K: Exited
    
    K->>S: SIGTERM (after main exits)
    S->>S: cleanup() - remove routes
    S-->>K: Exited
    
    Note over K: Pod fully terminated