Hybrid MSP Infrastructure & AWS Kubernetes Service

The brief

Emerging Technologies hands a team a blank-slate managed-service contract: design, build, and operate shared infrastructure for two client organisations across two physical sites, to a stated set of availability and security requirements — then extend it with a public-facing cloud service.

The two clients are deliberately mismatched. Lumora is a mission-driven organisation that wants a professionally supported environment, predictable cost, centralised identity, and Canadian data handling. ClearRoots is a budget-conscious NGO that prefers open-source platforms, wants to avoid vendor lock-in, and still needs fault-tolerant core services. One tenant pulls toward Microsoft; the other pulls away from it. They share hardware regardless.

Deliverables were a working environment, a 122-page technical handover, a client-facing SLA, and a live defence.

Scope of my work

End-to-end ownership of the public-cloud site — the MSP edge, both tenant stacks, the dual-bastion operations layer, the backup target, and the AWS-hosted service extension were designed, built, and operated solo. The on-prem site was delivered collaboratively.

The MSP boundary

A single OPNsense instance, rp-msp-gateway, carries the WAN edge and anchors eight internal segments: a shared management network, a LAN and a DMZ for each tenant, and two isolated storage bridges.

Segmentation here is not only a security control — it is a triage aid. Each network tells a support engineer where to look first. If the problem is on the MSP segment, it starts at the gateway, a bastion, or the backup host. If it is on C1DMZ, it is a Lumora publication issue and not a ClearRoots identity issue. If storage breaks, the routed network can be perfectly healthy while the file service fails — which is why the SAN bridges sit outside the routed path entirely (see note 1).

Firewall policy is written against named aliases rather than one-off addresses, so a tenant’s reachable scope is described once and reused. Each tenant LAN may reach its own scope, the shared web nodes, and the DNS authorities — and is explicitly blocked from the other tenant’s scope.

Tenant services

Lumora runs a Windows-centric stack: two Windows Server 2022 domain controllers holding AD DS, DNS, DHCP, and Group Policy; a Windows file server presenting SMB shares from iSCSI-backed storage; an IIS server in the DMZ; and both Windows and Ubuntu domain-joined endpoints.

ClearRoots runs the Linux equivalent: two Samba AD nodes paired for DHCP failover, providing AD-compatible authentication, DNS, and SMB; a Linux file server with iSCSI-backed and replicated shares; an nginx server in the DMZ; and a domain-joined Linux client.

The interesting work was not either stack individually — it was keeping them cleanly separated on shared hardware without letting DNS or DHCP scope bleed across the tenant boundary (see note 3).

Operations

Site 2 is operated through a dual-bastion model. Jump64 is the Windows inspection platform, best for AD and backup administration. MSPUbuntuJump is the Linux platform, faster for identity, file, and web validation. Splitting them by operating model resists the temptation to overload one host with every tool in the environment.

S2Veeam anchors recovery: it holds the local repository and receives the offsite copy from Site 1 across the VPN tunnel, on an explicitly enumerated set of ports rather than an open path between sites. Only the two bastions are published at the edge (see note 2).

The cloud extension

The public service is a containerised site running on a two-node kubeadm cluster on EC2, published at clearroots.omerdengiz.com.

Every AWS resource is declared in Terraform — the IAM role and instance profile, the security group, both instances, the Elastic IP, the Route 53 record — with remote state in S3 so the environment can be destroyed and rebuilt reproducibly. That mattered concretely: during the defence, the entire cloud site was torn down and rebuilt from scratch in front of the panel.

The worker carries the Elastic IP because it is the traffic-serving node, which keeps the DNS record valid across worker reboots. The control plane needs no stable public address for browser traffic, and publishing both would add exposure without improving delivery. Caddy on the worker terminates TLS and proxies to a NodePort service, so certificate issuance and renewal need no separate subsystem.

The cloud stack shares nothing with the internal environment (see note 4).