Running OpenShift across several datacentres creates two related availability problems. First, every cluster needs a reliable local entry point for the Kubernetes API, machine configuration service, and application ingress. Second, users need a global entry point that stops returning a datacentre when its service is unhealthy.
This guide builds both layers with a three-node HAProxy Fusion Control Plane, HAProxy Enterprise pairs in each datacentre, redundant GSLB nodes, delegated DNS, and OpenShift service discovery. The example uses four datacentres and five OpenShift clusters, but the pattern scales down or out.
The addresses and names below are examples. Replace them with approved values from your IPAM, DNS, security, and platform teams.
Target architecture
The design separates responsibilities:
- Fusion Control Plane centrally enrolls HAProxy Enterprise nodes, groups them into clusters, pushes configuration, and provides health and version visibility.
- HAProxy Enterprise provides the local data plane. Each pair exposes virtual IPs for OpenShift API, machine configuration, and ingress.
- VRRP or a network-native equivalent keeps each datacentre VIP available when a load-balancer node fails.
- GSLB answers for the delegated application zone and removes unhealthy datacentre endpoints from DNS responses.
- OpenShift service discovery keeps backends aligned with platform services where dynamic discovery is appropriate.
HAProxy documents Fusion as a centralized UI and API for managing multiple Enterprise clusters, including service discovery for Kubernetes environments. Its GSLB module can act as an authoritative DNS service for health-aware datacentre selection. See the official Fusion overview, GSLB guide, and OpenShift external load-balancer requirements.
1. Freeze the deployment inputs
Do not begin installation with unresolved DNS, routing, or firewall decisions. Record the final values in a controlled inventory.
domain:
parent_zone: example.net
delegated_zone: apps.example.net
nameservers:
- name: ns1.gslb.example.net
address: 192.0.2.53
- name: ns2.gslb.example.net
address: 198.51.100.53
fusion:
nodes:
- { name: fusion-cp-01, address: 10.20.1.11 }
- { name: fusion-cp-02, address: 10.20.1.12 }
- { name: fusion-cp-03, address: 10.20.1.13 }
datacentres:
dc-a:
hapee_nodes: [10.20.10.11, 10.20.10.12]
ingress_vip: 10.20.10.50
api_vip: 10.20.10.51
dc-b:
hapee_nodes: [10.30.10.11, 10.30.10.12]
ingress_vip: 10.30.10.50
api_vip: 10.30.10.51
For each additional datacentre, define the same minimum set: management addresses, local VIPs, OpenShift control-plane and ingress targets, DNS endpoints, routing gateway, NTP, and log destination.
2. Validate capacity and connectivity
A conservative production starting point for the project is 16 CPU cores, 32 GB RAM, and 256 GB storage per designated server, subject to measured connection rate, TLS workload, log retention, and vendor sizing. HAProxy’s own Linux installation guide stresses that actual sizing depends on traffic and cryptographic workload.
Check time synchronization and forward/reverse DNS first:
timedatectl status chronyc sources -v getent hosts fusion.example.net getent hosts lb-a-01.example.net dig -x 10.20.10.11 +short
Confirm management reachability from Fusion to each load balancer. Fusion-managed Enterprise nodes use HTTPS for the Data Plane API; the enrollment workflow shown later supplies the precise port and certificates for your release.
nc -vz lb-a-01.example.net 5555 nc -vz lb-a-02.example.net 5555 curl -kIs https://lb-a-01.example.net:5555/
Open only the flows approved by the design:
- Client or upstream load-balancer traffic to TCP 80 and 443.
- Administrator and automation access to the Fusion UI/API.
- Fusion-to-node management traffic.
- DNS UDP/TCP 53 to both GSLB nodes.
- VRRP protocol traffic between local peers when VRRP is used.
- OpenShift API TCP 6443 to control-plane nodes.
- Machine Config Server TCP 22623 to control-plane nodes, restricted to the provisioning and cluster networks.
- Ingress TCP 80/443 to the nodes running router pods.
- NTP, DNS, repository, monitoring, and remote logging flows.
Red Hat explicitly requires 6443 to be balanced across control-plane nodes and 80/443 across compute or ingress nodes, while 22623 must not be exposed outside the cluster.
3. Prepare every RHEL host
Use a supported RHEL release, patch it, set the hostname, and verify that security controls are active.
sudo hostnamectl set-hostname lb-a-01.example.net sudo dnf update -y sudo systemctl enable --now chronyd sudo getenforce sudo firewall-cmd --state
Create separate mount points or retention controls for product data and logs. Avoid disabling SELinux or the firewall as a shortcut; add documented policy and firewall rules instead.
Capture the baseline:
uname -r lscpu free -h lsblk ip -br address ip route
4. Build the three-node Fusion Control Plane
Obtain the Fusion installation bundle and entitlement from the HAProxy customer portal. Installation commands and supported operating systems vary by Fusion release, so use the exact command supplied with your package rather than copying a bootstrap URL from another environment.
Run the installer on the first node with its permanent hostname and advertised management address, then join nodes two and three using the generated cluster join material. The following commands show the complete shape of a typical bundle-driven installation.
export FUSION_VERSION="1.x.y"
tar -xzf "haproxy-fusion-${FUSION_VERSION}-installer.tar.gz"
cd "haproxy-fusion-${FUSION_VERSION}-installer"
sudo ./install.sh \
--config ./config/fusion-cp-01.yaml \
--node-name fusion-cp-01 \
--advertise-address 10.20.1.11 \
--cluster-name fusion-production
Command note: This is a representative Fusion installer invocation. Confirm the archive name, install.sh entry point, configuration schema and supported flags in the licensed installation bundle for your exact Fusion release before execution.
sudo install -m 0600 /secure-transfer/fusion-join.token \ /root/fusion-join.token sudo ./join.sh \ --controller https://fusion-cp-01.example.net:9443 \ --node-name fusion-cp-02 \ --advertise-address 10.20.1.12 \ --token-file /root/fusion-join.token sudo rm -f /root/fusion-join.token
Command note: This join sequence is an operational example. The actual join utility, controller port, certificate bootstrap and token handling are release-specific; use the one-time join material produced by the installed primary node and the instructions shipped with the licensed bundle.
Repeat for the third node, then verify the cluster from the Fusion administration page and the vendor-supplied health command.
sudo systemctl --no-pager --full status haproxy-fusion sudo journalctl -u haproxy-fusion --since "15 minutes ago" \ --no-pager curl --fail --silent --show-error \ --cacert /etc/haproxy-fusion/pki/ca.crt \ https://127.0.0.1:9443/healthz
Command note: These checks demonstrate the expected systemd, log and HTTPS-health workflow. Verify the installed service-unit name, health endpoint, port and CA path for your Fusion version; a successful HTTP check alone does not prove that all three control-plane members have quorum.
Put a stable management VIP or approved load-balancer address in front of the Fusion UI. Install a certificate whose SAN list contains the user-facing FQDN, enforce RBAC, connect enterprise identity, and send audit logs to the central platform before onboarding production nodes.
5. Create one Fusion cluster object per HAProxy pair
In Fusion, open Infrastructure → Clusters, create dc-a-hapee, and repeat for the other datacentres. A cluster object should contain nodes that are expected to share one synchronized HAProxy configuration.
Use names that communicate location and role:
dc-a-hapee dc-b-hapee dc-c-hapee dc-d-hapee
Do not place unrelated load balancers in the same cluster merely because they run the same software version. Configuration synchronization and operational actions follow the cluster boundary.
6. Enroll the first HAProxy Enterprise pair
Open Infrastructure → Nodes → Add Node → Automatic node addition. Select bare metal, enter the node’s advertised management address, choose the Data Plane API version supported by your Fusion release, and copy the generated one-time command.
Run the generated command locally on the intended host through the approved privileged-access path.
# Run only the command generated for this exact node and cluster. sudo <FUSION_GENERATED_ENROLLMENT_COMMAND>
Treat the enrollment command as a secret: it can contain a short-lived token and licensing material. Never paste it into tickets, chat, screenshots, shell history exports, or a public repository.
After the node appears in Awaiting Approval, verify its hostname, address, target cluster, and certificate identity. Approve it, wait for both health checks, and repeat for the peer.
The ready state is unambiguous: both nodes are active, health is 2/2, configuration is OK, no node is out of sync, and the last push completed successfully.
7. Configure local high availability
For networks that support VRRP, install and configure the HAProxy Enterprise VRRP module. HAProxy’s active/standby guide requires the same VRID and VIP on both peers, but a different priority. Each VRRP cluster must use a unique VRID between 1 and 255.
On the preferred node:
vrrp_instance OCP_INGRESS {
interface ens192
state MASTER
virtual_router_id 51
priority 150
virtual_ipaddress {
10.20.10.50/24
}
}
On its peer:
vrrp_instance OCP_INGRESS {
interface ens192
state BACKUP
virtual_router_id 51
priority 100
virtual_ipaddress {
10.20.10.50/24
}
}
Add a tracked interface and a health script so a running host with a failed HAProxy service cannot retain the VIP. Allow VRRP in the local firewall and confirm that the switching fabric accepts gratuitous ARP updates. If the network blocks multicast or floating IP movement, use the supported network-native alternative such as routed anycast, ECMP/RHI, or an upstream load-balancer tier. HAProxy also documents active-active patterns.
Verify ownership:
ip address show dev ens192 sudo systemctl status hapee-3.3-lb sudo tcpdump -n -i ens192 vrrp
8. Build the OpenShift frontends and backends
Create reusable Named Defaults and Globals, then define explicit frontends and backends. Fusion’s structured sections make these objects visible without losing the underlying HAProxy model.
Kubernetes API
frontend ocp_prod_a_api bind 10.20.10.51:6443 mode tcp option tcplog default_backend ocp_prod_a_control_plane backend ocp_prod_a_control_plane mode tcp balance roundrobin option tcp-check server master-0 10.20.40.11:6443 check server master-1 10.20.40.12:6443 check server master-2 10.20.40.13:6443 check
Machine Config Server
frontend ocp_prod_a_mcs bind 10.20.10.51:22623 mode tcp option tcplog default_backend ocp_prod_a_mcs_nodes backend ocp_prod_a_mcs_nodes mode tcp balance roundrobin option tcp-check server master-0 10.20.40.11:22623 check server master-1 10.20.40.12:22623 check server master-2 10.20.40.13:22623 check
Restrict this frontend at the firewall and routing layers. It is for node bootstrap and configuration retrieval, not public clients.
Application ingress
frontend ocp_prod_a_http bind 10.20.10.50:80 mode tcp default_backend ocp_prod_a_router_http frontend ocp_prod_a_https bind 10.20.10.50:443 mode tcp default_backend ocp_prod_a_router_https backend ocp_prod_a_router_http mode tcp balance source option tcp-check server infra-0 10.20.50.21:80 check server infra-1 10.20.50.22:80 check server infra-2 10.20.50.23:80 check backend ocp_prod_a_router_https mode tcp balance source option tcp-check server infra-0 10.20.50.21:443 check server infra-1 10.20.50.22:443 check server infra-2 10.20.50.23:443 check
Use the actual nodes hosting router pods. If routers move dynamically, configure Fusion Kubernetes service discovery after the static path is proven.
Validate and publish through Fusion. Take a configuration snapshot before every material change and confirm both peers reach the same configuration version.
9. Install the GSLB module on both DNS nodes
On supported RHEL-family systems, install the GSLB module from the entitled HAProxy Enterprise repository:
sudo dnf install -y hapee-extras-gslb sudo cp /etc/hapee-extras/hapee-gslb-example.conf \ /etc/hapee-extras/hapee-gslb.conf
The service listens on DNS port 53 by default and can also expose a loopback listener. Confirm the packaged environment file before changing it:
sudo grep -E '^(GSLB_LISTEN|GSLB_CONFIGFILE|GSLB_RUNPATH)' \ /etc/sysconfig/hapee-extras-gslb
Build the zone with a low TTL during commissioning. This example prefers DC-A, then fails over through the remaining datacentres when health checks fail:
zone apps.example.net ttl 30 record @ SOA ns1.gslb.example.net hostmaster.example.net 2026082001 3600 900 604800 30 record @ NS ns1.gslb.example.net. record @ NS ns2.gslb.example.net. record api ttl 30 list DC_A DC_B DC_C DC_D record api-int ttl 30 list DC_A DC_B DC_C DC_D record * ttl 30 list DC_A DC_B DC_C DC_D answer-list DC_A up_threshold 1 method multi-up option httpchk fall 3 rise 2 http-check connect port 443 ssl http-check send uri /healthz/ready hdr host health.apps.example.net http-check expect status 200 answer-record dc-a 192.0.2.10 answer-list DC_B up_threshold 1 method multi-up option httpchk fall 3 rise 2 http-check connect port 443 ssl http-check send uri /healthz/ready hdr host health.apps.example.net http-check expect status 200 answer-record dc-b 198.51.100.10
Add corresponding answer-list sections for DC-C and DC-D. HAProxy Enterprise 3.2r1 and newer can proxy HTTPS health checks through a local HAProxy frontend; follow the official GSLB guide when enabling that path.
Validate, restart, and inspect:
sudo /usr/sbin/hapee-extras-gslb -t \ -f /etc/hapee-extras/hapee-gslb.conf sudo systemctl restart hapee-extras-gslb sudo systemctl --no-pager status hapee-extras-gslb sudo ss -lunpt | grep ':53'
Run the same configuration on the second GSLB node, with configuration management or a controlled replication process. Do not rely on manual copy-and-paste as the steady-state operating model.
10. Delegate the DNS zone
Create address records for the two authoritative GSLB nameservers, then delegate the child zone from the parent DNS:
ns1.gslb.example.net. A 192.0.2.53 ns2.gslb.example.net. A 198.51.100.53 apps.example.net. NS ns1.gslb.example.net. apps.example.net. NS ns2.gslb.example.net.
Where the nameserver itself sits below the delegated zone, add the required glue records in the parent. Keep both UDP and TCP 53 reachable; large DNS responses and fallback behavior can require TCP.
Test each authority directly before relying on recursive DNS:
dig @192.0.2.53 apps.example.net SOA +norecurse dig @198.51.100.53 api.apps.example.net A +norecurse dig @192.0.2.53 test.apps.example.net A +norecurse dig +trace api.apps.example.net
11. Connect OpenShift service discovery
In Service Discovery → Kubernetes, register each OpenShift API endpoint with a dedicated service account. Grant only the read permissions required to discover services, endpoints, and endpoint slices; do not use cluster-admin.
apiVersion: v1
kind: ServiceAccount
metadata:
name: haproxy-fusion-discovery
namespace: haproxy-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: haproxy-fusion-discovery
rules:
- apiGroups: [""]
resources: ["services", "endpoints", "namespaces"]
verbs: ["get", "list", "watch"]
- apiGroups: ["discovery.k8s.io"]
resources: ["endpointslices"]
verbs: ["get", "list", "watch"]
Bind the role to the service account, issue a short-lived token or use the supported identity method, install the OpenShift API CA in Fusion, and scope discovery rules to approved namespaces and labels.
oc auth can-i list services \ --as=system:serviceaccount:haproxy-system:haproxy-fusion-discovery \ --all-namespaces oc auth can-i create secrets \ --as=system:serviceaccount:haproxy-system:haproxy-fusion-discovery \ --all-namespaces
The first command should return yes; the second should return no.
12. Execute failure tests
Treat failover as a test plan, not an assumption.
Local node failure
- Record the current VIP owner.
- Stop HAProxy on that node during the approved window.
- Confirm the peer takes the VIP.
- Verify API and application flows.
- Restore the node and confirm the intended preemption policy.
watch -n 1 'ip -br address show ens192' curl -k https://api.ocp-prod-a.example.net:6443/readyz curl -kI https://console-openshift-console.apps.example.net/
Datacentre failure
- Make the GSLB health endpoint fail for DC-A.
- Query both authoritative nodes repeatedly.
- Confirm DC-A’s address disappears after the configured fall count.
- Wait through the DNS TTL and test client traffic.
- Restore the endpoint and confirm it returns only after the rise count.
for i in {1..10}; do
date
dig @192.0.2.53 test.apps.example.net A +short
sleep 5
done
Also test one GSLB node down, one Fusion node down, a broken configuration push, expired certificates, loss of OpenShift API reachability, and rollback to the previous Fusion configuration snapshot.
13. Roll out to the remaining datacentres
Use a repeatable wave:
- Build and harden the two local HAProxy Enterprise nodes.
- Create the Fusion cluster object.
- Enroll, approve, and health-check both nodes.
- Configure local VIP failover.
- add OpenShift API, MCS, and ingress paths.
- Test locally without GSLB traffic.
- Add the datacentre to GSLB with zero or test-only weight.
- Run functional and failover tests.
- Enable production selection.
- Capture as-built documentation and acceptance evidence.
Do not add a new datacentre to public DNS merely because its nodes appear green in Fusion. The acceptance gate is end-to-end: DNS answer, VIP ownership, HAProxy health, OpenShift API readiness, router response, logs, metrics, and failover.
Production acceptance checklist
- Three Fusion nodes are healthy behind the approved management endpoint.
- SSO, RBAC, certificates, backups, audit logging, and monitoring are enabled.
- Every HAProxy Enterprise pair is synchronized and reports 2/2 healthy.
- Each local VIP fails over within the agreed recovery objective.
- OpenShift 6443, 22623, 80, and 443 paths match the platform design.
- Port 22623 is not exposed to untrusted networks.
- Both GSLB nodes answer authoritatively over UDP and TCP.
- Parent DNS delegates the child zone to both GSLB nameservers.
- GSLB removes failed datacentres and restores them only after the rise threshold.
- OpenShift discovery uses least-privilege, scoped credentials.
- Configuration snapshots and rollback have been tested.
- Operations teams have the runbook, diagrams, credentials handover, monitoring dashboards, and support escalation path.
The most reliable sequence is control plane first, one complete datacentre second, GSLB and delegation third, and the remaining datacentres in controlled waves. That approach turns every stage into a repeatable pattern and keeps the blast radius small while the platform team proves health checks, failover, and OpenShift integration.
Leave a Comment
Your email address will not be published. Required fields are marked with *