vSAN & Stretched Cluster Design

vSAN is easy to switch on and easy to get subtly wrong. Storage policies, fault domains, witness placement and network design decide whether a cluster rides through a failure or goes read-only at the worst moment. Our vSAN stretched cluster design work, and our reviews of existing clusters, focus on those decisions and on testing the failure scenarios before production depends on them.

When a vSAN stretched cluster makes sense

A stretched cluster spans two data sites with a witness at a third location. Writes go to both sites synchronously, so if one site fails, vSphere HA restarts its VMs at the other site with no data loss. That is a strong fit when:

• Two sites are close enough for low-latency, high-bandwidth links between them, such as two data centers in the same metro area or two buildings on one campus.
• Applications need a recovery point of zero and a recovery time of minutes, and would otherwise need application-level clustering.
• You want planned site maintenance without downtime.

It is a poor fit when the sites are far apart, when the links are shared or unreliable, or when the real requirement is recovery from corruption or ransomware. In those cases two independent clusters with replication between them, as described on our disaster recovery page, is usually the better design.

Inter-site network design

Every write crosses the inter-site link before it is acknowledged, so link latency adds directly to storage latency. Broadcom's published design guidance for stretched clusters calls for a round-trip time between data sites of no more than 5 ms, and for bandwidth sized to the write rate of the workload with headroom for resynchronization after an outage. Check the vSAN design guide for your exact version, because guidance is revised between releases. In practice we:

• Measure the real write profile of the workloads before sizing the links.
• Plan for resync traffic after a site or link outage, which can be far larger than steady-state traffic.
• Use redundant, dedicated paths for vSAN traffic and keep the witness traffic path separate from the inter-site path where possible.

vSAN witness placement and requirements

The witness holds metadata components that let the cluster decide which site keeps running when the sites can't see each other. It must be in a third failure domain, not in either data site, and it needs far less bandwidth and tolerates more latency than the data sites. The witness is usually a virtual appliance running at a third site or in a public cloud. Design decisions we cover include witness location, network routing and static routes to it, whether a shared witness fits your estate, and how the witness itself is patched and monitored.

Storage policies and site affinity

vSAN protects data according to storage policies, and stretched clusters add a second dimension. Site disaster tolerance (the setting formerly known as primary failures to tolerate, PFTT) controls whether a VM's data is mirrored across both sites. Failures to tolerate within each site (formerly SFTT) controls protection against host or disk failures inside a site. Getting these right per workload is where capacity and resilience are traded off:

• Mirroring across sites, plus RAID-1 or RAID-5/6 within each site, for VMs that must survive a site loss and a host failure.
• Site affinity (no cross-site mirroring) for workloads that are protected at the application level, such as clustered databases, so they don't pay for both.
• vSphere DRS rules and VM/host groups that keep each VM running at the site where its data has preference, to avoid cross-site reads.

Failure scenarios to design for and test

A stretched cluster design is only finished when each failure scenario has an expected outcome and has been tested:

We test these in a controlled window before go-live and record the timings and outcomes, so your team knows what normal recovery looks like.

vSAN ESA vs OSA

vSAN 8 introduced the Express Storage Architecture (ESA) alongside the Original Storage Architecture (OSA). At a design level the differences that matter are:

• OSA uses disk groups, each with a cache device and capacity devices, and runs on a broad range of certified hardware.
• ESA uses a single tier of NVMe devices with no separate cache tier, and requires hardware certified for ESA, typically vSAN ReadyNodes. It changes how RAID-5/6 performs relative to mirroring and how capacity is planned.

ESA is generally the direction for new hardware; OSA remains relevant for existing clusters and hardware that isn't ESA-certified. Which is right for you depends on your hardware refresh plans and licensing, which we work through as part of the design.

Compatibility, firmware and drivers

Many vSAN problems trace back to hardware compatibility: a controller, NVMe device or network adapter running firmware or a driver that isn't on the Broadcom Compatibility Guide for your vSAN version. Every design and review checks the hardware, firmware and driver combinations, and sets up vSphere Lifecycle Manager images so hosts stay consistent through upgrades.

vSAN design review and health check

For an existing cluster, a vSAN design review and health check gives you an independent view of where it stands. It covers:

• vSAN Skyline Health findings, explained and prioritized
• Hardware, firmware and driver compatibility
• Storage policy compliance and whether policies match workload needs
• Capacity, slack space and growth headroom
• Network design, including inter-site and witness links for stretched clusters
• Failure-scenario readiness and HA/DRS configuration