AI clusters have gotten a shared infrastructure. Neoclouds, enterprise AI platform groups, monetary providers organizations, life sciences groups, and analysis teams must share GPU capability. This shared infrastructure can undergo from decrease monetization, elevated operational complexity, and restricted management and visibility throughout tenants, workloads, hosts, and the community material.
EVPN/VXLAN is the sensible community basis. It offers tenant-scoped overlay segmentation utilizing VRFs, VNIs, route distinguishers, and route targets. Nonetheless, tenant-aware segmentation just isn’t job-aware segmentation. The scheduler understands jobs; the community sometimes understands routes, interfaces, queues, drops, and flows.
Why AI clusters want multitenancy
Devoted GPU clusters are easy to isolate, however they’re inefficient to function at scale. As GPU estates develop, organizations desire a shared useful resource pool that may serve a number of groups, prospects, and workload courses with out forcing each group into its personal bodily cluster. In any other case, one group can have stranded GPUs in a devoted island whereas one other waits in queue.
The requirement seems in a number of patterns:
- A GPU-as-a-Service supplier maps every tenant to an exterior buyer with its personal tackle and coverage area (per-customer isolation whereas preserving the GPU pool shareable).
- An enterprise platform staff maps tenants to improvement, testing, manufacturing fine-tuning, mannequin analysis, or regulated analytics (constant setting boundaries with out constructing separate clusters).
- A monetary service division separates fraud analytics, danger modeling, and analysis workloads on one coaching cluster (stronger management boundaries and auditability with out duplicating GPU islands).
- A analysis group assigns shared GPU capability to unbiased analysis teams (clearer quota, utilization, and troubleshooting accountability throughout competing initiatives).
For this reason multitenancy can not cease at compute allocation. Distributed coaching will depend on east-west GPU communication, sometimes over Ethernet materials, so the community turns into an integral a part of the isolation and efficiency boundary.
How business solves it right this moment
Present AI multitenancy is often carried out throughout three layers:
- Orchestration and scheduler layer. Kubernetes-based platforms, GPU cloud orchestration techniques, and Slurm schedulers outline the logical possession mannequin for the cluster. They monitor tenants or initiatives, customers, queues or namespaces, job requests, node placement, and GPU allocation. For instance, Tenant A would possibly submit Job 100 requesting eight GPUs throughout two servers, whereas Tenant B submits Job 200 requesting 4 GPUs on a special set of nodes. As an example, in an orchestration platform like Rafay, the platform can personal tenant onboarding and infrastructure intent, whereas the precise job scheduling might occur in Kubernetes, Slurm, or a tenant-operated scheduler.
- Host isolation layer. The host enforces the native system boundary for every workload. If a tenant receives complete servers, isolation is easier as a result of the server, GPU set, and NIC set might be handled as one tenant-owned unit. If a number of tenants or jobs share GPUs throughout the identical server, the runtime should expose solely the assigned GPU units and bind the workload’s communication libraries, similar to NCCL or UCX, to the supposed NICs. This host-side mapping issues as a result of a GPU server might have a number of NICs linked to totally different switches or tenant-facing community segments. Material segmentation can isolate site visitors as soon as it enters the community, nevertheless it can not appropriate an incorrect native task the place the workload is allowed to make use of the flawed GPU or NIC.
- Community segmentation layer. EVPN/VXLAN offers scalable tenant segmentation throughout the material. VXLAN encapsulates tenant site visitors and makes use of VNIs to determine the overlay section or routing area. EVPN makes use of BGP to promote endpoint and prefix reachability and to regulate which VTEPs import a tenant’s routes by route targets. In a routed AI material, a tenant generally maps to a VRF and a number of VNIs, with route distinguishers preserving tenant routes distinctive and route targets controlling import-export coverage. This permits overlapping tenant tackle area and scoped reachability throughout a shared underlay.
ACLs or safety group ACLs can add supply and vacation spot coverage, particularly in brownfield L3 designs or the place the material can not but devour richer workload id. Their limitation is operational scale: static or manually up to date ACL and VRF insurance policies don’t naturally observe fast-changing AI job placement.
Collectively, these layers present a workable tenant-level mannequin. The remaining hole is job context: the community can often see tenant context, interfaces, routes, queues, and counters, however not the particular scheduler job operating inside a tenant. Tenant segmentation itself doesn’t robotically isolate Job 100 from Job 101 inside the identical tenant except job id can also be carried, derived, or programmed into community coverage.
Cisco Nexus One integration with AI iorchestration platforms
Cisco Nexus One is properly positioned because the broader basis for making tenant-aware AI materials extra deterministic. On this structure, Nexus One is the entire material automation, integration, and visibility floor for the complete material.
Nexus One can present material topology context to an AI infrastructure orchestration platform similar to Rafay by integration workflows or APIs. That lets groups map tenant VRFs, VLANs, and port assignments on to a tenant, slightly than managing them solely as an summary tenant label.
As well as, Nexus One extends the mannequin past provisioning. Tenant-level visibility can present the tenant’s material path and related well being alerts similar to congestion, drops, and so forth. This enhances AI job observability: job-aware views can correlate scheduler, topology, optics, GPU telemetry, analytics, and anomalies, whereas tenant-specific Job-ID enforcement stays a separate future-facing coverage functionality.
Tenant-aware just isn’t job-aware
Tenant segmentation solutions the query, “Which buyer or group owns this site visitors?” AI operations typically want, “Which coaching job is creating or experiencing this site visitors inside a tenant?”
This distinction issues for segmentation in addition to throughout troubleshooting. A scheduler can determine the job, allotted nodes, GPUs, and runtime state. The community can determine interfaces, routes, queues, drops, ECN marks, PFC occasions, optics well being, and paths. With out correlation, operators should manually join these two views.
The result’s a standard operational drawback: the material reveals a sizzling uplink or lossy interface, whereas the platform staff sees a gradual coaching job. The lacking hyperlink is the workload id within the community working mannequin.
Future route: AI Job-ID-aware segmentation
Job-ID-aware segmentation route—patent-pending know-how from Cisco—is the logical subsequent step. (Word that this describes our architectural route, not a delivery characteristic.) The purpose is for infrastructure orchestrator (similar to Rafay) and scheduler (similar to Slurm) intent to hold each tenant id and job id into the material management and data-plane mannequin.
In that mannequin, the material controller interprets job intent into coverage. The change information airplane carries or derives a job ID, for instance by VXLAN GPO bits, and enforces that solely endpoints in the identical licensed tenant and job can change RoCEv2 site visitors.
The anticipated advantages are operationally vital:
- Less complicated operations, as a result of groups can motive in tenants and jobs as a substitute of translating each grow to be static community objects—essential for NetOps and material operations groups.
- Deeper visibility, as a result of drops, congestion, paths, and optics might be correlated to workload context slightly than solely to interfaces or tenant VRFs—useful for platform engineering and SRE groups.
- Extra granular segmentation, as a result of coverage can observe the lifecycle of a job slightly than stopping on the tenant boundary—vital for safety, compliance, and tenant governance groups.
This method is constructed on open requirements—not a proprietary overlay. EVPN/VXLAN is IETF-defined, and the Group Coverage Choice (GPO) follows the identical path: an IETF-defined mechanism that encodes a bunch/coverage identifier within the VXLAN header alongside the VNI, which Cisco NX-OS implements in alignment with the open specification. Tenant scope (VNI) and workload/job scope (GPO) are due to this fact expressed in constructs a standards-compliant material can interpret—letting operators evolve from tenant-aware to job-aware enforcement and not using a material forklift.
Technical instance: tenant and job boundaries
Take into account a GPU-as-a-Service setting with two prospects, Tenant A and Tenant B. Every tenant is mapped to its personal VRF/VNI within the EVPN/VXLAN material. Tenant-level segmentation prevents Tenant B from reaching Tenant A.
Now assume Tenant A runs two concurrent coaching jobs. Job 100 makes use of GPUs on servers 1 and a couple of. Job 101 makes use of totally different GPUs on the identical shared material. Tenant-level EVPN/VXLAN nonetheless treats each jobs as Tenant A site visitors. Job-ID-aware segmentation would add one other enforcement dimension: Job 100 endpoints may change RoCEv2 site visitors with different Job 100 endpoints, however not with Job 101 endpoints, even inside the identical tenant.
That’s the architectural shift: EVPN/VXLAN stays the tenant basis, whereas Job ID turns into the longer term workload-level coverage and observability attribute.
Advancing safety from tenant-level to job-level segmentation
AI information middle multitenancy begins with EVPN/VXLAN tenant segmentation, nevertheless it doesn’t finish there. The stronger working mannequin combines topology-aware provisioning, tenant-level enforcement, and end-to-end visibility right this moment, then evolves towards Job-ID-aware segmentation as scheduler and orchestrator integration matures.
If you’re designing a shared AI cluster right this moment, tenant-aware EVPN/VXLAN is the inspiration. Job-aware enforcement and observability are the subsequent frontier.
*Particular due to Ramesh Ponnapalli and his staff, whose engineering management has been instrumental in bringing this know-how to life.
Further sources:
