Businesses rely on data to get things done. For example, airlines depend on data to analyze passenger demand and travel patterns to optimize schedules, pricing, and staffing. You can imagine how much data an airline must crunch through to provide the best services to their clients. AI, in particular Narrow AI, can help us analyze massive amounts of data to help us understand the data quicker. But should businesses run AI in the cloud, or is Private AI a good idea?
VMware came to AI Field Day to explain their Private AI offering. They say it’s a way to cut costs, provide governance, and use AI hardware to its full potential.
Problems organizations face when developing AI solutions
Given the critical role of data, leveraging AI technology becomes essential for businesses to stay competitive. Tasha Drew, Engineering Lead for the AI, and Advanced Services Engineer Group in the VCF division at Broadcom explained some of the challenges their customers are facing.
A big problem in the industry is GPU underutilization. GPUs (graphical processing units) are vital to AI because they are what provides the compute power to process such large amounts of data in a reasonable amount of time.
Despite the high demand, GPUs are often underutilized because of how inference workloads are managed. If one group within an organization purchases them, they don’t want to share because they want to be sure that the hardware is ready when they have a job to run.
In addition to wasting time, this is a real problem because GPUs are extremely expensive. It means expensive hardware is sitting idle while another division of the same company is scrambling to find GPUs to complete a job. Wouldn’t it make a lot more sense to have this capacity managed by a central IT team that could schedule usage across the entire organization?
You could run AI in the cloud to shift the cost of ownership to a cloud provider. But things get expensive as projects scale. There is also model governance to consider. When AI models are built on proprietary data, it is crucial that they are managed and secured to protect the training data as well as any new data generated.
Could on-premises AI be more cost-effective and provide a way for secure model governance?
VMware Private AI Foundation
Broadcom thinks it on-premises AI can be more cost-effective. Justin Murray, Product Marketing Engineer at Broadcom, introduced VMware Private AI Foundation with NVIDIA (VPA). VPA is based on VMware Cloud Foundation (VCF), which is based on vSphere, VSAN, NSX, and vRealize.
That’s really important. VCF is the main virtualization package that Broadcom sells. It is the basis for many enterprise infrastructures. Enterprise operations teams understand how VMware’s product suite works and have decades of experience under their belts running enterprise workloads on virtual infrastructure.
Justin reminded us before really digging into the architecture that it’s data scientists who will evaluate new models, training the models, etc. But the data science teams work in isolation on VMs dedicated to their deep learning processes, using isolated Kubernetes clusters that run independently of each other. And they struggle to manage the underlying infrastructure.
But virtual operations teams understand how to manage diverse environments. AI workloads are no different. VPA can manage the infrastructure underlying AI, providing isolated environments for data scientists to work independently and efficiently. NSX provides micro-segmentation, enabling GPU sharing across different user communities while maintaining isolation and security. All the operator or platform engineer needs to know is how much GPU power the model will need.
The platform makes the underlying infrastructure invisible to developers and data scientists by providing self-service automation. This allows them to focus on their work without worrying about infrastructure. That is what platform engineers or operators do best!
VPA can also provide model governance, ensuring safe testing and promotion of models in isolated environments before deploying them in Kubernetes. How cool would that be when new models like DeepSeek pop up unexpectedly?
How VMware helps organizations provide Private AI services
Alex Fanous, Staff Architect for VCF Advanced Services group at Broadcom, dove into an example customer journey with VMware Private AI, describing how their customers typically go from a C-level initiative to production.
Many times, their customers find themselves trying to meet a demand from senior leadership to provide “something with AI.” A common example is a request to create a chat app using retrieval-augmented generation (RAG). Where do they even start?
Many companies look for the low-hanging fruit, such as HR policies or apps. You’ve probably needed to find one thing in a 500-page employee document – what a pain! But with a chat app you can search by asking a question. This is a good way to understand how the systems work before tackling models that need to deal with private data.
AI environments are complex, and the infrastructure is just the start.
The console also surfaces the technical info such as the infrastructure as code required to run it, the model, etc. This can all be edited as needed. The developers will adjust the UI and train the model as needed. Operators will manage the technical requirements for hosting, managing, and protecting these workloads.
Conclusion
VMware’s AI Field Day presentation focused on the operational side of AI, and reasons enterprise organizations should consider keeping these workloads on premises. One thing I find interesting is that a cloud environment is virtualized by default.
For some workloads a cloud environment may work. But VMware lays out a good argument that using Private AI can help you save money, utilize your GPUs better, and providing a way for safe model governance.
Using VPA you can manage the entire organization’s AI infrastructure needs. This can help share infrastructure costs over the entire organization, including GPUs. It gives operations an easy way to set up a dashboard for data scientists and developers, so they never have to worry about the underlying architecture again.
To listen to the entire presentation, please check out Tech Field Day’s AI Field Day 6 page.
