IT Ops Query: Exec Who Built Nvidia DGX Pivots to Sovereign AI

0:01 from Informat Tech Target. I'm Beth Pariso and this is IT Ops Query. This podcast distills the signal from the noise about enterprise software development and platform engineering. Each week, we'll talk to expert guests about the latest tech industry news and trends that engineering and IT leaders need to know. Don't forget to subscribe to IT Ops Query for more conversations on AI and the future of the enterprise digital workspace. Stefan Fable spent nearly six years at NVIDIA where he launched and ran its DJX cloud AI supercomputing service. In April, he joined sovereign AI infrastructure company this way Global as CEO. This Way Global markets sovereign cloud services including amalgamy.ai contractbased AI

0:45 infrastructure orchestration software along with workforce training services. Welcome Stefan. Thanks for joining us. >> Thanks for having me Beth. Excited to be here. So um sovereign AI is a hot topic right now but it has different definitions for different people. So how do you define it? >> Well we define sovereign AI or sovereign infrastructure as infrastructure that serves a sovereign entity for the purposes of AI development, national defense or public service. So when you think about it you know the state might have different research interests that they want to push. We see initiatives for example in the dementia research space that is now being well funded by the state of Texas. So those initiatives need to execute somewhere. They need

1:34 accelerated computing to be made available to them and so uh we're seeking to to help them achieve that. >> Okay. I mean overall what would you say is the state of sovereign AI especially among enterprises? Is it still kind of a primordial soup? Are things starting to coalesce? >> Yeah, I I think that we're in this buzzword phase of sovereign. >> I think a lot of people like to say sovereign >> just because that's what you say. I I think we're we're more in a in the space of it's not sovereign unless a sovereign is in the data center. M >> if uh you know if it's something like data sovereignty that's certainly a concept and it's certainly also part of of serving a sovereign entity or collaborating with a sovereign entity

2:22 and certainly you know data isolation auditability and chain of custody and things like that are important also to regulated enterprises and not just sovereign entities but that doesn't make a Loheed Martin sovereign it just means that they're interested in not sharing their data with you. >> Correct. Correct. Okay. That's an important distinction. How about more broadly? I mean, having, you know, run DGX, what would you say is the state of AI infrastructure in general? Do you think enterprises and other entities have a good grasp on how to orchestrate AI workloads at this point? >> I don't think so. I think that >> I think there's plenty of work left to do. I think what you what you see is a high concentration of uh GPU capability

3:09 in the hands of very few >> and there's many views and perspectives that people have on this and depending on which side you look at it from it presents itself slightly different. So let's say a Wall Street analyst might look at it as a circular economy, >> you know, we might look at it as well or or an enterprise let's say would look at it as my AI strategy is using copilot, you know, and it's it's not really an AI strategy. It's just I'm using a tool that Microsoft is making available to me and so that is essentially my AI strategy. M so >> so you hear about you know enterprises especially and large entities bringing AI infrastructure back on premises because of security and in some cases sovereignty concerns is that something

3:59 that you see actually happening or you know how do you predict that's going to unfold >> I don't think that that is happening at at grand scale yet >> first of all I think it's very hard to secure the supply of these chips, you have extraordinarily long lead times for, you know, the latest or even not the latest GPUs from Nvidia and others to be able to to even get their hands on this is pretty daunting in and of itself, then I think the scale of of that's needed to actually be effective in using this GPU infrastructure typically exceeds the capability of an operational team that might be present in some of these companies. So there is a skills gap and we saw that being the same really when we introduced private cloud about you

4:51 know 15 years ago. There was the whole thing about we have a skills gap. All of these private cloud uh initiatives fail because or they're they're not really very successful because of this gap in scaling up the workforce. Uh and so we see similar evidence actually on the AI market as well. We have an MIT report that said that only 5% of all AI initiatives actually then yield a productive result or go into production. >> So, it's not the last word. Of course, this is a moving target. The the market is evolving and it's not to say it's all doom and gloom, but I'm just saying >> this is certainly not without its challenge, >> right? How does this way global navigate that challenge in terms of getting its hands on infrastructure capacity? big

5:38 answer, a big piece of the answer is sovereign and you know it's really our ability to get sovereign contracts and work with the sovereign offtake essentially that lets us get access to all of these GPUs uh often times at you know even discounted rates. It's not an easy thing to get because you actually have to get these state contracts >> and that's not a a normal endeavor. you have to work and be able to work with all of the bureaucracy and the red tape etc. But once you're at the other end of that, you're actually you're actually getting uh getting what you need the Yeah. >> So governments essentially are are procuring infrastructure. Is that what you mean? >> I think that they would like to procure infrastructure access. So it's not

6:26 necessarily the capex, it's really the ability to to support their research. So really our entry point is then universities and they're able to to then facilitate uh that that budgeting through the state. >> Okay. And so there are many entities outside the United States that are worried about working with USbased US-owned companies. So how do you address that concern? >> Well, I mean I think that you cannot really alleviate worry. That's a that's a decision that I think everybody else needs to make on their own >> time. But I think you can minimize risk. And I think if you if you convert the conversation from how can I help you not to worry into how do I minimize the risk that you can see then you establish trust. And I think that is sort of the

7:21 judo move you need to make here. If you if you talk to another sovereign entity, they are already going to be relying on a high degree of American technology. The GPUs will not be produced in their country. >> Most certainly not. The the networking will not have been developed. The software will mostly not have been written. And so there's a lot of things about it that not are not going to be from their country, right? And can't be homesourced. However, if you if you couple that with a workforce development plan that actually lets you scale up the workers locally and do that across the whole stack like we do, I mean, this is literally we're working with vocational schools to establish programs that are associated

8:10 with our AI learning and training centers. then you're actually now letting the local community participate in the AI economy and oftentimes take that next step in their career. >> And so now that skills transfer as it happens, it actually establishes a lot of trust to be able to say, look, you might be using our software, you might be using American chips, you might be relying on, you know, Taiwanese produced network interface cards, right? But >> you know, you can trust the people that are in there because they're your sovereign citizens. >> And so you draw the distinction between a sovereign in terms of essentially a government entity and a regulated enterprise. But are there things that are emerging in among sovereigns that

8:58 will inform regulated enterprises going forward in this world? >> That's an interesting question. I think that that it's I mean almost certainly there's an interplay. I mean just via policy and you know and I think that when you when you think about what the effect of access of regulated industry to GPUs would be >> then you could imagine different ways AI could actually play a role in our lives. So what do I mean today there's like six models. >> Yeah. >> And and they're all large language models and they're all really about >> language actual language. And so it stands to reason that some of those are open source because language doesn't belong to anybody and we sort of have a a very intimate relationship with that

9:44 aspect. But then there's other ways you could think about models, right? You could think about a model that detects cancer sooner in in radiology. Now if you know a GE or seammens or somebody like that were to be able to train a model such as this, hey, I would be very interested in getting you know scanned by that. But uh but also you could imagine that be highly proprietary >> and it would be licensed right and it wouldn't be open source and it wouldn't be something that belongs to everybody because you know it was very expensive to create and would be a competitive advantage for these companies and similarly you could imagine a point in time where I mean it's not unreasonable to think anyway that there will be a

10:27 point in time where there's legislature that says you are allowed to produce autonomous vehicles in the but only if your foundation model is this ISO standard approved model because that's the one you know we checked out and that's the one we are sure about and so on and sure the Toyota model can be the Toyota model and the you know the GMC model can be the GMC model but it has to be based on this foundation model that is uh that is approved so you could imagine regulatory interaction between the legislative or or the state and and these regulated industries Yeah, AI regulation is such a thorny issue in this country anyway and around the world because you know this this stuff is still being born. I mean it's still so

11:14 early in the game that >> Yes. >> I think few people really understand it to that level especially in government where you know they're not technologists. I mean, is that something that that you kind of help out with in a consultative role or >> Absolutely. I mean, we're working very closely with to educate politicians about, you know, what is AI and how do you use it and how could could it actually be to the benefit of of of all >> rather than the very few, right? I mean, there's certainly an aspect of, you know, these large companies build these large data centers in our backyard. they take all the profits and what are we using AI for? We're doing cat pictures, have to do our homework and then they take our jobs, you know, it's like why

12:01 would I support something like this, right? And so a part of a part of that conversation and I think that's also very timely right now given all the moratoriums that you know that data centers have been undergoing here. >> I think it's important to understand that data centers have been around for a very long time. None of our phones would work without data centers and they didn't bother anybody 10 years ago and certainly not five years ago. And so I think it's important to have a differentiated conversation that at least takes into account what are you using this for? >> Are you you know so I think we had a lot of negative impact by social media and I think we all sort of saw that. I mean, you know, it's I think it's reasonable

12:49 to expect that something similar might be happening with the way that we go about it in AI and it's largely because the players are the same. >> I mean, there is the sense of sort of fool me once, right? >> That's right. >> Yeah. So, >> yeah. So, my audience, I keep steering toward enterprises because my audience is primarily enterprise IT leaders, enterprise IT buyers. what should they know about AI regulations, sovereign AI, you know, that might be coming down the pike that they don't necessarily know today. >> I think that uh it's it's very important to understand that if you're using somebody else's uh service, then that that typically means that you're trusting them with your most valuable data. >> And I think it's not to say that they're

13:37 not going to be good stewards of your data. It's just not you who's going to be that steward. And so the core question that every buyer should ask themselves is what degree of trust am I comfortable extending to the entity I contract with? >> Because I can tell you when when you do get access to somebody else's model and you start training fine-tuning that model with your own data, you're not doing this because yet you know it's not just for fun, right? you're you're actually trying to improve missionritical processes and accelerate your business. And that typically means the data that you are now handing over is highly sensitive, highly proprietary, highly confidential and contains competitive advantage for somebody else

14:29 if they were to get it. And so looking at some of the recent news with data leaks and you know escaping you know the the the sandboxes >> I just think there isn't a lot of trust to go around right now. >> No certainly not >> sovereign is a way to avoid that. It is a dedicated space. If the state is in the data center, then you know that a specific regulation and compliance bar has been met. And you also know that the state has basically nothing but confidential data. I mean, virtually any of the planning that's required to do literally anything requires access to PII, you know, driver's license information, you know, child support payments and who hasn't issued them, you know, things like that, right? So, these are all

15:25 parts of the data set that are available to the state. And so if you if you think about taking custody of something like that, if you measure basically the trust that the state's now extending that then you can sort of get possibly get more comfortable with the fact that you're that you're also trusting that that that data center or that innovation center or that company with your data. So I would I would just encourage everybody to think about very very carefully about where their data goes and what they do with it and what the impact would be if there was a leak. >> Are states necessarily or other sovereign entities necessarily comfortable you know having non-s sovereign entities share that data center? >> They are

16:08 >> okay >> actually they derive trust from it. So it's it's a little bit you know who's on first. So yeah, so what's happening is is that because this is a zero to one problem, they don't really know how to cross that bridge to go from no AI and now I'm using AI. Yeah. In in in a a good fashion, >> they seek to be in good company. M >> and so what we have found is if we bring in the right amount of in regulated industry the right amount of seriousness into the data center as peers who are also in there often times frankly in a collaborative fashion where the research university may already be in a project uh with one of these industry partners. So having a joint data center often times is is of advantage. And then of course you have

16:57 you have purely you know dedicated let's say defenseoriented data centers that aren't you know there's certainly not going to be a commercial aspect to that >> right and are those you know research institutions are those data center projects meeting with public backlash the way data centers are in some communities? >> We have not seen that if we're framing it correctly. We have not seen that. And part of it is also we're not really calling it a data center because it's more than that. >> Okay. >> So there is a data hall in there, but we're calling it an AI training and innovation center. I'm being fairly open about this because everybody's going to figure out that there's GPUs in there. But yeah, the you know, but it's it's

17:39 not just slay of hand. M >> what we're doing here is we're really saying look there's we all learned how to build airports at some point and we messed up the first couple and then eventually we figured out where to put them and I think with data centers is pretty much the same way. We're we we're making all the mistakes right now because we're probably moving fast and breaking things and so that's that's what we're doing and and it has an unfortunate side effect and I wish we didn't do that. At the same time, you can apply urban planning principles to figuring out how to create something that actually drives workforce development and then saying, "Okay, look, if this is the place where you send your son to become a data

18:25 center certified electrician and then makes 30% more, it's a different proposition than than just saying that's just Facebook, you know, >> it's a different conversation. What do you think are the mistakes that are being made right now and how do you see them kind of being corrected? >> Transparency. I think that we're we're the industry is not very transparent about what they're doing. >> Mhm. >> Um in general, I mean, they're not really transparent about the the data center plan. They're not really transparent about what they use these GPUs for and what they do with your data when you give it to them and so on. So, I think there's just a lack of transparency. Um and and this is I actually don't know if it is intentional or unintentional.

19:08 In many cases it might be a combination of the two to be honest. They don't see themselves as the bad guys, right? So they don't necessarily feel the innate need to disclose and also there is a you know there's this notion that if I disclose too much then I'm just giving away my advantage right so >> there's there's a lot of dynamics here that I think play into this role but uh that's you know what a sovereign data center doesn't do it is fully transparent about what it is that you're getting and what happens and where it is and here's your compliance and here's the endorsement and here's the letter from the state and >> you So >> there are also ways that the conversation about AI is turning from capability and you know no matter the

19:51 cost and how to make it workable from an ROI and efficiency and you know realistic sustainability perspective. So are there ways that people are using or setting up data center infrastructure right now that might get refined to be more efficient in terms of GPU use or cooling power? >> Yeah. all the time. It's an engineering problem. >> Mhm. >> It's not that nobody's working on it. It's just that it's very hard. >> Yeah. >> So, you can you can talk about immersion cooling for example, >> which would be incredibly efficient and likely a better solution. Uh but also you're losing a lot of density in the chips because you need to set up these big tanks that require clearance and you know and all that and machinery to lift

20:40 the the equipment out and you know there's like a whole lot of overhead there and so now all of these projects become not financable anymore and you know you're having to deal with just a lot higher capex in order to even get started. And so there's a lot of aspects to this that aren't visible to to the public. But I think I don't know anybody who isn't trying to figure out how to make this more efficient. I mean lots of strides have already been made. Closed loop glycol based uh cooling for example operates at a much higher efficiency level than air cooled data centers. It's also much quieter >> you you know. So there's I was at a data center ironically in uh Pennsylvania just a couple days ago and crickets were louder.

21:27 >> What about at the higher levels of the stack in terms of you know GPU provisioning or you know a lot of the server vendors are coming out with these rack scale systems that they're claiming pack so much more you know power into a lower footprint. I mean do you see that being you know meaningful in terms of shifting these conversations or is there still work to do? I think there are efficiencies that you can gain from an engineering perspective if you build a scale up system. You know, you can leverage more. >> Uh you'll see efficiencies being made at the system level and at the rack level, but those are likely coming from folks like Nvidia. I mean, if you look at the the the advantage the advances that they've made in in the rack scale

22:13 design, it's now about basically eliminating points of failure inside the rack to make sure that they stay up more predictably and uh to not cause so much pain on the data center where you have essentially almost like a clean room requirement in the data center for those things to work. And you can imagine that is high maintenance, high cost etc. So like the more these things uh progress the better the world will be in terms of uh the data center efficiency and so you know I I would expect there to be a lot of progress made. >> Yeah. And then you know speaking of Nvidia is there concern among sovereigns about just how much of a strangle hold Nvidia has on the AI infrastructure market and 100%. >> Yeah. Yeah. >> 100%. Every single sovereign initiative

23:08 that we've been involved in has a requirement to be heterogeneous. >> H okay. Is there anybody that can even furnish the capacity to to compete or to make a dent in their market share? >> Probably not. I I mean I'm not right, >> you know, but this I mean this is certainly I mean look, Nvidia is is absolutely dominating the space. there is no way around this and and so right now that's just what it is. >> Yeah. >> But that doesn't mean AMD doesn't have reasonable GPUs for certain use cases and some of those just are a technical fit. Uh right. So you have higher floatingoint accuracies in in AMD uh performance in that space that they simply focused on whereas uh Nvidia has been focusing more on the AI training performance and the inference

23:57 performance and that's just what it is. I mean that's an engineering reality and you know some just need to do >> hypersonic flight research and not AI model training and so that means it's a different it's a different use case but also look this is an ever evolving space I'm pretty sure that >> you know there's going to be new chips there's going to be CPUs there's going to be LPUs there's going to be GPUs there's going to be QPUs any letter of the alphabet in front of PU and they will I'm sure do all great things and you and it'll remain a point of competition. None of this is static. It's just how it is right now in August of 26. >> What about on the model front? I mean, there's a lot of debate right now in the

24:40 US about openw weight models >> and their [clears throat] safety where where do you see that headed? >> Well, it's a step to to transparency, right? So, uh that is certainly a step in the right direction, but >> also you know the question is well what are they good for? and you know what can I turn them into and you know do they really serve every every use that that you know a regulated industry might even have I I will say that I don't think we actually understand what the industry and the market really needs because of this high concentration of GPUs >> you really only I mean think about the data points you have basically you know how does Jensen Sam Elon and Larry think about this is essentially [clears throat] the data

25:30 points of of every analysis that's out there because nobody else has GPUs to even talk about this or have an opinion, >> right? So, >> so a lot of what we think we know what a good new use of of a GPU is is really only how how they think about it. It's not necessarily how you know power companies think about it, how car companies think about it, etc. And so I I think this is an incredible opportunity to open that uh conversation up to a more differentiated conversation across all layers of the stack. >> More openness, more transparency. That's kind of the theme I'm picking up on here. >> Absolutely. And and then you know and honestly I I I think that also more use cases. >> I I just think it's it's almost boring,

26:15 you know? I mean, we're we're doing essentially a chatbot after the other, a coding assistant after the other. It's like, okay, I I would like to understand Alzheimer's better. I would like to understand cancer better, right? I I think there's plenty of use cases here and we're not really focusing on what I would believe and I would guess are use cases that are a lot more that would find a lot more support by the populace than, you know, the next cat picture generator. So what's what's holding that back? Is it the concerns about risk and sovereignty primarily? >> No, I I I think it's what's holding it back is is I would say it's a combination. So I think that certainly there's a huge financing aspect to this. So uh part of the pattern and when

27:10 you're trying to get going and I've now seen see this firsthand is that the sort of the journey that you engage on essentially is a one a single chicken and egg problem. [clears throat] >> It is a okay I will I will do this but only if somebody else is doing something else right and and so everybody's trying to derisk everything. And so what happens is is that everybody reverts to the thing that's the least risky and the least risky thing to do is to give the entire data center to Microsoft, >> right? >> Because you know they're going to be good for it, >> right? I mean the same is true for all the others, but that also just reinforces the same, you know, 10 companies in the world getting all the GPUs. So we're we're so that's a

28:00 big aspect of it. A second aspect of it is is that I think a lot of the uh the teams that are in these industries just don't know how to deal with a lot of GPUs and it's this you know we made this mistake 15 years ago with cloud and you know we've do we have to do everything is this really going to be very helpful right I mean I've seen people pretty high up the food chain come to me and say hey I used chat chatbt for the first time last week >> and so there's a lot of lack of exposure I would say at the leadership levels to really understand AI in their context not just in the generic context that the consumer would know or that they would know as a consumer but also like how does this help my manufacturing process how does

28:49 this help my research my R&D and so on >> right so the theory is that if if those risks can be addressed then maybe the use cases start to become um a little bit a little bit more expanded or or advanced. >> Yeah. More diversified for sure. >> Right. Right. Well, it's an important mission. So, I appreciate you sharing that and thank you very much for your perspective. >> Thanks for having me, Beth. I really appreciate it. >> Thank you for tuning in to IT Ops Query. To learn more about enterprise software development and platform engineering, explore our content on Informa Tech Target sites. Find us on YouTube at our channel ion tech. Subscribe to our podcast to receive the latest episodes as they drop. And if you liked what you heard

29:34 today, give us a rating and review on Apple, Spotify, or wherever you're listening. Thank you for joining us.