Why Enterprise AI Fails After the Demo | David Bauer

Vit Lyoshin (00:00)
Welcome, David. Thank you for joining me today.

David Bauer, PhD (00:02)
Thank you for having me.

Vit Lyoshin (00:04)
Yeah, of course. ⁓ So I just wanted to start by asking you, you're working with so many enterprise clients and companies and ⁓ what are things that they're doing in demos that are working and then in production it doesn't. Can you tell us a little bit about that?

David Bauer, PhD (00:23)
Yeah, I think probably the biggest thing that we see is that demos are designed for small-scale data. They work really well in almost like an offline mode. ⁓ When it comes to things like model training, model serving, ⁓ leveraging LLMs, et cetera, ⁓ those all seem to work fairly well in a demo environment with small amounts of data.

What we do that's a little bit different is we actually integrate with production data. Our platform is integrated with production data from the start so that what we're doing is training models, doing federated AI over data as it's coming into an environment such that as we're working through that process and developing things,

when it comes time to actually put something into production, ⁓ it's already trained and evaluated and tested on that production data and can go directly in. ⁓ So there's no additional setup that needs to be done. ⁓ All of that is kind of done upfront, connecting to the data set. And so everything that we're doing is built off of that production from the very beginning. ⁓ What I always like to say is,

Vit Lyoshin (01:44)
Okay.

David Bauer, PhD (01:46)
When people are approaching a project for the first time, if they're opening up a notebook and calling pandas read CSV, they're probably doing it wrong because there are no systems that output CSV data, right? ⁓ So if that's what you're doing, you're not working with production data and that's not gonna probably end very well for you.

Vit Lyoshin (02:07)
Right. Okay. And I also know that you kind of call this last mile AI problem, a problem in AI. ⁓ Can you explain this ⁓ last mile idea?

David Bauer, PhD (02:21)
Yeah, so as you know, we're primarily focused on federated AI, which is this notion of enterprises have maybe access in centralized environments, cloud computing environments, or other systems, ⁓ maybe 10, 15, 20 % of their data. ⁓ So if I'm developing a model and I go and I fetch some of the data from these different silos,

Vit Lyoshin (02:26)
Mm-hmm.

David Bauer, PhD (02:47)
⁓ In my enterprise and work with it on my laptop to train a model. That's great I may end up with a great model, but then putting that model in production, which is the last step You know, I go to my boss and he says yeah the accuracy looks good. The recall looks good the precision the you know, low false positives Let's put it in production. That's the last mile, right? but I can't really do that because the data is all over the enterprise, right it's in different places and often

Vit Lyoshin (02:58)
Mm-hmm.

David Bauer, PhD (03:17)
You know that data has compliance requirements associated with it. That's usually why it's in a silo to begin with ⁓ So with federated AI, you don't really have that problem. You're accessing the data as I mentioned It's already integrated into the production data. You're accessing it where it lives and then in our environment Because our environment is federated at the data layer at the ML layer Etc all of those elements that go into

collecting your data for training are also there. What we call a data recipe is what effectively we're building ⁓ that you use in production. So for us, putting a model into production is you just click a model and say, start serving it. And you can integrate it into your existing environment because you've already connected everything up before you even started devising what you were going to do with the model.

Vit Lyoshin (04:03)
Mm-hmm.

So ⁓ people might be familiar with like in general with idea of Federation. Can you explain what is Federated AI? So we get to understand what we're talking about here.

David Bauer, PhD (04:26)
Yeah, yeah. So Federated AI is all about leveraging enterprise data that's stored in silos, right? So the example that I always like to give is, because I think every company probably has an HR department, it probably has a finance department with, you know, contract information and other things. ⁓ HR has access to employee reviews or salaries or whatever it might be. ⁓ And then you have sales and

you know, CRM system. ⁓ So most companies, if they have customers, will have a CRM system. And what you want to do, right, when we talk about, we're going to pre-integrate all that data, what we're talking about is we're not talking about centralizing all that data in a cloud or a data lake or as we like to call them, we call them data swamps. What we're doing with Federated AI is we're emplacing a federate at each of those

⁓ data locations at each of those data silos. And then we're leveraging something called data level security to control access to the data. ⁓ It's similar to ⁓ data level security is attribute based access control. ⁓ It's similar to role based access controls, except ⁓ rather than controlling access to an entire row of data or an entire column of data, what we're able to do is go down all the way to the cellular level.

of the data. And that is really critical when you're bringing all these silos together and there's sensitive information in them. And you don't want to expose that to just anybody in the environment. And that's why you've maintained that data in a silo to begin with, right? Maybe it's healthcare information or banking information, or like we said, HR, finance and other things. So you want the protection of having that data?

Vit Lyoshin (06:12)
Mm-hmm.

David Bauer, PhD (06:21)
via its silos. The attribute-based access controls, the ABAC controls, allow for that. ⁓ And what that allows us to be able to do then is expose that data for model training, for model serving, ⁓ but also at the same time being able to control exactly who gets to see what. So different ⁓ model developers, different subject matter experts.

⁓ They're able to do things like construct the data set for modelers. They may have a wider preview overview over the data. The modeler is maybe not going to see all of the information that exists in that data set. ⁓ The model will, the model will then be able to train on that data. It'll be able to learn, you know, whatever patterns it's trying to learn. Maybe it's an LLM. Maybe you're not even doing model training. Maybe you're just serving an LLM model in your environment.

And you want that LLM to be able to have access to the silo data But it has to be done on an authenticated basis and that authentication includes not just the our back But then the a back to control what that LLM can see and so ultimately what that does is it allows in this catch-all term of federated AI It allows an organization to be able to bring all of that silo data together And use it

Vit Lyoshin (07:18)
Mm-hmm.

David Bauer, PhD (07:47)
for really whatever the AI use case is, whether it's training, whether it's serving, whether it's applying an LLM to it, which is mostly what we see. You can do that in a very secure way.

Vit Lyoshin (07:55)
Mm-hmm.

I see. So basically, the centralized system would not work for security reasons. Basically, that could be the main point why federated AI is better and more flexible because models can access only pieces of information. They have access to what is needed, not everything. Is that accurate?

David Bauer, PhD (08:24)
That's right. mean, imagine if at your company that you work at, they said, okay, we're going to centralize all the data. But you know, what I always tell people is envision a room and you're going to take all your documents, you're going to print them out on paper and you're just going to dump them on the floor in the middle of the room and let the modelers go crazy with it. Nobody's going to do that, right? That's what a data lake is. That's why we call it a data swamp because, ⁓ you know, trying to implement all the controls that are necessary as you're centralizing the data.

It honestly doesn't make sense because one, you're duplicating all the data. So you have the cost of data duplication. ⁓ Two, there's the time involved with transferring all that data. Some of our customers have literally exabytes of data and couldn't duplicate it if they wanted to. But there is, you know, the exabytes case kind of highlights the amount of time it takes to transfer that data to a centralized location.

Vit Lyoshin (08:56)
Mm-hmm.

David Bauer, PhD (09:19)
then you need to do things like normalize it, unify it, cleanse it. And of course, when you start to do that, that data moves away from what it looks like in production. And so it's not useful for AI, right? If you're normalizing all of your data, ⁓ let's say to a particular, maybe it has timestamps in it, you're normalizing them all to UTC.

Vit Lyoshin (09:30)
Mm-hmm.

David Bauer, PhD (09:43)
when you go to put a model in production, how is that model going to work if all the timestamps in production are in a particular time zone? So that's just a simple kind of use case to illustrate, again, how that last mile can kind of bite you a little bit. So keeping the data in its system, making that data accessible over the network through federated AI, through a federated data layer, and then ⁓

Vit Lyoshin (10:01)
Mm-hmm.

David Bauer, PhD (10:12)
So what we do is rather than moving the data to a centralized location, what we'll do, for example, in the case of model training, is we'll move the model to where it needs to go in order to be trained on the data. And so ⁓ federated model learning ⁓ is one category of federated AI. What we're in fact doing is training the model in all these locations and then aggregating the results at a centralized location.

It's a product called federated averaging. And so we average the weights and losses together and then distribute it back out and do another epochs worth of training. And in that way, you're able to get a model trained across all the data without any data movement at all. So you eliminate time, you expense, you eliminate infrastructure, eliminate... I mean, the time aspect is really the biggest one.

Vit Lyoshin (10:40)
Mm-hmm.

Mm-hmm.

David Bauer, PhD (11:09)
I've had customers, enterprises that we've worked with where I ask them, how long have you been building this data lake? And they frequently will tell me, it's been a couple of years, and it'll be ready in a couple of years more. And I ask them who the consumers are, and they say, yeah, nobody really wants to work with it because we've normalized away all the data. ⁓ And it's too far away from production to be useful in models. ⁓ It doesn't relate back to the source system.

Vit Lyoshin (11:34)
Mm-hmm.

David Bauer, PhD (11:39)
⁓ Sometimes people will spend a lot of time building multiple layers of abstraction over the data, ⁓ but then they can't give access out to most of it. So it's hard to train a model when you can't access the data. a lot of many different problems with that approach that federated ⁓ AI solves.

Vit Lyoshin (11:59)
Yeah, I see. Are there any trade-offs also when implementing federated AI that people should keep in mind?

David Bauer, PhD (12:07)
Well, there is, right? mean, the whole threat surface changes, right? One of the things that I didn't mention was the whole security aspect. I mean, how many times have we seen an article in the paper about, that's how old I am, I still read the paper. But, you know, I get the paper, read it online. But, you know, how many times have we seen like Snowflake got hacked, right? Or Amazon buckets that have been made public or

Vit Lyoshin (12:16)
Mm-hmm.

you

Hmm.

David Bauer, PhD (12:36)
You know something along those lines right when you put all of your data into a centralized location Especially if it's sensitive data, right it becomes a massive target for you know Hacker or an insider threat. It just becomes accessible to too many people We have federated AI that threat surface changes now if we're talking about doing federated model learning ⁓ Where you now you have to protect the model?

Vit Lyoshin (12:39)
Mm-hmm.

David Bauer, PhD (13:06)
Because as we all know, if I give you access to my model, I can actually reproduce the training data from nothing more than the internals of the model itself. There's no actual data stored inside of it, but there are definitely scientific techniques by which I can ⁓ reconstruct the data used to train the model. I think some people have seen that with LLMs, right? They'll give some test prompts and it'll come back with

you know, whole chapters of existing books, for example. So that would be a good example of reproducing the training data. ⁓ So you have to protect against that. And we use a multitude of techniques to protect the model, both on the inside as well as from adversarial attacks as well. ⁓ So for example, that federated averaging where we collect all the weights and measures, or I'm sorry, all the weights and losses and average them together.

⁓ So we use a multitude of techniques. There's secure multi-party communication. There's differential privacy. ⁓ We fuzz the models so that you can't reconstruct the training data using garbled circuits and oblivious transfer. And then we also implement homomorphic encryption so that the model weights and losses aren't ever even decrypted. They're encrypted in memory. They're encrypted over the wire.

And so even though you are centralizing the model for federated averaging algorithm, it's never ⁓ in an unencrypted state. ⁓ So yeah, that's probably the biggest thing is your entire threat surface changes.

Vit Lyoshin (14:41)
Merci.

Mm-hmm.

I see. And then ⁓ is there any differences in, we saw you mentioned a few times training the model on the enterprise data, but then we also have cases when they just using LLMs to retrieve knowledge from their own data, right? For training, for onboarding, for all sorts of things. Is there a different approach in setting this up for training versus just like rack, I guess, cases?

David Bauer, PhD (15:14)
Yeah, LLM accessing data, agentic workflows, ⁓ being applied to the data across the enterprise are really a major aspect of what we do. ⁓ And that federated data layer that enables federated model learning is extremely useful in that case as well. So as I've already mentioned,

Vit Lyoshin (15:29)
Mm-hmm.

Mm-hmm.

David Bauer, PhD (15:38)
You know, the data level security. we come out of the intelligence community. That's where we've worked for the last 20 to 30 years. ⁓ my co-founder and I, and we have built these systems over time, ⁓ to operate on highly classified data. And so at its foundation, we have brought all of those principles with us. So there's all the certification and compliance requirements you have to make for cybersecurity. That's good.

Um, but that's only a part of the process. There's also the notion of the defense in depth. Um, we actually coined that term at the NSA back in 2009. There's actually a requirement when you're working on classified data, you have to do it inside of a skiff. Um, we all wanted to work from home. And so the way that we did that was we said, Hey, we're going to implement a new way.

of ⁓ securing things and that is we're going to secure the data throughout its life cycle. So everywhere it's touched. It's a very GDPR type approach to ⁓ securing the data in that you're really looking very closely at every point in the data life cycle, where it's touched, where it's viewed and protecting it each step along the way to the best of your ability. ⁓

With that in mind, ⁓ that is what we are doing in our systems is we have developed a base layer, what we call the unified data space, that allows us to unify the data across all the different federates, but to do so with data level security using attribute-based access controls to control data down to the cell level. And for us, a cell is, if you just think about like an Excel spreadsheet,

It's just what it sounds like. It's a cell in the spreadsheet. And I can make it so that ⁓ you can see a paragraph or a sentence in a document, and I cannot. That's what we're talking about. Something in the intelligence community we call portion controls. So being able to filter data to that level on the server side and control access to it on the server side is the key to what we're doing in our agentic workflows with the LLMs.

Vit Lyoshin (17:33)
Mm-hmm.

Hmm. Mm-hmm.

Wow, that's pretty secure, I guess, if you get into that level. Wow, interesting. Do you have a couple of examples where this approach helped for real businesses, maybe without names or specific companies, but like a use case type example?

David Bauer, PhD (18:22)
Yeah, we're partnering with a company right now ⁓ named VRS. They're doing a smarter contract ⁓ type effort ⁓ with financial institutions. And ⁓ essentially what we're doing is we're leveraging ⁓ that federated data layer along with data fusion and semantic lensing to be able to identify at risk individuals within the banking system.

Vit Lyoshin (18:41)
Mm-hmm.

David Bauer, PhD (18:51)
And as you can imagine, these individuals, without going into a lot of details about what it means to be at risk, you can imagine that these are just simply people who really, ⁓ if their identities were known, it would be ⁓ very damaging to them at a personal level. So how does one bank verify a person they've identified as being at risk?

Vit Lyoshin (19:17)
Mm-hmm.

David Bauer, PhD (19:18)
And there are accounts at another financial institution. And how do we identify that we both have records pertaining to that without exchanging any information, without exposing any information? So we use the federated data layer, along with federated data fusion ⁓ and federated semantic cleansing, to be able to do that ⁓ in that environment. And so this is probably ⁓ the most

⁓ sensitive data that we've worked with ⁓ to date, ⁓ just because of the nature of the individuals. And I just want to be clear, we're not talking about like billionaires and exposing their portfolios, right? We're talking about, you know, like individuals who are at risk, you know, for many different reasons. So.

Vit Lyoshin (19:51)
Mm-hmm.

You

Yeah, I see. great. And then also if there's some organization out there and they'd like to start doing something like this with federated AI and stuff, is there any sort of framework or checklist that you have to how to get ready before start implementing this? Is there anything at all they should be preparing for?

David Bauer, PhD (20:31)
Yes So it my answer is it depends it depends to the degree at which you want to apply some of the techniques that I'm talking about so First of all what we will when we go into an enterprise the first thing we will say is Listen, you you really want to you have to take an inventory, right? You would be surprised how many enterprises Don't know where all of their siloed information is right?

Vit Lyoshin (20:43)
Mm-hmm.

David Bauer, PhD (20:59)
I can't tell you how many times I've talked to a CDO or a CTO and they say, I don't know, we might have a thousand databases, but I don't know where they are or how to get to them. So that's step number one. And then within them, it doesn't make sense to just take the approach of, hey, let's expose everything in them. It's perfectly valid to keep some of the data out of the Federation. So why expose everything?

Vit Lyoshin (21:20)
Mm-hmm.

Mm-hmm.

David Bauer, PhD (21:29)
only expose what you need to expose for the particular use cases that you have. So if it's an LLM accessing it and automating some sort of a workflow process, maybe only ⁓ expose what you need to expose. And then the third step on top of that is, and back to the paper again, what I always tell people is ⁓ if you're going to do data level security, the first thing you need to do is really take out a piece of paper and a pencil.

There's a very high tech process of determining who should have access to what elements of the data. And so that really means going across all the rows and all the columns, but also going down into the cells themselves. So if you think about, for example, a PDF document, ⁓ you could say in an RBAC system, might say, yes, Dave Bauer can see that document.

Vit Lyoshin (22:05)
Hmm.

Mm-hmm.

David Bauer, PhD (22:25)
But usually we want much more granularity than that, right? So we want to say, OK, he could see these paragraphs, but not those. He could see the tables, but not the images, or something along those lines. So ⁓ being able to really ⁓ outline that is a very non-technical process, right? It is a process of really literally using a paper and a pencil to outline, what do I need to protect?

Vit Lyoshin (22:46)
Yeah.

David Bauer, PhD (22:53)
and who can have access to certain things. ⁓ After that, after you've written it all down, then of course you can implement it in the technology in a very easy way. But ⁓ otherwise, you know, what are you doing? know, it's what we see people do in centralized systems. I don't know what to protect. I don't know who to give access to. So I'll just push everything. You know, those are the people who end up being really at risk.

Vit Lyoshin (23:03)
Mm-hmm.

David Bauer, PhD (23:22)
And the risk that they're taking is, or the trade-off they're making is, if I expose this to an LLM or if I train a model on it, I'm going to get something of value out of it. But the reality is the value is not as much in the model as it is in the data. And so that trade-off almost never works out.

Vit Lyoshin (23:22)
Mm-hmm.

Mm-hmm.

I see. Yeah, I can imagine people have to write all these like rules and policies who can access what ⁓ if they don't have it yet in place, that would be the challenge. And I think it sounds like many organizations have this problem. ⁓ So, it's even like from different conversations I have with other people, it seems like it's data. They have the data, but nobody really knows what's going on and who have access to what.

⁓ That's very interesting to see as a pattern. ⁓

David Bauer, PhD (24:19)
Scary to see as a pattern.

Vit Lyoshin (24:21)
Yeah, it is. ⁓ Another question I had is, and this is maybe not very related to this, but when I was preparing for conversation with you, I noticed that you did something with the, your team ended up doing basically the COVID data tracking for the White House task force. Can you tell us the story? How did you get into this and ⁓ like what did you help them with?

David Bauer, PhD (24:49)
So. ⁓

It was, so disease surveillance is something that I've worked on in my, in the academic part of my career since about 2006, 2007. When I worked at the MITRE Corporation and I worked for the White House National Communications Division on avian influenza.

Vit Lyoshin (25:06)
Mm-hmm.

David Bauer, PhD (25:19)
We did a lot of work on that over a three or four year period doing these really super high fidelity down to the person level ⁓ disease spread models and Really ⁓ what we had done at that time was look at the impact of different mitigation strategies to ensure that the US economy will continue to function during a Pandemic, ⁓ you know like a global pandemic event ⁓

Vit Lyoshin (25:44)
Mm-hmm.

David Bauer, PhD (25:44)
And so it's kind of funny when you look back to 2007, it was, hey, what if we turn off BitTorrent? What if we turn off gaming? You know, what if we turn off, you know, different things? We didn't really have video at that time. Video streaming wasn't that big of a driver. And if you think about it for a moment, you can probably imagine what some of the biggest drivers are for internet traffic. Right. So what the government really wanted to know was how do we keep the economy flowing?

That's online bill payments. Those are loan credit loan applications, right? Liquidity and credit. How do we keep those things going during a pandemic event if we tell everybody to go work from home? ⁓ Which is exactly what they did in COVID, right? They said, ⁓ everybody stay home. So COVID came along. I'm just going to pepper in a little bit of a personal story. ⁓ I was

Kind of paying attention, ⁓ being a member of that disease surveillance community, I started telling my wife back in January and February of 2020, hey, you're gonna think I'm crazy, but we're gonna go buy a few hundred dollars worth of over-the-counter medications. We're gonna, know, ⁓ yes, I got a couple cases of toilet paper and other paper products shipped to the house. My wife thought I was insane. And then by March, right, it was in full swing and everybody kind of knew what was going on.

Vit Lyoshin (27:04)
You

David Bauer, PhD (27:09)
And I said, this is going to be fantastic because they're going to tell everybody to work from home. And I get to spend some time with the kids and, you know, lay back a little bit. And, and, and then we get a phone call. And it is the Joint AI Center in the Pentagon. And they said, Hey, we came across some of your GAO studies and reports that you had done. And we think we could really use that for highly targeted mitigation strategies.

Vit Lyoshin (27:09)
Yeah.

Mm-hmm.

David Bauer, PhD (27:39)
⁓

so we, ⁓ brought that back, to life. We changed out some of the statistical modeling with AI models. ⁓ and we began working in the Pentagon with, ⁓ it was really kind of a funny thing because we're there with, Google, Microsoft, IBM, Amazon, you know, all these big players. And then there we are, this tiny little company that no one's ever heard of before. and.

but we were the only ones that actually had experience in the space. And we knew that many of the AI approaches weren't going to work. ⁓ and so, ⁓ and we had validated these techniques with CDC and MIT and other places over the years. So we really knew that our stuff was going to work extremely well. And more importantly, that it would be able to go down to that individual level. ⁓ but in order to be able to do that, you have to have scalability. You have to be able to.

Vit Lyoshin (28:31)
Mm-hmm.

David Bauer, PhD (28:38)
You know be able to protect the data right because you're talking about you know ⁓ Activities of real people how they move where they go to who's going to work who's not going to work who's going to the grocery store and church and other places ⁓ and The government was purchasing and leveraging ⁓ Demographic data so, you know, they they could tell you ⁓

Vit Lyoshin (28:43)
Right?

David Bauer, PhD (29:06)
about 15,000 different data points for every man, woman, and child in the United States. Really kind of a scary thing, the data that gets collected on this. So the government was able to purchase some of that and we were able to incorporate it. And the value of our models were that they were accurate out to 90 days. They were actually measured to be 92 % accurate or greater 90 days out.

Vit Lyoshin (29:14)
Yeah.

David Bauer, PhD (29:32)
Whereas all of these other players that I mentioned they were accurate, you know, no better than a coin flip Predicting seven days into the future What was happened and then of course? ⁓ You know, they're predicting at a statewide level at a county level at a zip code level Whereas we're predicting all the way down to a building level, right? So we are predicting how many people in a police department are going to be sick in three months, right? So

Vit Lyoshin (29:42)
Hmm.

Mmm.

David Bauer, PhD (30:02)
That's something you want to know how many people in a nursing home how many of the staff how many of the Residents are going to be sick in a nursing home in three months because What that allows you to do is it allows you to position? your PPE and your doctors your nurses your anestheticians Your ventilators and well, you know depending on the disease ⁓ It lets you move your resources around

Vit Lyoshin (30:14)
Wow.

David Bauer, PhD (30:31)
I think by the end of 2022, we were the number one model that was being used by Northern Command to do all of that logistic planning within the country. were the primary system for the US military. were using it. So let's say, for example, in Germany, in the EU, they wanted to...

hold a training event with thousands of troops in Poland. They would use our system and pull it up and say, okay, this is where we're going. What does the environment look like that we're sending troops into? And it was even used in the intelligence community to be able to do predictions for Belarus, Ukraine, Russia, and the impacts on their military. So if you think about this for a second, right, go back to 2020, this is two years.

for the Russian invasion. ⁓ And so, you know, that's what we're doing in the military domain is ⁓ looking at, you know, force readiness levels ⁓ and is that going to potentially delay things. So you could tell, you know, I think you could probably figure that out for yourself if it delayed things.

Vit Lyoshin (31:34)
Yeah.

Yeah, yeah, no, this is amazing. ⁓ Like, ⁓ I'm always interested in things like that, like predictive analytics and things like that and people's behaviors and how things can be done in the future for, like you said, for economic planning, logistics planning, for all of that, because it was a mess and it was scary. People didn't know how long what's gonna happen.

and to have some system like that, it's amazing ⁓ and can help sort things out and plan, at least short term, but still. ⁓ Yeah, that's amazing. Yeah. And I also know that you worked with DARPA and national security, in national security with agencies and stuff like that. And I just wanted to in general ask you how about...

how working in those high stakes secure environments is really ⁓ like for innovation specifically. How should I phrase that? Like how that environment with national security and DARPA and things like that is a little bit different in innovating in AI compared to like let's say Silicon Valley for example.

David Bauer, PhD (33:02)
Yeah, it ultimately it shapes you and it forms you so that when you build something, certain things are just baked in, right? The security aspects are just baked in. But then also, you know, when we develop products at Exonus, for example, right? Every product we develop, we develop to work in an air-gapped environment. Now, you may not think, you know,

Vit Lyoshin (33:17)
Mm-hmm.

David Bauer, PhD (33:31)
working without the internet, okay, that's something maybe only the US government might need to do or some other government might need to do. But that is not the case. Most financial institutions have offline data centers that they operate. ⁓ When you're talking about working with highly secure data, one of the best ways to secure it is to keep it off the network. ⁓ And so being able to work in those types of environments,

Vit Lyoshin (33:58)
Mm-hmm.

David Bauer, PhD (34:01)
right out of the box is a big thing. And then ⁓ also just thinking through the processes, right, ⁓ is very different, right? In an organization like DARPA, you have access to some of the world's leading experts in their field. And so they each bring, it's a very multidisciplinary environment and each contributor is bringing a very different perspective to

Vit Lyoshin (34:21)
Mm-hmm.

David Bauer, PhD (34:31)
the work that you might be doing. And so we really try to facilitate that and everything that we do as well. And that translates into, for example, interfaces that are highly collaborative, right? That, ⁓ you know, very much like Miro is an online collaborative whiteboard. Well, we bring all of that to the AI process, right? ⁓ We make it so that, for example, you and I could collaborate together. ⁓

Vit Lyoshin (34:50)
Mm-hmm.

David Bauer, PhD (34:58)
in the development of a new model in a very similar way to how you use Miro. ⁓ It also impacts how we do things like the semantic cleansing, right? The example that I give there is, ⁓ you know, everybody has a different notion of what a bad guy is, right? So maybe if you work for law enforcement, ⁓ you know, a bad guy to you might be a criminal, somebody who's broken a law.

If you work in the military domain, a bad guy might be the person who's your enemy. ⁓ You know, and if you are in, ⁓ you know, healthcare, the bad guy might be somebody who is, ⁓ you know, kind of going around doctor shopping, trying to get, you know, drugs for a work that purposes, right? So everybody has a different notion of what a bad guy is. And you have to be able to serve all those different people.

Vit Lyoshin (35:42)
Mmm.

David Bauer, PhD (35:51)
And so ultimately what that forces us to do when we do our development, when we create products is to build into them ⁓ these abilities that allow different people to be able to collaborate and work together and bring their expertise to the problem, of course, in a highly secure way, ⁓ but always thinking about data first and how someone is going to access that data. ⁓

So yes, DARPA is phenomenal ⁓ because you do get exposed to all these different concepts. And when you're repeatedly exposed to them, shapes you to think in that way and to incorporate those aspects into everything you do afterwards. I know I'm kind of going on a little bit, but ⁓ I guess the point I'm ultimately trying to get to is when we look at Cilif

Vit Lyoshin (36:38)
Yeah.

David Bauer, PhD (36:49)
Silicon Valley products were always amazed when we see like a press release this company got 10 million in funding and they might implement like one feature of One of our products right? And I'm like it just kind of blows my mind that you know They're getting funding for something that is just a single feature and what we're doing, right? so

Vit Lyoshin (37:11)
Yeah, yeah.

Yeah, it's interesting how it works, With startups. yeah, the story about offline systems reminded me ⁓ in my previous job at the startup, we did work for like oil rigs and cruise lines. And those guys definitely needed something for like...

David Bauer, PhD (37:28)
Mm-hmm.

Vit Lyoshin (37:32)
at least short-term, like 30-40 day offline systems where they can collect data and then come back to the mainland and synchronize it to the servers. So, yeah. Okay.

David Bauer, PhD (37:43)
Yes. And we work

in those types of environments. In the military domain, they call them D-Dell. They're distributed. There's denial. Sometimes they just purposely turn the radios off. ⁓ In some places, your network connection may only be a 4G cell phone connection. So it doesn't all have to be military. It's like you said, if you're working in the bacon oil field in ⁓ North Dakota, you're

Vit Lyoshin (38:03)
Mm-hmm.

David Bauer, PhD (38:11)
Now, work connection might go down for a day or two, but you still have to keep operating, right? So, yeah.

Vit Lyoshin (38:14)
Yeah.

Yep, yep.

Another thing I wanted to ask you is ⁓ we're moving towards this AI agents being at the working alongside us basically. And I'm curious, what's your opinion on... ⁓

how the systems like yours, federated AI and data and everything will work with those AI agents that will be doing something, some tasks and how does it, how is it gonna look maybe let's say five years from now.

David Bauer, PhD (38:50)
in the federated domain specifically or just in general?

Vit Lyoshin (38:55)
Well, just in general and then also if somebody have federated AI also.

David Bauer, PhD (39:03)
The reason why I asked that question is because we're constantly having an internal fight over, do I actually need Federation, right? With an LLM, if it can reach out and touch four or five different systems over the network, do you need Federation? And the answer is, well, you absolutely need Federation because how much of a database is an agent going to be able to consume? And when it's accessing four or five of them over the network, how much, you know, how much of the context are you going to use just trying to populate it with some information? ⁓ So.

Vit Lyoshin (39:23)
Mm-hmm.

David Bauer, PhD (39:34)
In our case in particular, we believe that Federation is actually much more appropriate. ⁓ What we're doing is we're ensuring that you have agents kind of throughout the entirety of the architecture, North and South and East and West, so that they can be both highly specialized, but then you can also have orchestration agents ⁓ that are orchestrating a workflow across those agents individually. And what that helps to do is one, it allows you to choose the right LLM for the task.

Vit Lyoshin (40:05)
Mm-hmm.

David Bauer, PhD (40:05)
All right,

so you can have some LLMs at a federate that are doing long running tasks. Maybe they last for like five, six days. You have different LLMs that are optimized to inference very quickly. You can have deep thinking or high reasoning agents that are really trying to sort through a complex workflow and orchestrate across all of the agents. ⁓ So those are. ⁓

things that are happening today. Your comment struck me as, you know, that's where we're going. I would argue that's where we are. And how are we going to work beside it in five years? ⁓ You know, that's like saying, do you remember a time before cell phones, right? Before a smartphone? You know, think about how fast we've accelerated from like 2000.

Vit Lyoshin (40:42)
where we are at,

David Bauer, PhD (41:02)
I don't know, when was first iPhone? 2007. So in 20 years, that acceleration, five years from now, it's gonna be like it's 2007 with mobile phones again. So.

Vit Lyoshin (41:05)
Yeah.

Yeah,

yeah. It's interesting how the velocity really accelerated with innovation, Everything used to take a year or year and a half for prototype. Now it's like a couple of days. It's amazing. Do you think with having all these AI agents at workplace and doing stuff at work, how does cybersecurity change with that?

David Bauer, PhD (41:47)
Well, I think we've talked about that quite a bit, right? I mean, you have to have data level security ⁓ throughout. You have to authenticate all of your agents. have to work. ⁓ This is kind one of the big questions in these agentic architectures right now is, ⁓ what persona do they take? How do you audit them?

and other things, right? Whose action are they acting on behalf of? So, you know, that's why we are an enormous fan of every prompt, every request, every data access is authenticated and auditable on the backend.

because ultimately you have to know who you're acting on behalf of. an LLM can behave, the same LLM can behave very differently if you task it versus I task it, if it's implementing a workflow for you versus me, just based on this notion of not just the RBAC controls, but also the ABAC controls. ⁓ And then what we're always very cautious to do is,

We not only are designing agentic architectures and harnesses and orchestration frameworks, but we also very carefully craft MCP services and tools ⁓ to do the very highly sensitive tasks to try to keep as much away from that LLM as possible. That's another way of kind of protecting your data. So it doesn't make sense to...

Vit Lyoshin (43:20)
Mm-hmm.

David Bauer, PhD (43:23)
have an LLM access a tool only to then aggregate the data itself, right? Have the tool aggregate the data and return the aggregated result if possible, right? So there are patterns like that that are beneficial for many different reasons. One, you're exposing less of the data. Hopefully, if it's aggregated, it's ⁓ maybe more anonymized. And then ⁓ the LLM is consuming aggregated data.

Vit Lyoshin (43:36)
Mm-hmm.

David Bauer, PhD (43:52)
rather than a large chunk of data that it has to aggregate. So you're preserving its context and using less tokens and ultimately it's just far more efficient. But you have to put that thought in throughout. ⁓ You can't just point and fire and forget. You have to really kind of think it all the way through. Does somebody really need access to that?

Vit Lyoshin (44:18)
Yeah, so comes down to, we're still gonna have to worry about this because these agents will have a specific purpose, hopefully. And I know with the rise of all this, like open source, yeah, I cloud just came out with something like last month and people are gonna try to play with it and give access to everything and just use it as a chat GPT as a general tool. But I think...

Yeah, this idea of specialized agents, think that's where it has to go. Otherwise, it's going to be really unprotected, unsecure.

David Bauer, PhD (44:58)
It is scary to think that every prompt, every file you upload, every image you paste into a chat, GPT or Gemini or a Claude or a mythos, ⁓ you know, are going back to these two, three, four, five major corporations. The data they must be collecting on all of us, on all of our companies and all of our personal lives.

This came up kind of recently. People are having a blast using chat GPT to help them file their taxes. You know, so great. You just gave them all your financial information, right? Every dollar you made and where it went and you know, medical expenses and it's just astonishing to me. I think I saw an article. Online about.

Vit Lyoshin (45:40)
Right.

Yeah.

David Bauer, PhD (45:56)
Elon Musk offering $420 for each employee's tax return so they could train models on how to do better tax returns. That's what made me think of it. So it's just scary how willing we all are to give up all of that information. security, yeah. mean, it's exciting what we're

Vit Lyoshin (46:05)
wow.

Yeah.

David Bauer, PhD (46:25)
have access to in terms of capability, but it's incredibly scary to think how much ⁓ of an advantage these handful of companies are being given.

Vit Lyoshin (46:38)
Yeah, I mean, I remember growing up and my parents were always telling me like if any stranger asking you where you live, you never tell them your address or even like approximately where you live. And then now I have the cell phone with my geolocation enabled and everywhere I go and everybody knows where I am. It's crazy how we got trained and brainwashed to share data and it's okay. Wow.

David Bauer, PhD (47:03)
Yeah, well, I

know I drive my family crazy because I don't do any of those things.

Vit Lyoshin (47:08)
Yeah,

yeah, yeah. All right. Okay. Thank you, David, for the insightful conversation. At the end, I usually ask my three innovation questions. And so I'm curious to know what your answers will look like. ⁓ Can you define innovation in a few words?

David Bauer, PhD (47:28)
Innovation in a few words. Yeah, yeah. think for me, innovation is, how would I describe it in a couple words? Like a useful surprise, right? Innovation is one of those things, it's kind of like it clicks, know, when you say it, you're like, oh, that's kind of cool. That's a good idea, that's a cool idea. But then,

So there's a surprising aspect of it, but then it also is useful, right? You're like, ⁓ wow, that's kind of clicked in a way. And to some degree, that's everything we just talked about, right? All these LMs and models and everything, they're super useful, right? And it's very surprising what they can do on both sides of the spectrum, right? Good and bad. ⁓ But yeah, I would say ⁓ maybe a useful surprise.

Vit Lyoshin (48:14)
Yeah, right.

Okay, great. And the second question is which innovation in the human history you think changes the world the most and why?

David Bauer, PhD (48:32)
⁓ What innovation? ⁓ Well, I pick, do I have to pick a thing or can I pick more like ⁓ an approach? ⁓ Yeah, I think, you know, being a scientist for me, the scientific method is the number one thing. you know, people talk about the printing press or a transistor or all these other things.

Vit Lyoshin (48:42)
anything approach could be yeah

David Bauer, PhD (49:01)
But for me, it's a methodology, right? It's an approach. It's an approach that we have been leveraging for a couple hundred years now. And it is what enables these exponential ⁓ improvements in technology over time, right? People talk about cars replacing horses, cell phones replacing landlines, AI replacing people. ⁓ But...

All of that is enabled not by a transistor, not by electricity, not by a book, but by a scientific method, right? Being able to build on human knowledge, either incrementally or through big leaps, using an approach that's reproducible, right? I can tell you, you know, this is how I did it. And then you can go and reproduce it on your independently is probably the most ⁓ innovative thing.

Vit Lyoshin (49:51)
Mm-hmm.

No, that's great, I've never heard that one before. Okay.

David Bauer, PhD (49:59)
Okay, good. was nervous that everybody would say that. That's what

occurred to me at the end. like, probably everybody says that.

Vit Lyoshin (50:09)
Yeah, and the last question is which one thing that we're using today and it could be software or just anything. We will be laughing at 10 years from now.

David Bauer, PhD (50:21)
that's easy. That's easy. Chatcha PT. This notion that you're going to be typing, like I know it's the new thing. Everybody's doing it. Your grandma's typing it into, you know, doing some prompt engineering, ⁓ you know, trying to get a better recipe or something. But yeah, what did you say? 10 years from now? Five years from now? 10 years from now? Yeah, this notion of being the typing things in, we're all going to laugh at that.

Vit Lyoshin (50:26)
You

Yeah.

10 years, yeah.

David Bauer, PhD (50:48)
⁓ I think we kind of touched on that a little bit earlier, right? The pace of innovation in this realm, it's moving so fast ⁓ that this notion of Google searches being replaced with AI, that's going to be quaint, right? We don't even know what these new systems are going to look like yet.

Vit Lyoshin (50:48)
Yeah.

Mm-hmm.

Mm-hmm

No, that's great. Yeah, I agree with that All right, David. Thank you very much for your time today. It was really nice talking to you

David Bauer, PhD (51:18)
Yeah, it was very nice talking with you. I really enjoyed it. Thank you.

Vit Lyoshin (51:22)
Thank you. Talk to you later. Bye.