Episode 1: Understanding the key role of trust in scaling AI

AI success must be built on foundations of confidence, governance and trust.

Discover how leading organisations turn AI ambition into real business value by building trust into every stage. PwC’s Warren Tucker and Sage’s Kerry Sinclair explore why governance, security and human accountability are essential to scaling AI—and why the winners won’t just have the smartest agents, but the most trusted.
Apple Podcasts  Spotify

Transcript

Vik Iyer, CIO Marketing Services:

Hi, I'm Vik Iyer, CIO Marketing Services. Trust in AI is essential if technology investments are to reap rewards. It's vital that IT leaders have a precise understanding of what this means and why it is central to achieving scale. In episode one of the podcast, we're going to explore how businesses can deploy trusted AI with two experts at the frontline of this epoch moment.

They are Warren Tucker, Partner at PwC, and Kerry Sinclair at Sage. It's great to have you both on this podcast. I'm sure we're going to have a fascinating conversation. I guess we may as well start at a very general level: what do you think trust means for AI and technology more broadly?

Kerry Sinclair, Sage:

It's a very big question, isn't it? I'll start us off. I'm sure Warren might have some different views on it. But when I think about trust for AI and technology broadly, I think around how I think about trust in humans first. So when I think about trusted individuals, I know that they're capable, they're reliable, and I know them quite well. And therefore that intimacy and understanding them as quite an important dynamic to trust.

And of course, in a human, if you believe that they're acting in self-interest or maybe not to, you know, towards your best endeavours with them, it starts to dilute trust. And I feel exactly the same when I think about technology. So I need to know the company that I'm working with, that they're capable, that their services are reliable and I understand their brand values and that they're going to use my data and my information for my good and not necessarily just their own. So that's important to me. And when I think about Sage overall, trust is a really critical part of our brand.

You know, being nearly right is really not good enough when you're a CFO or a HR professional and you're maybe filing a VAT return or a tax return or paying your people. Being nearly right isn't good enough and being accurate is important. So if you think about our brand as an organization, we need to be accurate and give our customers confidence that we've taken care of their legislation and they're they know us to be reliable for that. And therefore the technology we deliver and the AI that we deliver for them creates that confidence for them because that's what their business is built on.

So for me, trust i it's an all encompassing word. And I think it's it reaches beyond just the AI and technologies themselves. It's about their brand and their values and how they deliver that. But Warren, I'd be interested if you've got a different view.

Warren Tucker, PwC:

Kerry, I think very aligned. And actually it's interesting, isn't it? I think we obviously as two organisations work very closely with Sage, both in a customer context and more widely. And of course our values or purpose, if you like, as an organization is to build trust in society and solve important problems. So very much at the heart of how we think about our own business is it's all about trust. That's trust in relation to the services we provide to our clients, the work we do.

For ourselves internally and also the technologies that we are in the process of helping organisations adopt and implement, including AI. So, it's at the heart of our own business. And I think I like to talk about trust in a slightly different way to the standard and I would say important areas of things like risk and governance and compliance. But think about it much more as an enabler of adoption scale. It's a critical enabler of scaling AI and ultimately therefore the ability to deliver business value from AI.

So I think still many organisations look at trust as some level of compliance exercise. But I like to think of trust as much more as about confidence. And I think that was echoed in your point there as well, Kerry. It's confidence that the AI will do what you expect. Confidence that it's secure, confidence that it ultimately complies with regulation and I'm not going to find myself on the wrong side of those kind of conversations.

And ultimately confidence that people that are going to be using it can challenge it and ultimately remain accountable. So it cannot be at the heart of transfer transferring accountability from a person to AI. And I think and that's always been the case, you could argue trust was at the heart of mass adoption of services like cloud, digital payments, online banking. So they've all gone through that wave of do I trust this technology to do what I know I need it to do, and AI I think is going through that same journey.

And so I think it's really about, as you rightly said, Kerry, it extends beyond technology. It's about trust into people, into the operating model, into the ways of working as well. And I think, you know, AI typically the barriers to AI scaling is not because the technology doesn't work, it's actually it's a question of whether or not the organisation and the people using it trust in trust it enough to use it and get value from it as a result.

Vik:

Warren, mean, can we build on that point? Because I think what we're seeing at CIO.com really is a lot of challenge around scale. So IT decision makers don't need to be persuaded to use AI. They don't need to be persuaded that AI is a vitally important technology in the medium term, in the short term, whatever. But how is trust actually impacting scale?

Warren:

Well, if I start off this time, as I see it, again the barriers to scaling are essentially how do I get this safely deployed across my entire employee base and across my entire suite of business processes. And AI starts to step in to as it gets used in different functions, whether it's finance, HR, customer service, sales support, other areas.

It's starting to move away from providing as in technology is moving away from providing a transactional system that enables work to be done to actually undertaking some of the work. So it's becoming a system of work and activity, not just of record and transaction and workflow. And I think as a result, you need to change how you think about scaling AI. And it throws up different questions, which the materiality that gets, of those questions becomes more important as they become systems of activity and action, not just of record with a human acting upon them.

So things like data quality become all important. Security becomes critical. The whole point around observability and explainability. So if I'm going to allow this AI agent to act and take an action, how do I have that audit trail around that allows me to understand why did it make that recommendation? And if I need to tweak and adjust that there are ways and means by which I can do that and in essentially empower my teams in those functions to take accountability of that agent almost as a as a team member. And all of that points to things like role-based adoption, role-based training, and evidencing that there's value there. And I think the challenge across all of those different dimensions from data quality, security, regulation, observability, all of those are challenges and questions that are new to organisations.

And I think those that are able to knock those down, are getting to scale. And I think the key stat that we've been using as part of our AI performance study is those that are getting there and scaling are genuinely capturing a pretty significant chunk of value, but few are managing to do it. So what we're finding is only about 20% of organizations are managing to knock those down to get to scale. And as it stands, they're capturing three quarters of AI's economic value. And so it's not necessarily winner-takes all, but it's certainly a winner-takes most for the companies who we're able to knock down some of those issues that prevent scaling and adoption.

Kerry:

What I'm observing as well, Warren, is confidence is quite quickly knocked in AI. Obviously we're all going through this adoption and trying to scale, but bad news travels really fast in this world. w we rarely see news headlines about an amazing AI outcome, but we quite frequently see bad news stories and they travel really fast.

As AI is shifting from generating just answers to taking action as you talk about it, them actions are on your behalf. and actually that means the stakes get higher for you personally. So your reputation is ultimately being delegated to some of these agents. So that personal judgment of your own has been codified. So there is an element of people put starting to become cautious with the more these bad news stories evolve. And I do think as an industry we've got a obligation to tell some more good news stories about the brilliant outcomes because I think that's how scale gets achieved. Of course, alongside all of the good things you've touched on with good governance, observability, etc. And building that into all of our programs is key. But that scaling gets risky when what we d only hear about is the bad news stories. Because it that personal judgment gets questions, right? When you lose trust around a

Warren:

I think spot on. I mean, part of the challenge we have there is of course, as we all know, social media in general, it's the bad news story generates more click through volumes of activity. The positive ones don't generate that level of interest. And actually, arguably some of the more recent high profile reported cases are fantastic PR vehicles for these companies who are all looking to try and elevate, if you like, the expectation of what these tools can do.

The reality is still, you know, far different. I mean, we're seeing again, I like statistics, you know, we're seeing eighty plus percent of CEOs in our recent survey are saying they're all in on AI, but only twelve percent are actually getting measurable value across efficiency, productivity and growth and revenue. So there's definitely the hype of the possibility and the risk and the danger definitely exceeds the reality. and I think there's also a point and it was tied up in your first point as well, Kerry.

People seem to still talk about AI as substitutional, you know, human or AI. And it's no, it's about capable, trained humans who understand specifically what AI tools they can be using to augment their teams and what they do to make them f able to focus on the higher order, higher value, higher consequence activities and remove some of that rope work that takes up time and ultimately then becomes a consumer of that brain power in areas that are not high value. And I think it's that reality of where you can add these things together that is the true picture of value. And it's interesting, I still think, Kerry, that in the world of IT, probably still the best use cases are in IT itself. We know that AI is used to accelerate IT development, IT delivery, IT design, but those designers aren't outsourcing code quality to some of these agents, they're actually just using it to get rid of the basic repetitive tasks that they would have historically had to do, and therefore they can focus on producing higher quality code faster and better. and they're not necessarily seeing a substitution effect. And I think that's an important part that again, like you say, Kerry, it just gets lost in the in the noise.

Kerry:

It does, We talk about it as a human in the lead, not human in the loop, because it's not just an FYI. It's a are you happy with this outcome and are you ready to go with it in this agentic flow of whatever it is we're doing? because I think human in the loop sort of suggests that you're secondary to the technology. so we've been trying to reformat how we talk about it in that way. But you're absolutely right.

Warren:

I like that.

Warren:

I agree, 'cause that's about a human accountability, exactly as you say. The human's still leading, owning, accountable, responsible. AI augments. It's not like you say, it's not the human inserting itself into check it every now and then. It's not a transference of accountability. I agree. It's interesting by the way. The one other thing I will say, Kerry as well that I'm seeing is still lots of organisations are just rolling out general purpose AI licenses to everyone. So here's, you know, insert A vendor here, have everyone have it. there's a massive difference between organisations that are doing that versus those that are offering curated role-based specific training. and I think there's an interesting trend. I think it's just under two times those organizations that are getting rewards are two times as likely to offer role based specific trainings. That which means to your point, the ability to put the human in the lead position and owning it, has to be in a position of defining and designing and understanding for their specific role, how and where and what LLMs and AI tools they use in what context. The absence of that role training just means everyone just uses it however they see fit. And I think that is where you don't necessarily get the highest and best use. and in fact you probably introduce more risk as a result of that as well.

Kerry:

It's interesting here where we're doing a little bit of both to be honest. not that we're expecting different outcomes, but the there is something about giving colleagues across the company in different roles just the opportunity to have a little play around and explore and build our own actual work and confidence with the tools and understand them a little bit. but you know, we are going through that approach of genuinely redesign then our processes of what work needs to be done with AI in mind and they're truly delivering the value and the outcomes that we want, I would say, versus the experiment and play around. But we've tried to take two lanes to it because it'll because of the I guess some of the noise in the industry, people are really nervous about what they can use for what purpose.

So, we're trying to give people just a w a wider purview of I guess what is p the art of the possible with some of that air exp experimentation.

Vik:

I mean, I there's so much dimensions, isn't there, to this trust debate. And it's been fascinating to just go through it and hearing you talk about the noise, but then also hearing about, you know, how the role of humans. But there's also another sort of dimension to this, and that's sovereignty, because it's becoming a major priority across both data and AI. So tell me how you define this, sort of emerging topic and what is the optimal strategic approach in your view?

Kerry:

That's a great but very large question, isn't it? so what so when I think about it, I think obviously you're talking about independent control and governance around your own AI stack. you know, and that's just such a vast array of things from data, the compute to models to power, water, residency, infrastructure, chips, and in even skills in the industry. And I think, I would say it's beyond my capability to decide what is the optimum mix. But I do think here in the UK certainly it would feel like a bit of an impossible task to be able to do that on our own without a fundamental shift in how we think about it. we're so dependent on so you know some of the big players, we all know who they are in the US and China and the likes. and I think it's about finding what's the optimum

Holistic and integrated plan really and mix of that to deliver the right mission it for our own destiny. I think it's such a vast topic to work out how we would solve it. But as a business who's using technology, no, I do just come back to my simple principles around trust, confidence and choice. We ground our decisions around, you know where our data's gonna be, who's gonna have access to it, you know, and what partners we use as part of that mix in a similar way. And for me it's about having trust and confidence and choices around how we do that.

Warren:

I mean I think I can I think that's spot on, Kerry. I think too many organisations and I think the conversation still sits around I describe them as dumb domain specific conversations. So sovereignty equates to a data centre or it equates to an L What choice of foundational model do I use? Or it relates to data location. And actually you're absolutely right, it requires full top to bottom understanding of the of the technology stack.

From and I by the way at the top of that stack is trust interestingly I put trust at the top of that stack. Then of course you've got applications, you've got the LLM derived, let's say specific models, you've got the foundational models, you've got the cloud and the data platforms that providing that, you've got the infrastructure, you've got the chip manufacturing, you've got the location and people who have that in ultimately where that location is, its resilience, its continuity, and again you've got power, land, cooling, everything else associated with it. So in some, kind of context there are debates that were saying that well sovereignty equals every single one of that layer needs to be geographically tethered and the full supply chain associated with it controlled. But in reality that is not a realistic prospect, particularly around when you get to layers like the chip manufacturing space as well. And certainly you see a view that sovereignty requires all of those things. I think exactly to your point, Kerry, I think my advocate what I advocate for clients to just think more differently about it's about having choice over those layers. So choice over data, choice over model, considerations of intellectual property. There's a regulatory standpoint. Obviously the position in Europe is very different where there's a much more stipulated set of regulations that need to be adhered to. Versus in the UK tends to be more commercially driven, commercial choice driven. But you've ultimately got to go through each one of those layers and ensure you have choice. I think typically seek to avoid, to greater an extent of vendor locking, or at least have that in mind. Think about resilience, business continuity, and how do you retain that optionality? And so it's not about owning every, you know, it's not about self build or kind of requiring geographical tethering of every single part of that layer, but it's about just making sure those strategic choices are understood and are weighed up and that it's going to be future proof for what you anticipate your business is going to require going forward. Very as you rightly say, care, a massive topic. And that you know, me on this podcast, very easy for me to say all that. Actually, working that through case by case, industry by industry, is clearly you know, very different depending upon the nature of you know, it's a defence business, is it a government department, or are we talking about a commercial product or retail business? Very different considerations against each one of those choices. But it's important to make those choices by layer and then reflect on what is the right the right path for you for you as a business and to retain that strategic set of choices, as you say.

Vik:

So there's a lot going on in that topic, I think. And we've also got agentic AI coming in, and that's going to see autonomous decision making becoming more frequent in the enterprise. So how does that impact trust?

Warren:

Kerry, did you want to give I'd be interested in your reflections on that, 'cause obviously, you know, y you guys are at the heart of offering products and services in which a genetic capability is now a core part of the value proposition.

Kerry:

Absolutely. so what we found with our customers is they're loving the AI that we deliver, but in a similar sort of our brand values really we've s you know partnered with our customers to say they're still in the lead. Often they're filing tax returns or paying people. so they need to be in control of that. So whilst parts of the process are truly agenic, we do have the right moments that matter for customers to be able to take the lead. and that's how I've thought about it from an IT perspective as well here in, you know, across our enterprise. what we do know from our customers with the AI that we deliver to help support them is they love it when it works. so that means we've got to have accurate data to help them save time. And we're not just throwing AI into the process without redesigning it.

We you know, it can't just be an afterthought that this gets you know, a little bit of efficiency. we've got autonomous regents running on processes today that the these processes are truly deterministic, repeatable, tested, and they've got governance around them. and we know that not all parts of the process are agentic. And when I think about how we decide what we make truly agenic versus what we don't, I always and it's been a bit of a theme today, but I always think about humans, right? So organisations of humans have been designed to operate for years and years. And there's clear governance and steps and how you organise yourself. there's certain points that you stop and have a maybe an approval or a sign off or an authority for good reason, right? That it might be for a fraud risk or a PO sign off between certain amounts. And We do that because history has told us that these steps are important. And I think about agents in the same way. it would be really easy to write agents that could do everything. But the question is do we really want that and what moments that really matter in this approach that we're taking? Do we want a little check or do we want an approval process or do we want the human to be in the lead? And that's how we're thinking about.

Kerry:

As we reimagine our business processes, of course we can take a lot of I guess heavy lifting out of what we do with this brilliant technology, but we're also not get not getting too carried away with how much we can delegate to an agent, because just because you can doesn't mean you should. So my question we're always asking is actually what do we do in a human process and was that designed for good reason?

You know, role based access, role approvals, they're really super important to the processes today. Don't get me wrong, there's lots of things we do that we don't want to be doing and we're using the technology to remove that. but that's how it gives me confidence actually in making sure we make the right decisions at the right points and what we truly make autonomous and you know them decision rights and what we delegate.

Warren:

And it's a massive topic. I you know, I agree with a lot of how Kerry described the ways to think about applying a genetic capability. And I think that you know that human in charge, in control, controlling the process is an important part of it. I think if I look across all of the different industries we're working with, one of the capabilities that we've invested in order to provide, a view on this, given what I see is lots of organizations doing, I would say, proof of concept, proof of technology, proof of value, but in a I would describe it as in a relatively isolated set of areas. And yes, there can be agents undertaking specific tasks, but the real value unlock comes when you look at coherence across an end to end process and how you could be looking at a gentic capability that allows you to string together a number of agents that actually makes them a more material impact to either growth or sales or cost out. And different businesses have different views to that. So one of the things we'll say we've invested in creating is we call it the orchestrated agentic operating model. As you can tell I'm not a marketer, it needs to be better branded. But it's essentially a tool that we're we can drop a we can drop a client's business model into the top of this and it will give you a view of where is their agentic opportunity within that business that is going to be both low risk, not necessarily disempower the human control of the process, but actually add some value in a way that is beyond just a proof of concept experiment and starts to look more horizontally across an entry process to get the value unlock. And I think that's a really important lens that because AI cuts across business functional boundaries, what we tend to find is you know, the finance department has a set of things going on. The customer service department has a set of things going on.

Sales teams might have a bunch of things. But the th the areas rarely hang together coherently across a concept of let's say lead to cash or book to bill, right? So it's how do I think more holistically around that? So that tends to be the gap. So I think that's an important part in terms of designing and you mentioned it there, Kerry as well. It's about designing for AI. You can't just drop AI into your current business as it stands today and expect it or a agentic capability and expect it to drive a performance unlock. You've got to redesign and decompose roles, processes, tasks and activities and then determine which ones of those can be agent enabled and equipped to be more efficient and effective, removing lower value work, but not control of the human, enabling that end to end process and the people on it to be adding the value of the more higher end task. And so I think that kind of structure is often missing because it's much easier to take an existing AI tool and agentic and do an experiment, proof of value, and see where it goes. And I think that's the missing sort of unlock of where agentic can really add value. And I and Beck, the last thing I would add i is just around obviously all of this points back to the trust question as well, just to kind of go full circle, is that, you know, agentic means it's moving from do I trust the answer that this AI age, or if you like this AI capability is providing to me you know the insights, you know, is it giving me the right data, can I trust the answer? To actually to can I actually then trust this system to act and do something for me in a way that means I have confidence that when it does act, to Kerry's point around things like deterministic outcomes, I can't have variability in a statistical model that means the answer might be different when I ask if I asked the same question five times, am I going to get the same deterministic answer every single time?

You need to have a level of testing and validation and verification to make sure that you can trust the system to act, not just trust the answers it's giving a human to interpret.

Vik:

I feel almost bad interrupting there because it's such a fascinating topic but we are unfortunately coming towards the end and I just want to give you both a chance very briefly if you can, just a final thought, just sum up, there's such an involving but such an important topic, just one final sum up from both of you please if that's possible.

Kerry:

I'll let you go first, Warren.

Warren:

So I would say it's important to frame the future not as humans versus AI. It's about humans leveraging and exploiting increasingly capable AI agents. And I'd say the organisations that get the most value won't be necessarily the ones that have the smartest agents. They'll have the organisation ready and equipped to operate with the most trusted agents.

Kerry:

Great. My mine's slightly different. I just keep thinking about this new world from new words, tokenomics, token maxim, value maxim. I just listened to Elon's podcast this week saying I AI is gonna take over human intelligence within five years. We had the sandbox escape, we've had the moonshot drop with Kimmy K three with three million three trillion sorry parameter models. The sheer speed of everything is obviously breaking boundaries and that comes with these massive infrastructure challenges and the scale up opportunities. I think we've got to ground ourselves in what can we get outcomes from fast and literally learn something new every day. It's an exciting ride, but it's important that we make sure that we get the outcomes we need and not just the overhead of AI. So I think I'm sort of supporting Warren's point. It's about outcomes for me that we need to get grounded in.

Vik:

That makes a lot of sense, And that does bring us to the end of episode one of this podcast, exploring that interface between trust and AI driven growth. My thanks to Warren and Kerry for coming on and sharing their expertise. Please do make sure you catch up with our other episodes as we explore this critically important issue. But for now, it's goodbye.