Transcript: Drive governance and accountability to create value from AI innovation
Governance and innovation need to co-exist, but that means building in controls from day one and taking a risk-based approach.
Discover why strong governance is essential to turning AI innovation into meaningful business value. Costa Coffee’s Katy Wallace and PwC’s Claire Reid explore governance by design, risk-based controls and human oversight and explain why data quality, security and cross-functional leadership matter even more as agentic AI increases the need for accountability.
Transcript
Vik Iyer, CIO Marketing Services:
Hi, I’m Vik Iyer, CIO Marketing Services. AI transformation promises growth and innovation, but that can all be undone without strong governance. It’s vital that IT leaders develop a strategy that uses humans in the right way to ensure AI oversight is a value driver.
In episode two of the podcast, we’ll talk with experts who are working on these very issues. They are Katy Wallace, Chief Digital and Information Officer at Costa Coffee, and Claire Reid, Chief Technology and Innovation Officer at PwC UK. Welcome to you both.
I’d like to start by looking at how AI governance has evolved so far. There’s a growing sense that the so-called “human in the loop” approach can create a false sense of security. Does that resonate with you both?
Claire Reid, Chief Technology and Innovation Officer at PwC UK:
Absolutely. Hi, Vik. Thanks for having me on.
I think the first thing I’d say is that we are definitely not where we need to be when it comes to governing AI. In fact, humans have not yet worked out how to work with AI properly. It’s amazing how a false sense of security can come with output generated by a technology platform.
In my view, it really is back to basics when it comes to looking at how we use AI in day-to-day workflows and rely on output from these models. We cannot underestimate the amount of governance, security and rigour that is yet to be built into how we manage the risks sufficiently, which I’m sure we’ll come on to shortly.
Katy Wallace, Chief Digital and Information Officer at Costa Coffee:
I totally agree with you, Claire, and thanks for having me on too, Vik. Your point resonates with me.
We’ve learned that simply having a human review an AI output — having a human in the loop — is not enough. I agree with Claire that it can give us a false sense of security. I remember sitting in many AI conferences and presentations where you’d see a slide with a “human in the loop” box on it, and that would make you think, “That’s enough. That’s fine. We’re safe and secure.” But it’s not enough.
Humans bring judgement, business context and accountability, but they can also be prone to automation bias. If a system continually produces plausible answers, people can trust it rather than challenge it.
At Costa, we’re thinking about governance by design: controls, data quality, governance and security right from the beginning, building these into the process before applying AI rather than expecting a human review at the end to be sufficient.
A Costa example from my day-to-day world might be an AI-powered store assistant that helps a store manager decide on the next best action, such as ordering a product or changing a work schedule. You’ve got a human in the loop making that decision at the end, but if you don’t have quality data coming in at the beginning, properly integrated systems, and controls and governance built into the workflow and process, then the outcome will not be good enough simply because you have a human in the loop at the end.
Vik:
Thanks, Katy. It’s really interesting to hear what’s happening at Costa.
Perhaps IT leaders need to think more about a “human in the lead” approach to ensure a more precise and effective model that protects the business while keeping innovation flowing. How do you effectively tread that path?
Claire:
I’m happy to jump in on this one. I would say the two can definitely coexist: innovation and governance or control. Without well-thought-through governance around AI, I don’t think you can achieve the innovation that will result in value creation or the intended outcomes.
You used the term “human in the lead”. I like that. I sometimes call it “human in control”.
Taking a step back and looking at Katy’s example, where you have high-risk or business-critical workflows, you absolutely would not allow AI to be fully autonomous. That’s where you need the human in control or the human in the lead.
You need to understand where the touchpoints are that require extra human involvement, judgement and another set of eyes before something goes out. At the other end of the continuum, you might be building AI into a much lower-risk process and can operate with less human involvement. Let’s face it, we don’t have enough people to be involved in every workflow at every critical point.
It’s about designing the right control points using a risk-based approach for the organisation. That gives the organisation confidence. The clients we see leading from the front have governance and risk decisions built in from the start, as Katy mentioned, and they have also addressed these issues at the right senior level of the organisation.
Katy:
I also really like the phrase “human in the lead”. I think it’s more sophisticated and considered, and it leads to a stronger governance model than the “human in the loop” concept.
It reinforces that organisations and leaders remain accountable for AI outcomes. AI can help accelerate a journey and an outcome, but humans need to determine the destination and where we want to get to with each AI use case.
In my world, if we were using AI in workforce planning, the human in the lead would define the hiring strategy, budget and overall workforce plan, and AI would help accelerate the journey to get there.
In customer service, which is a prime AI use case, the human in the lead would need to define the experience we want consumers to have when they interact with the business and the standards we expect.
I also agree with what Claire said about the risk profiles of AI use cases. Humans need to define those profiles because not all uses of AI are equal. Using AI to support the productivity of team members or colleagues in internal processes and day-to-day activities has a much lower risk profile than something consumer-facing that could affect brand reputation or commercial decisions.
Different risk profiles require different levels of governance and control. Not all use cases are equal in that respect.
Claire:
If I can build on that, Katy, we’re recognising within PwC and among our clients that this is fundamentally changing the skills and job profiles of our people. That ranges from the partners signing reports or work outputs for clients to the more junior people carrying out the day-to-day work.
We’re having to rethink those workflows and ensure that we train people to do their jobs differently in the world of AI.
People will bypass additional layers of governance and control. This isn’t about slowing things down or stifling innovation. It’s about ensuring that checkpoints are commensurate with the risk and aligned with how these new processes operate.
Humans are smart and know how to bypass controls. I saw that many years ago when I worked in risk, audit and assurance. Those tricks of the trade will develop in the world of AI as well.
We need to design control points that help users of the system rather than hinder them. That’s something I would add to the conversation.
Vik:
We’ll take a short production pause now.
I think that’s a really good point, Claire. It would also be useful to examine how this is playing out in business. What lies behind the businesses that have achieved success?
Claire:
The first thing I would say goes back to something Katy mentioned earlier. There is a difference between using AI as a productivity tool to save time or costs in day-to-day workflows and using it to deliver commercial benefit.
The organisations that are winning and standing apart are thinking about the commercial benefit. It’s not a cost play; it’s about the outcomes the business is trying to deliver.
In Katy’s world, that might mean attracting more customers or creating greater brand loyalty to Costa — please correct me if I’m wrong, Katy. The question is how AI can be used to achieve those business outcomes rather than simply removing people from a process.
That is where we’re seeing a real difference and a widening gap between organisations that will benefit and drive better business outcomes and those that are simply trying to automate what they do today and take out costs.
The successful organisations are rethinking how their businesses operate, where the touchpoints are, where the risks are and where the opportunities lie as their businesses transform. I see many clients blindly going down the automation and cost-reduction route because they don’t know any better.
Katy:
I agree. For me, the biggest thing that will help us be successful at Costa is connecting AI to real business problems, which is also what you were saying, Claire.
AI for the sake of AI, shiny tools, over-engineering or investment in the wrong places — where it isn’t connected to a real business problem — will not produce the outcomes we want. We see companies spending huge amounts on AI and then having to roll it back or failing to achieve the outcomes they want.
One of the simple principles for success is focusing on value before technology. I come from a non-technology background, so I spend time with my executive colleagues and teams across the business thinking about the real customer challenge we’re trying to solve, the team member pain point, the productivity opportunity or the commercial outcome we want to achieve, and then working back from there. The technology and AI come almost at the end of the process.
Secondly, trust is critical. When I talk to other CIOs and organisations, trust is one of the key themes. Trust can be lost in moments and take years to build, and that’s very important for us at Costa.
We need to ensure we have a responsible AI framework, a consistent and well-governed approach, strong identity controls, good data and all of those foundations. When we implement AI, either in pilots or across the business, people need to trust the outcomes, the process and what is happening with their data. If we don’t do that, trust could be lost in moments and take years to rebuild.
Vik:
Absolutely, Katy. We’ve seen examples of that.
We have enough time to look ahead to the future. We’re seeing the rise of agentic AI, and it is beginning to embed itself in a number of enterprises, possibly yours as well.
How do you expect oversight strategies to mature as this technology accelerates?
Claire:
One of the first things we’re starting to see — and something I encourage our clients to consider — is the importance of having the right stakeholders around the table when developing oversight strategies.
We have to move away from siloed ways of working, where the technology function might be separate from the commercial, business or operational functions. AI doesn’t allow that model to work; it turns it on its head.
You need cross-functional leaders around the problem and the strategy. Otherwise, siloed working will create friction or tension in the organisation and slow down progress.
The CIO, CTO or other senior technology leader needs to be part of conversations about the organisation’s strategic direction, ensuring that the right voices are represented in decision-making forums.
That will become even more critical with agentic AI, which, in my view, is a horizontal, cross-cutting and transformative capability. If you get it right, it can help an organisation transform. If you get it wrong, you may waste a lot of time and money.
Technology teams can be drawn to building shiny new tools, while the business may want new gadgets without a genuine commercial or market imperative behind them.
Katy:
Agentic AI changes the game completely in terms of oversight strategy.
With generative AI, large language models and other types of AI, systems generate content and make recommendations. We need to check, oversee and challenge those outputs before using the information to make decisions.
In the agentic world, agents are doing things in real time. They are taking actions on behalf of the business, making decisions, triggering workflows and interacting with other systems. The oversight required therefore has to be much more mature.
Before we use agents, we need to be very clear about what an agent is allowed to do, what role permissions it has and when it needs to escalate to a human.
We also need to determine how it will be set up in the system, how we will monitor everything it does and what its decision rights are.
It’s an exciting opportunity, but there is a lot of thinking to do around governance, oversight, controls, auditability and other important considerations as we move further into the agentic world. It presents a great opportunity, but also a great responsibility for leaders in this space.
Claire:
They need to tread carefully.
I saw a LinkedIn story last week in which Claude Code confessed to deleting a number of files on a user’s system. That was the first time I had seen an AI agent confess to deleting files. It might have gone unnoticed had the AI not been trained to report when it did something wrong.
Going back to what we said at the start, we have not worked out how to use AI safely and effectively in day-to-day work. We must ensure that these models are appropriately tested and that the right guardrails are in place.
Concerns such as AI bias and hallucinations will grow exponentially in risk when they extend to file deletions or AI agents going rogue.
We need the right checks and balances. As much as there is opportunity, there are also new risks that we need to be mindful of.
Vik:
It’s remarkable. That LinkedIn story made me think, “Wow, the world is changing before our eyes.”
It has been a great conversation. I’d like you both to give us some final thoughts. It’s a big topic, but what is the one takeaway you think our IT leader audience should reflect on?
Katy:
For me, it would be that governance and innovation are not opposing forces; they support each other.
Claire spoke about this earlier and I agree. The organisations that govern AI well will be the ones that gain the most value from it.
We talked about trust, and governance engenders trust. Balancing proportionate governance with the right guardrails and frameworks will enable and encourage innovation within an organisation. Governance and innovation are not opposing; they are complementary.
Claire:
That’s a great one. I’ll give you a slightly different takeaway.
I would advise listeners to be relentlessly curious and continue to ask questions. Don’t assume things have been well thought through in the organisation.
When something lands on your desk, dig into it and ask why. What is the commercial reasoning behind what we’ve been asked to do?
Every day is a school day, as I always say. Keep learning and stay curious.
Vik:
Very good advice indeed.
That brings us to the end of episode two of this podcast, exploring the interface between trust and AI-driven growth. My thanks to Claire and Katy for joining us and sharing their expertise.
Please make sure you catch up with our other episodes as we explore this critical issue. Bye for now.
