Tag: Risk

OpenAI Agent Hacks Australian Government Portal

The Australian Prime Minister, Anthony Albanese, has disclosed that an AI agent, developed by OpenAI, hacked a government portal. This seems to be one of the world’s first examples of an AI-led attack on a government website. It comes as international leaders and AI firms call for faster regulation, with growing concern that the technology is developing more quickly than current safeguards can manage. 

Speaking today, whilst attending the UN General Assembly in New York, Mr Albanese said that the agent infiltrated part of the Australian government’s healthcare scheme Medicare. The hack took place in June, though OpenAI became aware of it in August and informed the Australian government in September. The company said, “our models took actions we did not intend” and that it found no record of patient data being accessed. The Australian government has launched an urgent review. 

Notably, an AI-enabled hack of this kind was anticipated by Caroline Wong, an AI expert, during an appearance on the Guardians of Data podcast in June. Listen to the clip here.  

This case highlights the importance of ensuring that all AI has effective governance controls in place and is properly tested before being deployed. This is particularly important in the case of AI agents. For the past few years, legal, compliance and data protection professionals have largely focused on the risks associated with deploying large language models (LLMs) such as ChatGPT and Claude.  

The key difference between LLMs and agentic AI is autonomy. Traditional AI tools usually operate within a defined task and rely on people to decide what happens next. By contrast, agentic AI can adjust its actions as information changes, tools become available or the task develops. This increases the risks substantially. 

An agentic system may be linked to internal systems, third-party services, customer information, APIs and external tools. Each connection creates potential cyber security exposure. If an agent has excessive permissions or is badly configured, an attacker may be able to influence its actions, redirect it towards an unintended outcome or gain access to sensitive commercial or personal information. 

For a detailed discussion on the risks of AI Agents see our blog post. You can also hear more on building trustworthy and responsible AI systems with AI expert Tahir Latif in this podcast. 

We are repeating our fully booked AI Agents webinar in November. It will equip you with the knowledge needed to understand, assess, and govern AI agents confidently and responsibly.

AI Agents: A New Frontier of Risk

For the past few years, legal, compliance and data protection professionals have largely focused on the risks associated with deploying large language models (LLMs) such as ChatGPT and Claude. The focus is now expanding to agentic AI, defined by IBM as: 

“…an artificial intelligence system that can accomplish a specific goal with limited supervision. It consists of AI agents – machine learning models that mimic human
decision-making to solve problems in real time.” 

The key difference between LLMs and agentic AI is autonomy. Traditional AI tools usually operate within a defined task and rely on people to decide what happens next. By contrast, agentic AI can adjust its actions as information changes, tools become available or the task develops. In this sense, “agentic” describes technology that can act purposefully, rather than merely produce a response. 

Let’s take a customer service scenario. An AI agent could receive a complaint, review the customer’s previous interactions in Salesforce, check delivery information in a logistics system, draft a response, create a follow-up case and route the issue to a human member of staff where judgement is needed. That is materially different from an AI chatbot that helps with basic tasks like write an email or answer queries. 

Many organisations are now deploying AI agents in areas such as sales and customer service, using tools offered by the likes of OpenAI, Google and Salesforce. In the health sector, tools such as Oracle Health Clinical AI Agent help clinicians by supporting documentation and workflow automation within electronic health record systems. Anthropic’s “2026 State of AI Agents” report, says that 57% of the 500 U.S. companies surveyed were deploying agents for multi-stage workflows and 56% planned to deploy agents for research and reporting in 2026.

Cybersecurity Risks 

But the deployment of AI agents is not without risk. An AI agent may have the ability to act, rather than simply advise. Depending on the use case, it could connect to business applications, recommend or make decisions, trigger workflows, send messages, alter records or begin a financial process. These capabilities mean that there is much more that can go wrong compared with a traditional LLM, where the main risk is
over-reliance on an output that may not be accurate. 

One of the key risks associated with deploying AI agents is cyber security. An agentic system may be linked to internal systems, third-party services, customer information, APIs and external tools. Each connection creates potential exposure. If an agent has excessive permissions or is badly configured, an attacker may be able to influence its actions, redirect it towards an unintended outcome or gain access to sensitive commercial or personal information. 

Testing an AI agent before deployment is crucial. Just yesterday, OpenAI revealed that its AI agent went rogue and hacked a start-up after it lost control of it during a security test. The joint guidance Careful adoption of agentic AI services, co-authored by the NCSC and international partners, recommends that organisations start small, use agents initially for low-risk tasks and apply established cyber security controls from the outset. 

In practice, this means applying secure design, least privilege, access management, monitoring, incident response planning and supplier assurance. For a detailed discussion on the impact of AI on cybersecurity, listen to the Guardians of Data podcast with Caroline Wong. 

Data Protection Risks 

Data protection risk can also increase where an AI agent needs broad access to information to carry out its objective. It may pull together customer records, identify patterns, summarise communications, classify individuals, suggest next steps or trigger further action. Much of this will involve personal data and so the UK GDPR will come into play. 

The ICO has taken a keen interest in this area. In its Tech Futures report on agentic AI, it states: 

“One of our key findings from this initial work is that the specific design and architecture of agentic systems impact how data protection law applies and how people exercise their data protection rights. Choices such as the data and tools that a system can access and which governance and control measures to put in place really matter.” 

The ICO’s AI and data protection toolkit helps organisations assess how AI systems may affect individuals’ rights and freedoms. Key data protection risk questions for organisations deploying an AI agent include: 

  • What personal data does the agent need to perform the task? 
  • Can the same outcome be achieved using less data? 
  • Is the agent making or informing decisions about individuals? 
  • Are special category data, children’s data or vulnerable individuals involved? 
  • Can individuals understand when AI is being used and how to challenge decisions? 
  • Are prompts, outputs, logs and feedback data retained, and if so for how long? 
  • Have the controller/processor roles been properly analysed? 

Governance 

Any organisation adopting AI will need an AI governance policy. With agentic AI, however, governance must be more than a static document. It should be a working process that follows each proposed use case from initial idea through to deployment, monitoring and later review. 

Higher-risk use cases should be assessed before launch by legal, data protection, security, product and operational stakeholders. That assessment should cover the agent’s purpose, degree of autonomy, access to data and systems, impact on users, contractual arrangements, supplier terms, monitoring approach and exit plan. 

Good governance also depends on records. Organisations should keep evidence of risk assessments, testing, known limitations, approvals, training materials, monitoring outcomes, complaints, incidents, remedial steps and changes to prompts or workflows. That evidence may become important if a customer, regulator or court later asks what happened, when things go wrong. The organisation will need to show not only that it had a governance framework, but that the framework was followed in practice. 

AI agents bring exciting possibilities, but also many risks. They may also change the structure of organisations for good. Caroline Wong, an AI expert speaking on the  Guardians of Data podcast, predicts that the future workforce could be a mix of human and agentic AI “workers.” You can listen to a short clip here. 

Learn more about AI agents and their safe deployment on our forthcoming webinar. You can also hear more on building trustworthy and responsible AI systems with AI expert Tahir Latif in this podcast.

Why Risk Management is Essential for IG Professionals 

GDPR compliance is very much about risk management. Throughout the UK and EU GDPR, Data Controllers are required to implement protective measures corresponding to the level of risk of their personal data processing activities. Consequently, risk management is a foundational skill which all data protection and information governance professionals need to develop.  

Risk in the UK GDPR 

Key provisions of the UK GDPR which mandate a risk-based approach include: 

Article 24 Responsibility of the Controller 

“Taking into account the nature, scope, context and purposes of processing as well as the risks of varying likelihood and severity for the rights and freedoms of natural persons, the controller shall implement appropriate technical and organisational measures to ensure and to be able to demonstrate that processing is performed in accordance with this Regulation. Those measures shall be reviewed and updated where necessary.” 

Article 25 Data Protection by Design and by Default 

“Taking into account the state of the art, the cost of implementation and the nature, scope, context and purposes of processing as well as the risks of varying likelihood and severity for rights and freedoms of natural persons posed by the processing, the controller shall, both at the time of the determination of the means for processing and at the time of the processing itself, implement appropriate technical and organisational measures, such as pseudonymisation, which are designed to implement data-protection principles, such as data minimisation, in an effective manner and to integrate the necessary safeguards into the processing in order to meet the requirements of this Regulation and protect the rights of data subjects.” 

Article 32 Security of Processing 

“Taking into account the state of the art, the costs of implementation and the nature, scope, context and purposes of processing as well as the risk of varying likelihood and severity for the rights and freedoms of natural persons, the controller and the processor shall implement appropriate technical and organisational measures to ensure a level of security appropriate to the risk,…” 

Article 33 Notification of a Personal Data Breach to the Commissioner 

“In the case of a personal data breach, the controller shall without undue delay and, where feasible, not later than 72 hours after having become aware of it, notify the personal data breach to the Commissioner , unless the personal data breach is unlikely to result in a risk to the rights and freedoms of natural persons. Where the notification under this paragraph is not made within 72 hours, it shall be accompanied by reasons for the delay.” 

Article 33 Notification of a Personal Data Breach to the Data Subject 

“When the personal data breach is likely to result in a high risk to the rights and freedoms of natural persons, the controller shall communicate the personal data breach to the data subject without undue delay.” 

Article 35 Data Protection Impact Assessments (DPIAs) 

“Where a type of processing in particular using new technologies, and taking into account the nature, scope, context and purposes of the processing, is likely to result in a high risk to the rights and freedoms of natural persons, the controller shall, prior to the processing, carry out an assessment of the impact of the envisaged processing operations on the protection of personal data.” 

Even where the word ‘risk’ is not explicitly used, the concept underpins a number of data protection principles in the UK (and EU) GDPR. For example: 

Accountability Principle  
Data Controllers must be able to demonstrate compliance. This involves documenting risk assessments, decisions, and mitigations; all of which are key components of risk management. 

Lawfulness, Fairness, and Transparency  
Fair and transparent processing demands that Data Controllers consider the potential impacts on data subjects; essentially, assessing and managing risks to data subjects’ rights. 

Data Minimisation and Purpose Limitation 
Ensuring that only necessary data is collected and processed inherently involves evaluating what is proportionate and appropriate, which are concepts rooted in risk assessment. 

Practical Skills DPOs and IG Officers Need 

Given the prominence of risk in the GDPR, DPOs and IG professionals should cultivate the following competencies: 

  • Risk Identification: Being able to recognise threats to data confidentiality, integrity, and availability; whether technical (e.g. cyberattacks) or organisational (e.g. poor access controls). 
  • Risk Analysis: Assessing the likelihood and potential impact of risks and understanding their relevance to the rights and freedoms of individuals. 
  • Risk Evaluation and Prioritisation: Comparing estimated risks against risk tolerance and legal thresholds (e.g. what constitutes ‘high risk’ under Article 35). 
  • Mitigation Planning: Developing and implementing controls to reduce risk to an acceptable level; whether through encryption, training, anonymisation, or policy development. 
  • Ongoing Monitoring: Risk is not static. DPOs must continuously monitor changes in technology, regulation, and business practices that may affect data risk profiles. 

For data protection and IG professionals, risk management is not a ‘nice-to-have’; it is a foundational skill.  

Interested in developing your risk management skills further? Consider enrolling on our new Risk Management in IG workshop.