The AI tool exposing your firm right now probably isn’t the one in the contract

Most of the AI conversations I have had with solicitors and clinicians over the past year have been about vendors. Which platform to choose. What questions to ask. How to evaluate a tool. These are sensible conversations, and I am pleased the profession is having them.

What strikes me is that they tend to skip past the AI tool people are actually using. It is rarely the one in the contract. It is the one already open in another browser tab.

The behaviour I want to talk about is using a free, consumer chatbot (ChatGPT, Claude, Gemini, Copilot) to do a piece of professional work that touches client information or patient records. Summarising a discharge letter. Drafting a section of a witness statement. Making sense of a complex GP record before a clinic appointment. The people doing this are not negligent. They are responding to a tooling gap. But the architecture beneath those consumer tools was not designed for the data being pasted in, and that gap is now the most underestimated AI risk in medico-legal work.

What’s actually happening

Three things tell us this is not a hypothetical risk.

The first is the regulatory response. Within the past two years, the Solicitors Regulation Authority, the Bar Standards Board and the General Medical Council have each issued guidance specifically on the use of generative AI by their members. The SRA’s 2023 Risk Outlook on Technology and Innovation called out generative AI as a live area of concern and reminded firms of their existing confidentiality and supervision duties. Regulators do not write rules against theoretical problems; they write them against behaviours they are already seeing.

The second is the surveys. Industry reports from the Law Society and from the major legal publishers in 2024 consistently found that a substantial proportion of UK fee-earners were using generative AI tools at least weekly, with the figures particularly high among trainees and junior associates. Surveys of NHS clinicians over the same period have found similar patterns of personal experimentation, particularly in primary care. These figures cover all generative AI use, not only consumer chatbots, but the consumer share is unlikely to be small given how much easier consumer products are to reach than vetted enterprise tools. The directly observable evidence is strongest for solicitors. For expert witnesses specifically, we are inferring from adjacent professions and incentives rather than from dedicated surveys.

The third is the courts. Ayinde and Al-Haroun, decided last year, exist because legal professionals used consumer AI tools to draft material that was then placed before a UK court without verification. In the US, Mata v. Avianca (2023) and several follow-on matters show the same pattern. These are the cases that became public; the cases that did not are, by definition, harder to count.

Taken together, this settles the question of whether shadow AI use is happening in the profession. It is. The useful question is what to do about it.

I am not making a moral point. The tools are useful, free, instant, and require no IT approval. When deadlines are tight and the alternative is reading 800 pages by hand, of course people reach for them. The problem is what happens to the data the moment it is pasted.

Where the data actually goes

When a paragraph from a clinical record is pasted into the consumer version of ChatGPT, several things happen, none of them visible to the person doing the pasting.

The text leaves the device and is sent to OpenAI’s servers, which sit in the United States. Under OpenAI’s consumer terms (free and Plus tiers, by default), conversations are retained and may be used to improve future models, unless the user has explicitly opted out. The conversation is associated with the user’s account and stored on infrastructure that is not subject to UK data-protection law in the same way a UK-resident system would be.

The same is broadly true of consumer Claude (Anthropic), Gemini (Google), Copilot in its consumer tier, and the various Mistral and Llama-based assistants. Each has different defaults, different retention policies, different opt-out mechanisms. None of them was designed with special-category personal data in mind.

Enterprise tiers are different. OpenAI Enterprise and the API both contractually exclude customer data from training. Microsoft Azure OpenAI offers UK regional deployments. Anthropic’s API is no-training by default. These are real, careful products. They are also not the products people open on their phones at lunchtime.

The regulators have been clear about the risk. In March 2023, the Italian data-protection authority temporarily banned ChatGPT for breaches of GDPR around lawful basis, transparency and the handling of personal data. OpenAI made changes and the ban was lifted, but in December 2024 the Garante imposed a €15 million fine on the same grounds. The ICO’s 2024 guidance on generative AI and data protection cautions specifically against the use of consumer LLMs for special-category data. The most public corporate failure remains Samsung in April 2023, where engineers pasted proprietary source code and meeting notes into ChatGPT and the data left the company’s control. Samsung banned consumer generative AI for internal work within weeks. JPMorgan Chase, Verizon and several others have done the same on the same grounds.

What this means for medico-legal work

For most professions, this is a confidentiality and intellectual-property issue. For medico-legal work, it is something more.

Medical records are special-category personal data under Article 9 of the UK GDPR. Special-category processing requires both a lawful basis under Article 6 and a separate Article 9 condition; for litigation, usually 9(2)(f), the establishment, exercise or defence of legal claims. None of those conditions relax the underlying obligations. Encryption, access control, retention limits and a documented Data Protection Impact Assessment are still required.

When a solicitor pastes a discharge summary into a consumer chatbot, those obligations do not pause. The processing is taking place. The data is leaving the firm’s control. There is no DPIA. There is no audit trail. The firm cannot tell a regulator what was sent, when, by whom, or what came back. If a patient subject access request later asks where their data has been processed, the firm cannot honestly answer.

The professional duties go further still. SRA Principle 7 requires solicitors to act in each client’s best interests, which the SRA’s 2023 risk outlook on technology has interpreted to include using only systems that protect client information. The GMC’s Good Medical Practice (2024) requires doctors to maintain confidentiality and to use only secure systems for patient information. The Bar Standards Board has issued similar guidance for barristers.

And then there is Ayinde, which I wrote about in the previous article. The High Court has already made plain that the duty to verify AI output cannot be delegated. The duty of confidentiality is older, deeper, and equally non-delegable. Both run in the same direction: a tool you cannot inspect, audit or control is a tool you cannot lawfully use for client work.

Why people do it anyway

It would be easy to write the rest of this article as a finger-wag. I do not think that is useful, because the people doing this are not negligent. They are responding to a tooling gap.

The unaided alternative — reading the bundle by hand at 11pm after a full caseload — is genuinely worse for the client than getting some help. The expert who asks ChatGPT to plain-English a discharge summary at 7am before clinic is not trying to leak data. They are trying to do their job within the time available. The free tools are right there, on every device, with no friction.

Procurement has not caught up. Many firms have an “AI policy” that consists of a paragraph in the staff handbook saying not to use consumer AI for client work. The policy is unenforceable, the supervision is patchy, and the workflow it implicitly demands (back to manual reading) is not realistic. The result is shadow AI use on a scale most managing partners would not want to admit.

What a responsible alternative looks like

Five things distinguish an AI tool that is safe to use for client work from one that isn’t. None is a marketing claim. All are answerable in writing.

1. 1Where the data lives — Data should be processed and stored within the UK, in infrastructure subject to UK law, with no transfers to other jurisdictions without explicit safeguards. “Global” is a red flag; “EU-equivalent” is not the same as UK.
2. 2Whether the data trains the model — The vendor should be able to point to a contractual clause confirming that customer data is excluded from any training, fine-tuning or evaluation. Default consumer tiers usually fail this test. Enterprise tiers and APIs typically pass it.
3. 3An audit trail — The system should keep a record of what was uploaded, what was asked, what was returned, by whom and when. Not for surveillance, but for the moment a regulator, a client or a court asks what happened on a given date.
4. 4Access control — Multi-factor authentication, role-based access, and the ability to revoke access in one click when a colleague leaves. Unglamorous controls; also the ones that fail first when something goes wrong.
5. 5A DPIA the firm can use — The vendor should make its own data-protection impact assessment available, so a firm’s DPO can rely on it without re-running every question from scratch.

Any vendor that bristles at being asked these five questions is telling you something useful about how seriously they take the duty.

A practical line

The line I would draw, if asked, is this.

For genuinely non-sensitive work — a generic question about NICE guidance phrasing, a query about a public legal principle, a draft of an internal email — the consumer tools are fine, with the same caution you would apply to any cloud service.

For anything that contains client information, patient data, or material covered by professional confidence, do not paste. Use a tool that has been bought, contracted for and risk-assessed by your firm. If your firm does not have one, that is a problem to escalate, not work around.

This is not a counsel of perfection. It is the same line the profession has always drawn between work email and personal email, between firm laptops and home laptops, between the case-management system and a Word document on a USB stick. The technology is new; the principle is not.

The interesting AI question for the profession is no longer whether to use AI. It is whose AI to use, and on what terms. The vendors who do this responsibly will be inspectable, accountable and visible — the principles I argued for in the earlier piece on responsible AI architecture. The ones who do not will continue to be useful, free, and silently dangerous.

If you are a partner or a clinical director reading this, the question to ask is not whether your team uses AI. They do. The question is what the firm or the practice is offering them as the responsible alternative.