AI isn’t a black box. That’s a design choice.

When people say AI is a “black box”, they usually mean something specific: you put a question in at one end, an answer comes out the other, and what happens in the middle is hidden from view. You are being asked to trust a result without being able to see how it was reached. For solicitors, doctors and expert witnesses, whose work depends on being able to defend their reasoning, that is a serious concern, and an entirely reasonable one.

It is not, however, the whole picture. The black-box framing fits a single AI model in isolation. A product built around a model is a different thing: it is a pipeline of separate steps, and each step can either hide its working or show it. Whether those steps are visible or opaque is one of the most important decisions an AI vendor makes. This article is about why visibility matters more in medico-legal work than almost anywhere else, and what we have done about it at ClinLexis.

When Dr Abdel-Aziz and I started talking about how ClinLexis would actually work, the question I kept returning to was not whether the AI would be capable enough. It was whether it would be inspectable enough. Could a clinician or a solicitor, sitting in front of a screen full of AI-generated chronology entries, ever satisfy themselves that they could trust what they were looking at? Most of what we have built since is an answer to that question.

Where black-box AI isn’t accepted already

The principle that AI should be inspectable, not just impressive, is not new, and it isn’t unique to medico-legal work. Several other domains have already drawn the line.

AI used directly in patient care is the closest parallel. In the UK, the MHRA regulates it as it would any other clinical tool, and explainability is part of what developers have to demonstrate before deployment. UK GDPR Article 22 reinforces this from the patient’s side, giving anyone subject to a solely automated medical decision the right to “meaningful information about the logic involved” and to human intervention. NHS England’s Code of conduct for data-driven health and care technology goes further still, requiring AI systems used in NHS care to be transparent in operation. When a radiology tool flags a region on a mammogram, the clinician sees the highlight, the confidence score and the visual overlay, not just a verdict. The FDA has now cleared more than 950 AI- and machine-learning-enabled clinical tools on those terms.

Financial services drew the same line earlier. The FCA’s Consumer Duty obliges firms to support customer understanding of decisions affecting them, and the Bank of England’s supervisory statement on model risk management (SS1/23, in force from May 2024) requires firms to evidence the interpretability of their AI and machine-learning models. Banks have spent the past decade investing in interpretability: counterfactuals, feature attributions, model risk frameworks. Regulators were never going to accept “the algorithm declined the loan” as a final answer.

Reasoning models themselves have moved on. The newest large language models now expose their intermediate reasoning rather than only the final answer, and national AI safety institutes, including the UK’s AI Security Institute, are publishing evaluations of how faithfully those traces reflect what the model actually does. The direction of travel across the industry is visibility, not opacity.

Medico-legal practice should expect the same standard. The verification duty is explicit, the data is the most sensitive there is, and the consequences of being wrong end up in court.

Why this matters

Two recent findings show why.

The first is the 2024 Stanford HAI study by Magesh and colleagues. They tested the leading legal-research tools, including ones marketed as “hallucination-free”, and found error rates between 17% and 33% on retrieval-augmented systems built on curated case-law databases. Lexis+ AI was at the better end; Westlaw’s AI-Assisted Research was at the worse. These were paid, professional-grade tools sold to qualified solicitors. The vendors were not lying about the underlying architecture. They were over-promising about what the architecture could guarantee.

The second is Ayinde and the parallel matter Al-Haroun, both decided in the High Court in 2025. Solicitors and a barrister had submitted skeleton arguments containing fabricated case citations, generated by AI tools and never verified. The court could have treated this as a novel problem requiring new doctrine. Instead, it applied an old rule with full force: a lawyer who signs a document is responsible for its contents, AI tool or no AI tool. Wasted-costs orders and regulatory referrals followed.

Take those two findings together and the design objective for any serious AI tool in this domain becomes clear. The duty to verify cannot be delegated to a machine. So the system has to make verification fast, obvious and unavoidable. Every claim the system makes about a clinical record has to be one click from the page that produced it. If that click is hard, or imprecise, or absent, the system has done a bad job, no matter how clever the underlying model.

One fact, end to end

Most of what we built at ClinLexis is a long argument with that design objective. The clearest way to show it is to follow a single fact through the system, from a scanned page to the screen.

1. 1OCR with coordinates — Before any language model is involved, we run the page through optical character recognition that returns, for every word, the coordinates of its bounding box on the page. The text is not “extracted by the AI”; it is read off the pixels by a deterministic process that records exactly where each word sat. Those coordinates are stored alongside the text and travel with it through the rest of the pipeline. They are how we will find the words again later.
2. 2A multi-pass pipeline, not a single model — This is the part that surprises people most. The popular image of generative AI is one giant model being asked one giant question. That is not how a careful system works. Inside ClinLexis, a single page is processed by several passes, each with a defined input and output: a classification pass that decides what kind of document this is; an extraction pass that picks out clinical events and their dates; a structuring pass that places them on the timeline; a linking pass that matches each event back to its source coordinates; and where appropriate, a reasoning pass that flags possible significance. Each pass can be inspected, replayed, and disagreed with. None of them is asked to “do everything”. It is unglamorous, multi-stage engineering. It is also why the failure modes of the system are observable rather than mysterious.
3. 3A citation that opens the page, not just the document — When the lithium-level entry appears in the chronology, the citation next to it is not an aspiration. It is a link to the exact bounding box on page 19 of the discharge summary. Clicking it opens the PDF in our viewer with the sentence highlighted. Not the page — the sentence. If the highlight does not appear, we treat that as a bug and log it; we have a service whose only job is to record citation failures and feed them back into the pipeline.
4. 4A “show your working” button on every entry — Next to each chronology entry sits a small lightbulb. Click it, and the system explains, in plain English, how the entry was assembled: which page it was drawn from, what the source text actually says, and how the date was resolved. You do not have to take the chronology on faith.
5. 5Inconsistency flags that point both ways — If the system finds that another document in the bundle records that the level was reviewed, it surfaces the contradiction, with citations to both sources. You see the conflict, you see the evidence on each side, and you decide which one is right.

That is the whole architecture in one example. Five layers of transparency, each independently verifiable. None of it is magic. All of it is engineering choices we could have skipped.

The line we won’t cross

There is a deliberate choice running through every step above, and it is the most important one. The system surfaces evidence; it does not make the call.

That sounds like a marketing line. It is a structural commitment that constrains everything we build. ClinLexis is not allowed, by design, to assert a clinical opinion or a legal conclusion. It will tell you that a review interval was 23 days. It will not tell you that 23 days was negligent. It will tell you that two records say different things about a medication. It will not tell you which one is right. It will tell you what NICE recommends. It will not tell you that the recommendation was breached. The language of the system is consistently neutral, and we have rejected feature ideas, some of them genuinely useful, because they would have crossed that line.

Drawing this line has consequences for how the product feels. Our chronology screens have a slightly clinical quality; they show data and let you reason. Our reports come with placeholders for opinion, not opinion itself. Some prospective customers find this less impressive than tools that produce confident-sounding conclusions on demand. We think the absence of confident-sounding conclusions is a feature. The point of a glass-box system is not just that you can see what it did; it is that you can see what it refused to do.

Three questions to ask any AI vendor

If you are a solicitor, an expert witness or a clinician evaluating an AI tool, you do not need to learn the engineering to ask useful questions. Three are usually enough.

1. 1Source in one click? — When the system makes a claim about a record, can you reach the source — the exact passage on the exact page — in one click? If not, the verification burden the courts have placed on you has not been engineered for. It has been pushed back onto you.
2. 2Visible reasoning? — When the system flags something as significant, can you see why, in plain English, with the guideline or the contradictory record cited? If the reasoning is not visible, the system is not glass.
3. 3Where does the system stop? — If it offers clinical or legal conclusions, ask who is accountable for them. If it surfaces evidence and stops short of the call, you are looking at a tool that respects what you do.

If the answers come back specific and technical, the tool has been built with the duty of verification in mind. Vague answers tell you it hasn’t.