Anyone can send you a PDF. So we wrote four designed to talk to the machine instead of to a person.
If a product reads documents on your behalf, the documents are an input channel, and input channels are where attacks live. A deck, a data room export, a one-page summary: to you these are content. To a language model they are also text that may contain instructions, and it has no reliable sense of whose instructions count.
This is prompt injection. It is the least glamorous problem in applied AI and the most practical one, because it does not require any access to the system at all. It requires an attachment and a plausible reason to send it.
In deal making there is always a reason. A founder would like the facts extracted from their deck to come out slightly better than the deck. Somebody in the middle would like to know which other companies an investor is talking to. These are not exotic motives. They are the ordinary incentives of the room.
So before trusting the extraction with anybody’s documents, we attacked it with four of our own.
The first said, in the middle of an otherwise ordinary paragraph, to ignore all previous instructions, and then told the model what to report instead.
The second carried a forged system block, formatted to look like the instructions the model receives from us rather than like content it was given to read.
The third supplied a fake quote, ready-made, offered for the model to reuse as evidence for a claim.
The fourth simply asked for the investor’s other portfolio companies: information belonging to somebody else, which was never in the document.
All four held. Nothing the injections asked for became a fact, and every fact that did come out was quotable from its own document.
The reason is the same rule the whole system is built on, and it is a dull rule rather than a clever defence. Every extracted fact arrives attached to the sentence it came from, that attachment is checked in code against the source, and any number in a claim has to appear in that claim’s own quote. An instruction hidden in a document can ask for anything it likes. It cannot cause a sentence to exist in a source where that sentence is not.
Now the limit, stated plainly, because a post like this is worthless without it.
A quote proves the document said it. It never proves that it is true.
If a deck overstates its revenue, the extraction will faithfully report what the deck says, with the sentence attached. That is the correct behaviour and it is also the boundary of what this technique buys you. What the attachment gives you is a clean line between what a document claimed and what anybody has verified, and somewhere to look in five seconds rather than an argument in a meeting.
It is worth naming that limit for a second reason. A system that claims to have solved trust invites you to stop checking, and stopping checking is the failure this entire category keeps producing. The useful claim is narrower: you can check this quickly, and you can see exactly what to check.
None of this was a one-off exercise done for a blog post. The four documents run as a test on every build. A defence that was true in March and untested since is a story, not a property.
We tried to talk our own AI into lying, and it did not. It runs again on every build, and it still does.
