Governance ·
Why your organisation needs a clear AI policy: lessons from real-world hallucinations
Real-world examples of AI hallucinations in official documents, and the six safeguards your organisation needs in an AI policy to protect its reputation.
Supreme Court of India, Vijay Ghanshyam Gadiya v. Union of India, judgment of 2 September 2026.
Real-world AI hallucinations in official documents
India's Supreme Court threw out (for now) a ₹425.28 crore penalty (roughly $48 million) in Vijay Ghanshyam Gadiya v. Union of India on 2 September 2026. The penalty order, issued by a customs officer in Surat, relied on case law that did not exist, complete with fake citations and legal reasoning attached to precedents that never happened. According to the Court, this "appears to be a hallucination of AI."
This was not the first time Gadiya challenged the penalty. In January 2026 he took it to the Gujarat High Court, which dismissed his challenge. The Supreme Court later set aside both that decision and the customs order. It is not clear if Gadiya's lawyers had already raised the invented citations at that stage.
The invented citations made it into the final order. It took Gadiya's lawyers arguing that the citations were AI-generated for the Supreme Court to check them. Some of the cases did not exist. Others were real, and the order claimed they said things they do not.
Another rather ironic example played out in South Africa. The government withdrew its Draft National Artificial Intelligence Policy just 16 days after publication because it cited fake research. In a letter to the minister, civil rights group Article One identified six of the document's 67 sources that appeared to be fabricated. The minister's own explanation was that AI-generated citations were included without proper verification. And Europe is not spared either: researchers at Germany's Westfälische Hochschule, cited by Der Spiegel, found 26 incorrect footnotes out of 492 in one report by ENISA, the EU's cybersecurity agency. ENISA acknowledged the errors, calling them human errors, and said AI had been allowed to make minor editorial revisions.
While we do not know if staff checked the sources before publication, we do know the checks in place did not catch the errors. When it comes to AI governance, it is important to have a named person who is accountable for the end result. The check has to cover what a source says, as well as whether it exists. As the Gadiya case shows, real case law can come with invented reasoning.
Why an AI policy matters for NGOs and mission-driven teams
Chances are that your organisation or business is using AI, even if you are not aware of it, to draft documents such as proposals, concept notes and external communications. These all contain claims, figures and references that need to be verified before publication.
Sometimes the stakes are high, as shown in Victoria (Australia) in 2023, where a child protection worker entered personal and sensitive information into ChatGPT, outside the department's control, to create a report which would be used to inform the Children's Court whether protection was needed. The report downplayed the risks to the child. It did not change the outcome of the case but it could have. The department reported the incident in December 2023, and the state's privacy regulator published its findings in September 2024.
In the humanitarian sector, where many beneficiaries are vulnerable, a 2025 survey of 2,539 respondents across 144 countries and territories found that 93% were using or had tried AI tools, while only 22% reported that their organisation has a formal AI policy.
What should an organisation's AI policy include?
Errors like these usually get found by someone: a regulator, a journalist, the other side in a dispute. Often after publication, when the damage is harder to undo. When donors or clients spot them first, trust breaks. Governing the use of AI with a policy staff actually use is a basic requirement to protect your organisation, business, your beneficiaries, clients and staff.
What does that look like in practice? We recommend mapping out:
- What data the AI can access
- What the AI is allowed to do, and what it is not allowed to do
- Which outputs require a person's approval, and who that person is
- Which tools are allowed, and who can use which tool
- What happens when something goes wrong
- How facts, figures and sources get checked, by reading them, not just finding them
Download our free AI policy templates →
A policy only works when people use it
As these examples show, just having an AI policy is not always sufficient. Staff need to know the importance of using approved tools, sticking to the guidelines and knowing what to do when things go wrong. For this reason, we include an AI literacy session with every build for free. AI can close a resource gap quickly, but using it responsibly is what makes it work in the long term.
Does your AI policy make it clear that a specific person is accountable for checking every fact?
This is the same thinking behind how we work, and it sits alongside the wider shift we wrote about in how nonprofits should use AI in 2026.
Sources
- Supreme Court of India, Vijay Ghanshyam Gadiya v. Union of India, judgment of 2 September 2026
- SAnews (South African government), Minister announces withdrawal of draft AI Policy, 26 April 2026
- GroundUp, Article One's account of the fabricated sources in the draft AI policy, 30 April 2026
- heise online, report on ENISA's incorrect footnotes, citing Der Spiegel, 11 January 2026
- Office of the Victorian Information Commissioner, investigation report into the use of ChatGPT in child protection, September 2024
- Humanitarian Leadership Academy and Data Friendly Space, Artificial intelligence in the humanitarian sector, 2025