In 2023, a pair of New York lawyers used ChatGPT to draft a court filing. ChatGPT convincingly cited six completely fabricated cases, and the lawyers submitted the filing without ensuring the accuracy of their brief, unaware of the tendency for generative artificial intelligence (GenAI) to “hallucinate” or falsify information in response to prompts without alerting the user. The lawyers were fined $5,000 for their embarrassing mistake. This well known incident sounded the alarm for the use of GenAI technologies for legal work product. While this has provided a lesson-learned for using caution with GenAI tools, it is certainly possible penalties will be higher for future infractions as the legal system becomes more familiar with the tools and their risks.

One of the reasons GenAI is so dangerous is that the law surrounding it is only just beginning to form. The use of GenAI to generate legal work may amount to the unauthorized practice of law in some jurisdictions and will potentially lead to litigation and breaches of confidentiality.