Researchers trained ChatGPT and Grok to analyze Senator Roger Wicker’s FoRGED Act, the transformative defense acquisition bill proposed in Congress.
The Greg and Camille Baroni Center for Government Contracting in the Costello College of Business at George Mason University today released a new whitepaper detailing the findings of a pilot program to explore the efficacy of using AI in the legislative process to draft and analyze bills.
Baroni Center Senor Fellow Richard Beutel and Defense Acquisition University Visiting Professor Art Nicewick trained two popular Large Language Models (LLM’s) - Chat GPT and GROK - to analyze one of the most ambitious legislative reform packages in recent years, the FoRGED Act from Senator Roger Wicker. The results were examined by an expert panel that included professional staff members of the Senate Armed Services Committee.
Highlights of the findings include:
-
AI accelerates research while its predictive tools help anticipate implementation challenges. For example, the AI could model how the FoRGED Act’s reforms impact deterrence against China using unclassified threat assessments.
-
While AI excels at technical drafting, it is less successful at understanding the nuanced stakeholder dynamics in politics.
-
AI-generated explanatory materials may lack the persuasive finesse or rhetorical flourish that human drafters bring to bear, particularly in politically charged contexts.
-
AI may lack the political acumen to weigh trade-offs or anticipate stakeholder reactions, areas where human judgment remains irreplaceable.
-
Over-reliance on AI could lead to “path dependency,” where staff favor incremental tweaks over bold reforms because AI excels at refining what already exists.
When using AI models for legislative purposes the panel recommends:
- AI trainers prioritize which questions the AI is likely to have sufficient information to appropriately answer rather than general questions.
- It is critical that researchers caveat their findings with an acknowledgment of what they do not know. The absence of such caveats from this model may be more problematic than the absence of sourcing.