DTA Tests AI for Faster Digital Marketplace Approvals
▼ Summary
– The federal government is piloting AI to assist with reviewing applications for the Digital Marketplace 2 panel, using a human-AI pairing model rather than full automation.
– The Digital Transformation Agency developed a proof-of-concept where AI rates supplier case studies on a 1-5 scale, similar to human reviewers.
– In testing, the AI agreed with human assessors 84% of the time, slightly higher than the 81% agreement rate between two human reviewers.
– The average rating difference between AI and human was 0.76, showing more consistency than between two humans, who had a 0.92 average disagreement.
– The DTA plans to expand the pilot using a larger dataset of 6448 applications, with further governance checks before potential live implementation.
The federal government is exploring the use of artificial intelligence to streamline the approval process for its Digital Marketplace 2 panel. A new pilot program developed by the Digital Transformation Agency aims to pair AI with human reviewers to assess supplier applications more efficiently, with plans to implement the system later this year.
Launched in October 2024, DM2 serves as a government-wide procurement platform for IT labor hire, professional services, and consulting. With approximately 20,000 applications submitted, the volume of work has prompted the DTA to investigate how technology can assist in the evaluation process. Former DTA principal technology advisor Ben Bildstein emphasized the significance of this initiative, noting that while AI cannot fully replace human judgment in procurement contexts, it can play a supportive role in preliminary assessments.
Initially, the idea of using AI to completely evaluate applications was dismissed after reviewing government procurement policies and AI ethics guidelines. Instead, the agency decided to focus on a hybrid model where AI assists in rating supplier case studies, a task traditionally handled by two human reviewers. These case studies are scored on a scale of one to five, and if the two reviewers agree within a one-point margin, the application moves forward. If they disagree, a third person steps in to resolve the discrepancy.
The proof-of-concept phase tested 268 previous applications, comparing AI-generated ratings with those from human assessors. Three key metrics were used to evaluate performance: agreement rate, average rating difference, and correlation. Interestingly, the AI agreed with human reviewers 84% of the time, slightly higher than the 81% agreement rate between two humans. In cases of disagreement, the process reverts to human-only evaluation.
The average margin of error between human and AI ratings was 0.76, compared to 0.92 between two humans, indicating that AI offers greater consistency when paired with a human reviewer. Correlation metrics also showed that AI rankings aligned closely with human assessments overall.
The next phase will involve a larger dataset of 6,448 applications to strengthen the statistical validity of these preliminary findings. While there are still governance and assurance steps to complete, the model shows promise for future procurement rounds. Bildstein advised other government agencies to prioritize AI assurance early in development and to clearly define success metrics before implementation. He stressed that while AI technology itself is advancing rapidly, ensuring its responsible and effective integration requires careful planning and oversight.
(Source: ITNews)
