‘Typical’ AI tool evaluation
The New York State Association of Counties (NYSAC) Deputy Director Mark Lavigne is one of many state and county leaders facing mounting pressure from elected officials and vendors to adopt the latest AI tools. However, identifying effective tools and modernization strategies on a limited budget has its challenges.
Counties lack the time, expertise, and resources to confidently evaluate AI tools and need a trusted, peer-driven framework to support informed procurement decisions. For organizations like NYSAC, which represent county elected officials and their staff across all 62 New York counties, this process has been manual and time-consuming, relying on surveys, unclear responses, and informal peer networks.
A fragmented, manual process with no shared framework
Faced with mounting pressure to adopt new technologies, county leaders struggled with limited resources, expertise gaps, and rigid procurement frameworks. The lack of standardized scoring systems, shared resources, and scalable peer-driven frameworks means counties struggle to evaluate tools responsibly. This urgency and approach leave counties vulnerable to rushed decisions, mismatched solutions, and wasted taxpayer dollars. This all exacerbates the challenge of responsibly navigating the rapidly changing AI landscape.
County IT directors and chief information security officers (CISOs) have no consistent AI evaluation metrics to compare tools or justify procurement decisions with data. It was time to employ a different method. CAI and New York county officials created a workgroup to collaborate on the creation of the GovAI Trustmark dashboard—a peer-driven framework for evaluating AI tools, improving project outcomes, and ensuring long-term savings using credible, data-driven insights.
How to evaluate whether an AI tool is credible and usable
The technology experts at CAI know that evaluating AI tools effectively requires a structured approach to ensure credibility, usability, and alignment with organizational goals. Following NYSAC’s Artificial Intelligence Summit in July 2025, it was clear that counties need vetted AI tools they could trust.
NYSAC sought out CAI’s expertise and requested the facilitation of a new AI workgroup. An AI advisory group was formed, made up of NYSAC and CAI, county IT directors, cybersecurity professionals, and county leadership officials. The working group then kicked off a series of meetings designed to create a matrix for the GovAI dashboard. In these development sessions, the team identified the criteria most important to counties, determined the scoring metrics for evaluating AI tools, and began assessing tools relevant to county operations.
NYSAC partnered with CAI under a one-year pilot to provide technical expertise, development, and support for the dashboard while keeping evaluations firmly grounded in county experiences. That connection led directly to discussions about building the statewide AI evaluation resource now known as the GovAI Trustmark platform.
This intuitive, web-based platform enables New York county officials to evaluate AI tools using a standardized, weighted scoring framework, which makes it possible for counties to go to a single, trusted resource for peer-reviewed AI evaluations, policy guidance, and educational materials. The site did a soft launch in March and then completed a full launch as of May 2026.
With the help of CAI, counties are now able to mobilize internal AI task forces, streamline vendor response processes, and gain confidence in AI procurement decisions. County attorneys and CISOs are using the dashboard to align programs with the NYS AI Acceptable Use Policy. What once required days of manual outreach and summarization now takes minutes.
“The NYSAC GovAI Trustmark dashboard is an essential resource for medium and smaller counties that lack the capacity to thoroughly evaluate the many available AI tools. This shared resource is invaluable for supporting informed and responsible adoption decisions.”
How to make AI tool utilization and procurement easier
The GovAI Trustmark platform’s standardized scoring system evaluates tools across 4 key AI evaluation metrics benchmarks, ensuring that the AI solutions are evaluated consistently. They are:
- Data (30%)
- Risk (30%)
- Ease of use (20%)
- Product evaluation (20%)
There are other gateway assessments for cost, return on effort, and security that must be cleared before a full evaluation proceeds. All county evaluations are shared transparently across the platform, meaning that counties also benefit from shared insights and completed evaluations from peers. This reduces time spent on research and boosts confidence in procurement decisions, making the correct AI digital government technology platforms easier for counties to access.
As of June 2026, the platform hosts 27 AI solutions across key categories including productivity, transcription, and human services, with 20 completed evaluations and 85 users across 32 counties. This also includes an educational resource library covering AI policies, key concepts, and guidance for counties at every stage of AI readiness.
Steps for adopting external AI evaluation frameworks:
The AI Advisory group focused on gathering statewide perspectives that have turned into recommendations that include the following:
- Building awareness: Educate officials on the benefits through workshops and webinars.
- Pilot programs: Suggest starting with small-scale demonstrations and testing to show purpose and value.
- Policy integration: Advocate for procurement policies that prioritize external evaluations.
- Collaboration: Encourage partnerships with trusted evaluation platforms and AI advisory councils.
Proper use of these platforms eliminates guesswork and manual effort by leveraging weighted scoring systems, expert insights, and real-world peer evaluations. They focus on critical dimensions such as data security, risk management, ease of use, and product performance, ensuring that only the most reliable and impactful tools are selected.
The impact and future of external AI evaluation platforms
The impact of GovAI Trustmark is faster decisions, greater confidence, and transparency. This innovative tool is changing how New York counties approach AI. What once required days of manual research and coordination now takes minutes. Counties are using the dashboard to build internal AI task forces, align with the NYS AI Acceptable Use Policy, and identify necessary infrastructure improvements in order to respond to vendor requests with clarity and confidence.
“CAI was skillful in being able to gather the collective insights from a small group of county leaders to put together a model for moving forward on AI. It turned out to be a more efficient process than any of our team expected.”
AI evaluation platform frameworks like GovAI Trustmark will play a critical role in helping governments scale more responsibly, maximize impact, go farther with a limited budget, and equip counties with the knowledge needed to future-proof their investments. By identifying high-performing solutions early, counties can avoid overspending on tools that fail to deliver returns and instead focus resources on technologies that drive meaningful innovation. Smarter tool selection minimizes costly missteps while optimizing project outcomes, ensuring that taxpayer dollars are invested wisely.
To learn more about how CAI can help your government organization develop stronger evaluation and analysis procedures for your next external AI evaluation platform, fill out the form below.
