What Makes Healthcare AI Good? A Practical Evaluation Guide
Episode 10 explains how to evaluate healthcare AI by workflow fit, error costs, representative data, integration, and privacy.
What Makes Healthcare AI Good? A Practical Evaluation Guide
“Good” only has meaning after the job is defined
Which AI platform is best for healthcare? Episode 10 of the AI and Healthcare Podcast starts with that common question and replaces it with a more useful one: good for what? In the conversation, recorded June 2, 2026, Dr. Joseph Yoon and Noah Vandal discuss why a model cannot be evaluated apart from the workflow, information, population, and consequences surrounding it. A system that performs well on a general benchmark may still fail inside a clinic. A less celebrated model may be more useful if it connects safely to the right information, handles the organization's terminology, and fits the way staff actually work. That is the episode's central idea. “Good healthcare AI” is not a permanent label attached to a vendor or model. It is a judgment about whether a particular system is fit for a particular use under particular conditions. NIST describes trustworthy AI through several connected characteristics, including validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy, and fairness with harmful bias managed. No single score proves all of them.
Start with a workflow, not a platform
A useful evaluation begins with a narrow workflow statement. For example: - Route routine scheduling calls while escalating clinical questions to staff. - Draft a visit summary for clinician review. - Identify messages that may require same-day follow-up. - Retrieve approved information from a practice knowledge base. - Transcribe a conversation that contains specialty-specific terminology. Each task requires different data, integrations, metrics, and safeguards. A documentation assistant may need access to the encounter and EHR workflow. A phone agent may need scheduling rules, escalation paths, and reliable identity checks. A clinical decision-support tool may require evidence that would be unnecessary for an administrative FAQ assistant. This is why access to context matters. A capable model without the practice's approved policies, availability, or patient-specific information may produce a fluent but unusable response. Integration is not automatically a virtue, however. Every connection should have a defined purpose, limited permissions, appropriate logging, and a plan for failure. The question is not “Can the AI connect to everything?” It is “What is the minimum information and authority it needs to complete this bounded task safely?”
Evaluate the mistakes, not only the average score
The episode uses a memorable example: a classifier can be 95% accurate and 100% useless. Imagine a dataset in which 95% of images do not contain the condition being detected. A model that always predicts “no condition” will be correct 95% of the time while failing every positive case. The average accuracy sounds impressive because the underlying classes are imbalanced. Healthcare teams therefore need to look beyond one headline number. Depending on the task, relevant measures may include sensitivity, specificity, precision, negative predictive value, word error rate, calibration, completion rate, escalation accuracy, or the frequency of unsupported statements. FDA research on AI-enabled medical-device evaluation makes the same broader point: different intended applications require different performance metrics. False positives and false negatives also have different consequences. There is no safe general rule that one is always preferable. The acceptable balance depends on what happens next. A low-cost, reversible safety check may tolerate more false alarms. A result that could trigger an invasive intervention requires a different threshold and confirmation process. Screening, diagnosis, treatment, and administrative prioritization are separate uses even when they involve the same condition. Before a pilot, define the failure modes that matter, who reviews them, and what the system must do when it is uncertain. Do not let the vendor choose the success metric after seeing the results.
Test the people and setting where the system will operate
Domain specialization can help when the task contains terminology or patterns that are uncommon in general data. The episode discusses speech recognition as a simple example: everyday conversation and specialty medical dictation do not present the same vocabulary. Specialization is not enough by itself. Evaluation data should represent the intended population, equipment, environment, and workflow. Aggregate performance can conceal weaker performance for particular subgroups or conditions. The conversation refers to an optical sensor example and skin pigmentation. The documented healthcare example is pulse oximetry, which estimates blood oxygen saturation rather than carbon dioxide. The FDA has said current evidence suggests accuracy differences between people with lighter and darker skin pigmentation and has proposed more representative testing across skin tones. That example is not only about one device. It shows why a system can perform acceptably in development and still create unequal errors in practice. Teams should ask: - Who was represented in the development and validation data? - Were results reported by relevant subgroups and operating conditions? - Does the evaluation resemble this organization's patients, staff, language, equipment, and data quality? - What happens when an input falls outside the conditions the system was tested on? - How will performance be monitored after deployment? Race should not be treated as a simple genetic variable, and skin pigmentation should not be reduced to a racial label. The underlying measurement, population, and mechanism need to be defined precisely.
Privacy and security depend on the exact configuration
An AI vendor's general security page is not an approval to send it protected health information. HHS guidance says a cloud service provider that creates, receives, maintains, or transmits electronic PHI on behalf of a covered entity or business associate is itself generally a business associate—even if the information is encrypted and the provider does not hold the key. The parties generally need a HIPAA-compliant business associate agreement, and the healthcare organization still needs its own risk analysis and risk-management process. Review the actual product and data path, not only the company name. Important questions include: - Is this specific service covered by the BAA? - Which features, models, endpoints, and subprocessors are eligible? - Is customer content used for model training, and is that the default or an optional setting? - What is retained, where, for how long, and for what purpose? - Can administrators limit access and export audit logs? - What happens to data in connected tools, search services, or third-party integrations? - How are incidents reported, and how can data be returned or deleted at termination? For example, official OpenAI API documentation says API data is not used to train OpenAI models unless the customer explicitly opts in, but retention and HIPAA eligibility still vary by endpoint, feature, and account configuration. “Not used for training” is not the same as “never retained,” “covered by a BAA,” or “appropriate for this workflow.” Every vendor needs the same product-specific review. Our earlier [episode on AI privacy and cybersecurity](/blog/podcast-episode-04-ai-cybersecurity-healthcare) covers this boundary in more depth. The [SpeechSage Trust Center](/trust) also separates product capabilities from the controls used to manage access, privacy, and accountability.
A small organization can run a disciplined pilot
Smaller healthcare organizations do not need to compare every model on the market. They need one clear problem, a controlled test, and a decision rule. A practical pilot can follow seven steps: 1. **Define the task.** State what the system may do and what remains out of scope. 2. **Map the workflow.** Identify the information, integrations, users, decisions, and failure points involved. 3. **Set the risk tier.** Separate administrative assistance from clinical or safety-critical use. 4. **Choose meaningful measures.** Include quality, errors, subgroup performance, staff effort, escalation, and operational outcomes. 5. **Verify data controls.** Complete security, privacy, BAA, retention, access, and subprocessor review before using PHI. 6. **Test with human oversight.** Use representative cases, record failures, and make it easy for staff to override or stop the system. 7. **Monitor after launch.** Recheck performance when models, prompts, integrations, policies, populations, or workflows change. The episode does not offer one winning platform because no platform wins every use case. Its more durable recommendation is to evaluate the whole system: model, data, workflow, people, integrations, contracts, and consequences. That standard may be less exciting than a leaderboard. It is much closer to what “good” needs to mean in healthcare.