New tool designed to detect biases hidden in medical AI data sets, identifies flaws in training

Researchers from Johns Hopkins University and the US Food and Drug Administration (FDA) have developed G-AUDIT, a new tool designed to detect hidden biases in medical AI datasets. Medical AI systems often underperform and exhibit implicit biases due to training data that isn't representative of the broader population. Mathias Unberath, an expert in AI-assisted medicine at Johns Hopkins, noted, "The models that drive precision medicine learn to infer clinical outcomes from the data they're trained on. In many cases, that works great, but it can also lead to interesting failures that aren't immediately apparent."

• G-AUDIT identifies potential problems in datasets before they are used to train medical AI models, addressing issues like "shortcut learning" where AI associates irrelevant patterns with outcomes.
• Examples of shortcut learning include AI associating clinician skin markings with malignant lesions or camera quality and ruler presence with cancer risk.
• When tested on an image dataset, G-AUDIT found image height, width, and year of collection were high-risk factors for bias.
• In a health record text dataset, G-AUDIT identified clinical specialty as the highest risk for bias, surpassing patient sex or race for several tasks.

The research team plans to expand G-AUDIT for potential use beyond healthcare.