§ A.I. · 2 MIN READ
Ask If a Machine Helped Decide
Hospitals are using AI to schedule you, triage you and predict what you need. Some of those systems have already been caught treating Black patients worse. Four questions to ask out loud.
By Culture
THE SHORT VERSION
Artificial intelligence is now used in scheduling, triage and risk prediction across American healthcare, and multiple studies have found it can worsen racial disparities. One scheduling algorithm produced 33 percent longer wait times for Black patients, and some tools still rely on outdated race-based equations for kidney and lung function that produced worse outcomes for Black patients.
Most people assume the AI conversation in medicine is about robot surgeons. It is not. It is about the quiet software deciding how long you wait, how sick the system thinks you are, and which patients get flagged for extra help.
A machine learning algorithm used for appointment scheduling produced 33 percent longer wait times for Black patients than for other patients. A 2024 systematic review found AI use associated with worsened racial disparities, including longer waits, weaker prediction of mental health outcomes, and underdiagnosis of conditions in Black and Hispanic patients.
Then there are the equations. For decades, common formulas for kidney and lung function included a race adjustment, which made Black patients' numbers look healthier than they were and delayed care, including referrals for transplant. Medicine has been removing those adjustments. Some AI chatbots and tools are still repeating them.
And large language models used in clinical settings have been shown to give different answers depending on the patient's race, including in cases where race was not clinically relevant at all.
“The machine did not invent the bias. It inherited it, then applied it faster and to more people.”
The famous example is worth understanding because it explains the whole category. An algorithm used to identify patients needing extra care was trained on healthcare spending as a stand-in for how sick somebody was. Less money had historically been spent on Black patients at the same level of illness, so the model concluded they were healthier and referred fewer of them. Every line of that code was neutral. The result was not.
Take somebody with you. A second person in the room is the most reliable way a dismissal gets caught, which is as true here as it is in the fourth trimester, where preventable deaths are most often a question nobody asked in time.
Write down what you were told and when. If you need to escalate, contemporaneous notes are the difference between a complaint and a file.
And know that this is the same machinery, in a different building, as the software reading your job application. If you need care and cost is the wall this week, the local health directory and a free drop-in clinic are the places to start.
- Is AI being used in my medical care?
- Very likely somewhere in it. Artificial intelligence is now common in appointment scheduling, triage, risk scoring and identifying patients for extra care. You are entitled to ask whether software contributed to a decision about you and what it recommended.
- What is race-based medicine and why does it matter?
- For decades common formulas for kidney and lung function included a race adjustment that made Black patients' results look healthier than they were, which delayed care including transplant referrals. Medicine has been removing those adjustments, but some AI tools still repeat them.
- How can an algorithm be biased if it never sees race?
- By using a proxy. One widely used tool predicted health needs from past healthcare spending. Because less had historically been spent on Black patients at the same level of illness, the model read them as healthier and referred fewer for extra care. Every rule was neutral and the outcome was not.
- What should I ask my doctor about AI?
- Ask whether software was used in the decision and what it recommended, whether any calculation uses a race adjustment and what the number is without it, whether a clinician will review it independently, and what would have to change for the answer to be different.
