Google says AMIE medical AI reaches expert level performance in simulated video consultations

Google Research and Google DeepMind have advanced their medical AI system AMIE to conduct real time clinical consultations over video, allowing the system to interpret visual and auditory information from patients while reasoning during the interaction.

AMIE is built on Gemini and Project Astra and uses a multi agent architecture. Google said the system can identify visual and auditory cues, guide patients through virtual physical examinations and perform diagnostic reasoning in real time.

The capabilities were tested in what Google described as a first of its kind randomized study involving simulated video consultations with patient actors and a group of primary care physicians.

Clinical evaluators rated AMIE favorably across several areas, including the thoroughness of its medical history taking, diagnostic accuracy, appropriateness of its proposed management and communication quality, according to Google.

Patient actors participating in the study also preferred interacting with AMIE through video compared with text based consultations.

The addition of video gives the system access to information that would not normally be available through text alone, including visible discomfort, movement, coughing and other physical or auditory cues that physicians can observe during an appointment.

Google said AMIE remains a research system and requires further study before it can be responsibly deployed in real world clinical settings.

The latest work expands AMIE beyond text based medical conversations as Google continues researching how multimodal AI systems could be used in clinical environments.