Why multimodal surgical data, first-person perception and contextual intelligence will be fundamental to the future of surgical AI.

Artificial intelligence is rapidly changing how we interact with information, interpret complex environments and automate tasks.

Yet surgery presents a fundamentally different challenge.

Unlike many digital environments, the operating room is dynamic, spatially complex and safety-critical. It involves continuous interaction between human operators, anatomical structures, surgical instruments, imaging systems and physiological signals.

Understanding surgery requires more than recognising objects in an image.

It requires understanding relationships, sequences, actions, context and intent.

This distinction could define the next generation of surgical artificial intelligence.

The problem with simply recording surgery

Modern operating rooms generate enormous amounts of information.

Surgical video, medical imaging, patient monitoring, robotic telemetry, instrument usage and clinical documentation all provide different perspectives on what happens during a procedure.

However, much of this information remains fragmented.

Video recordings capture visual activity but may not explain why a surgeon performed a particular action.

Patient monitoring systems capture physiological signals but do not necessarily connect them to specific surgical events.

Robotic systems generate movement and operational data, but these signals may remain disconnected from the broader clinical context.

Even when individual data streams are available, bringing them together into a coherent representation of surgical activity remains challenging.

The problem is therefore not simply a lack of data.

It is the lack of structured, contextualised and synchronised surgical intelligence.

Why surgical understanding is fundamentally multimodal

Consider a surgeon performing a complex procedure.

To understand what is happening, an intelligent system may need to interpret several simultaneous signals.

Visual perception

What anatomical structures are visible? Where are the instruments? What is changing within the surgical field?

First-person perception

What is the surgeon looking at? How does the surgeon's perspective change during the procedure? What actions are being performed from the operator's viewpoint?

Movement and interaction

How are instruments being manipulated? What is the relationship between hand movements, instrument trajectories and tissue interaction?

Audio and communication

What is being communicated between members of the surgical team? Are there verbal instructions, requests or important procedural events?

Temporal and procedural context

What stage of the procedure is underway? What happened immediately before the current action? What is likely to happen next?

Clinical context

How do patient-specific factors, imaging findings and procedural objectives influence what is happening?

No single modality provides the complete picture.

Meaningful surgical understanding requires the ability to connect these different information streams.

Why the surgeon's perspective matters

Most conventional surgical recording systems capture the procedure from an external viewpoint.

These recordings are valuable, but they do not necessarily represent how the operating surgeon experiences the procedure.

A surgeon's perspective is fundamentally different from that of an external camera.

The surgeon continuously changes visual focus, adjusts body position, manipulates instruments and interacts with anatomical structures.

The relationship between what the surgeon sees and what the surgeon does is particularly important.

First-person, or egocentric, perception provides an opportunity to study this relationship.

By capturing the surgeon's viewpoint alongside other operating-room signals, it becomes possible to investigate:

• Visual attention and gaze patterns.

• Hand and instrument coordination.

• Surgical action sequences.

• Interaction with anatomical structures.

• Procedural transitions.

• Differences in technique between surgeons.

This could provide a richer representation of surgical expertise than conventional third-person video alone.

It is particularly relevant to open surgery, where the surgeon's visual field and physical interaction with the surgical environment are central to procedural execution.

From perception to cognition

The development of surgical AI can be understood through three broad technological eras.

The first generation of surgical AI has largely focused on perception.

Examples include instrument detection, anatomical segmentation and surgical phase recognition.

These are important capabilities, but recognition alone does not constitute understanding.

A system may identify a surgical instrument accurately without understanding why it is being used, what objective it serves or how its use relates to the current procedural situation.

The next frontier is contextual intelligence.

This means developing systems that can connect what they see with what is happening, what has happened before and what may happen next.

Eventually, these capabilities could support more advanced forms of predictive intelligence and physical action.

The emergence of surgical foundation models

The development of multimodal foundation models offers an interesting direction.

Rather than developing isolated models for individual tasks, researchers are increasingly exploring systems capable of learning from multiple data modalities and supporting different forms of surgical understanding.

Recent research on specialised operating-room foundation models demonstrates both the potential and the challenges of this approach.

For example, the 2026 ORQA research introduces a multimodal framework combining video, audio, point clouds and structured information to evaluate operating-room understanding.

It also highlights the limitations of general-purpose vision-language models when applied to the spatial, temporal and safety-critical demands of surgery.

This reinforces an important point:

Surgical intelligence requires surgical data and surgical context.

General-purpose AI capabilities alone are unlikely to be sufficient.

Building the data foundation for physical AI

Physical AI refers to intelligent systems that perceive and interact with the physical world.

In surgery, this means moving beyond digital analysis towards systems that can understand their environment and potentially translate clinical intent into physical action.

But this requires a reliable foundation.

Robots need to understand the relationship between movement and outcome.

Intelligent instruments need to recognise the context in which they are being used.

Surgical copilots need to interpret what is happening before providing relevant assistance.

Systems that eventually perform autonomous tasks need to learn from expert demonstrations, structured procedural information and validated interactions.

This creates a fundamental dependency:

The development of physical AI for surgery depends not only on advances in robotics and algorithms, but also on the quality, diversity and structure of the surgical data available to train and validate these systems.

The opportunity for open surgery

An important part of this challenge lies in open surgery.

While minimally invasive and robotic procedures have established video and digital data capture workflows, open surgery presents different challenges.

The surgeon's head, hands and instruments move continuously. The operative field changes. External cameras may not capture the surgeon's actual perspective.

This creates an opportunity to explore new approaches to first-person and multimodal surgical capture.

At Nuevata, our work with IRIS and MIRA is directed towards this broader challenge.

IRIS explores surgeon-centric capture of surgical activity, while MIRA focuses on transforming surgical recordings into structured information and actionable intelligence.

Our longer-term interest is to move beyond video analytics towards a richer understanding of surgical procedures.

The ambition is to help build the data and intelligence foundations for the next generation of surgical technologies.

From recording procedures to understanding expertise

The most valuable information generated during surgery may not simply be what happened.

It may be how, why and in what context it happened.

Capturing these relationships could help make surgical expertise more measurable, transferable and eventually computationally understandable.

That has implications for surgical education, clinical research, performance analysis, predictive assistance and the development of intelligent surgical systems.

The journey towards autonomous surgery will not begin when robots become capable of moving independently.

It will begin when intelligent systems become capable of understanding the complexity of what they are being asked to do.

Before machines can perform surgery, they must first learn to understand it.