Research
Multimodal models
Connecting language, images, audio, video, and documents through grounded representations and traceable evidence.
In brief
How can different forms of evidence support one defensible interpretation?
- Preserve source identity, measurement definitions, and temporal alignment across modalities.
- Choose fusion methods around the task and the information each modality contributes.
- Test missing inputs, extraction errors, and conflicting evidence separately from answer fluency.
Intelligence across forms of evidence
Enterprise knowledge rarely arrives in a single clean format. A presentation combines prose, charts, footnotes, and visual emphasis. A meeting adds speech, timing, and uncertainty. A process produces video, sensor readings, and structured records. Multimodal intelligence seeks useful relationships across these forms while preserving the differences that make each source informative.
NYIC’s multimodal research program focuses on cross-modal representations, temporal grounding, document understanding, and evidence reconciliation. We ask how models can combine complementary signals without losing provenance or concealing disagreement. The goal is a system that can explain what supports a conclusion, which sources remain ambiguous, and what additional evidence would change the answer.
A convincing multimodal response requires several capabilities at once. The model must perceive relevant content, identify corresponding entities, align time and context, apply appropriate reasoning, and communicate uncertainty. A failure at any stage can produce a polished answer with a weak evidentiary foundation.
Representation and alignment
Different modalities encode different structures. Text is sequential and symbolic. Images carry spatial relationships. Audio adds frequency, timing, and speaker information. Video joins visual change with temporal order. Structured tables encode relationships through rows, columns, units, and labels. A shared representation must retain enough of these structures for the task at hand.
CLIP demonstrated a contrastive approach to learning visual representations from paired images and text. It provides a foundational example of cross-modal alignment. That learning objective creates useful relationships between representations, while enterprise applications still require task-specific tests for precise reading, source grounding, and domain shift. Read the CLIP paper.
Alignment is broader than matching a caption to an image. A system may need to associate a spoken statement with a particular slide, connect a table cell to a footnote, or identify which equipment state a maintenance note describes. These relationships can be many-to-many and uncertain. Preserving them explicitly helps downstream reasoning avoid treating approximate matches as settled facts.
Early fusion, late fusion, and structured intermediates
One architectural choice concerns where modalities interact. Early interaction can allow a model to learn fine-grained relationships across raw or encoded inputs. Later fusion can combine outputs from specialized components. A third approach introduces structured intermediate representations such as document regions, transcript segments, detected events, or normalized measurements.
Each choice changes the error surface. A specialized chart parser can make units and series labels inspectable, yet a parsing error can propagate into every later answer. A general multimodal model can use visual context directly, yet its interpretation may be harder to validate. A robust architecture should identify where evidence becomes a structured claim and attach checks at that boundary.
Our research perspective favors explicit comparison of these alternatives. The evaluation should use the same tasks and source material, include difficult documents, and measure end-to-end behavior. The relevant question is which design preserves the information needed for reliable answers under realistic resource constraints.
Open the conceptual diagram at full size
Conceptual NYIC diagram. Each modality contributes evidence whose source, timing, and uncertainty remain traceable.
A worked evidence-reconciliation example
Imagine an analyst reviewing a synthetic quarterly presentation, a transcript, and a spreadsheet. The chart shows growth in a business line. The transcript discusses a decline in a narrower geographic segment. The spreadsheet contains a restated historical value. A superficial summary might describe these sources as contradictory even though they refer to different scopes.
| Evidence item | What must be preserved | Common failure |
|---|---|---|
| Chart | Series, axes, units, period, footnotes | Reading visual height as an exact value |
| Transcript | Speaker, timestamp, qualifying language | Dropping a condition or uncertainty |
| Spreadsheet | Formula context, units, revision date | Mixing original and restated figures |
| Slide title | Business scope and framing | Treating a headline as quantitative proof |
| Source metadata | Version, publication time, origin | Combining incompatible reporting periods |
A grounded answer first establishes whether the sources describe the same entity, measure, period, and basis. It then separates compatible statements from unresolved conflicts. If a chart reports constant-currency growth and a table reports nominal revenue, the answer should preserve that distinction. It should avoid inventing a reconciliation when the necessary exchange-rate information is absent.
This workflow illustrates a research principle. Evidence combination should include a consistency model. The system needs to know which properties must agree for two statements to be directly comparable. That model can be partly deterministic, partly learned, and explicitly reviewed for consequential tasks.
Temporal grounding in audio and video
Time is a first-class dimension of multimodal understanding. A statement may refer to an event that happened earlier in a recording. A visible condition may last only briefly. Audio and video may be offset, edited, or incomplete. A model that recognizes content without grounding it in time can misattribute actions and causes.
We examine temporal granularity as a design variable. Frame-level analysis can preserve brief events but increase cost. Segment-level summaries can reduce volume but erase transitions. Event representations can make sequences easier to inspect while depending on the quality of event detection. Evaluation should include the smallest time interval that matters for the intended decision.
Speaker attribution deserves similar attention. A transcript can contain a correct sentence assigned to the wrong person. In a meeting summary, that changes who made a commitment. In a research call, it changes whether a statement was an executive assertion or an analyst question. The evaluation should score attribution and content separately.
The Qwen2.5-Omni report presents a model spanning text, images, audio, and video, with streaming text and speech output. Its architecture includes explicit treatment of audio-video timing. This illustrates why multimodal research must consider synchronization and response generation alongside recognition. For enterprise evaluation, we ask whether those capabilities preserve source attribution and task-specific evidence under realistic recording conditions. Read the technical report.
Missing, degraded, and conflicting modalities
Production inputs are imperfect. Scans can be rotated. Charts can be compressed. Speech can be obscured by noise. Video can lose frames. Some modalities may be unavailable because of access restrictions. A useful system should degrade predictably and communicate which conclusions remain supported.
Our proposed stress tests remove or corrupt one modality at a time. They also introduce plausible contradictions. For example, a transcript can omit a negation while an audio segment retains it. A table can contain a corrected number while an older slide remains in the packet. These tests expose whether the model privileges one modality automatically or reasons about source quality and version.
Conflict resolution should follow task-specific authority rules. An approved final record may supersede a draft. A raw sensor value may require calibration before comparison with a report. There is no universal rule that images, text, or audio should always win. The system needs a documented policy appropriate to the workflow.
Evaluation that locates the failure
An end-to-end answer score is useful for understanding overall performance. It is insufficient for choosing an improvement. We separate perception, alignment, reasoning, and communication. Perception asks whether the source was read correctly. Alignment asks whether related evidence was matched. Reasoning asks whether the conclusion follows. Communication asks whether support and uncertainty are represented faithfully.
A document evaluation might include exact values, unit correctness, table structure, evidence regions, and answer support. An audio evaluation might include transcription, speaker attribution, timestamp accuracy, and preservation of uncertainty. A video evaluation might include event recognition, ordering, duration, and causal restraint. These measurements should be paired with realistic tasks rather than isolated recognition benchmarks alone.
Human review should expose the source alongside the model’s claim. Reviewers can then identify whether an error originated in extraction or inference. Disagreement between reviewers is itself useful evidence about task ambiguity. Such cases may need revised instructions, alternative acceptable answers, or a designated escalation path.
Enterprise implications and open research
Multimodal systems can help people navigate document collections, inspect operational evidence, understand meetings, and connect signals across formats. The engineering challenge is to maintain a traceable relationship between an answer and its source. This relationship should survive preprocessing, retrieval, summarization, and subsequent model calls.
Several research questions remain difficult. How should models represent uncertainty when modalities disagree? How can alignment remain stable across long recordings? Which details should a compressed representation retain for future questions that are not yet known? How can evaluations distinguish useful synthesis from plausible interpolation?
NYIC’s program connects these questions to architectures that preserve source identity, temporal context, and reviewability. The resulting systems should help users investigate evidence more effectively. Their strongest contribution is the ability to combine information while making the basis of that combination visible and testable.