Research
Large language models
Investigating how language models acquire representations, adapt to specialist work, and reason reliably with evidence.
In brief
What must a language model learn to become useful in specialist work?
- Separate representation learning, current knowledge, and task behavior when defining the research problem.
- Compare adaptation methods against a credible baseline with the same evidence and task interface.
- Evaluate reasoning, source support, abstention, and regressions together.
Language as a substrate for intelligence
Language models offer a common interface to knowledge, reasoning, software, and human intention. Their value depends on how learned representations support a task under realistic conditions. A model may produce a persuasive explanation while missing an exception in the evidence. It may understand a technical term while failing to preserve its meaning across a long workflow. These gaps define a research agenda that connects model learning with observable system behavior.
NYIC’s language-model program focuses on representation learning, adaptation, reasoning with evidence, and evaluation. Our central question is how to build models whose capabilities transfer into specialist environments with clear limits and useful feedback. The program connects scientific questions about learning to engineering questions about deployment. A successful adaptation should improve the intended work, preserve important existing capabilities, and remain economically supportable throughout its lifecycle.
Representations and learning objectives
A language model learns statistical structure through a training objective. Token prediction can encourage representations of syntax, concepts, relationships, and recurring patterns of reasoning. The objective still provides an indirect signal for many qualities enterprises care about. Correct citations, justified uncertainty, and safe tool selection require measurements that reach beyond training loss.
The Transformer introduced an attention-based architecture that became a foundational design for language modeling. Attention lets a representation depend on other positions in a sequence. Its architectural significance does not settle how a particular model represents a domain or whether it uses the right evidence for a decision. Those questions require targeted investigation. Read the Transformer paper.
We organize representation questions around invariance and sensitivity. Which changes in wording should leave an answer unchanged? Which changes in dates, units, entities, or assumptions must alter it? Consider a financial covenant. Paraphrasing its prose should preserve the interpretation. Changing an effective date may change the controlling provision. A useful test set contains both transformations and evaluates the distinction explicitly.
Representation probes can help reveal whether information is recoverable from internal states. Probe success alone cannot establish that the model relies on that information when answering. We therefore pair exploratory analysis with behavioral interventions. Removing a critical fact, introducing a conflicting document, or changing one decisive field provides stronger evidence about the system’s actual dependence on its inputs.
Open the conceptual diagram at full size
Conceptual NYIC diagram. Learned behavior and current evidence meet at inference, with evaluation spanning both.
Adapting a model to specialist work
Domain adaptation can target vocabulary, document conventions, response behavior, or decision procedures. These goals should be separated before selecting a training method. Continued pretraining may expose a model to domain language. Supervised fine-tuning can teach task examples and output structures. Preference optimization can refine choices between plausible responses. Each introduces a different learning signal and a different failure surface.
Parameter-efficient methods provide another dimension of choice. LoRA learns low-rank updates while holding base-model weights fixed. This creates an adaptation path with fewer trainable parameters. Whether that path is adequate for a particular task remains an empirical question involving data coverage, target modules, optimization, and evaluation. Read the LoRA paper.
Our research perspective treats the dataset as an executable specification of intended behavior. A corpus should capture routine examples, hard exceptions, missing information, contradictory evidence, and legitimate abstention. Training only on polished successful answers leaves the model with little guidance about recovery. Training on undocumented expert intuition can encode inconsistency that later appears as model unreliability.
A worked adaptation design
Imagine a model that extracts obligations from agreements for subsequent expert review. This is an illustrative research design, not a report of an NYIC experiment. Its task is to identify an obligation, identify the responsible party, preserve conditions, and cite the controlling passage. A fluent paragraph is insufficient if any of these elements is wrong.
| Training signal | Intended behavior | Held-out challenge |
|---|---|---|
| Annotated clauses | Recover obligation and responsible party | Unseen document templates |
| Conditional examples | Preserve exceptions and prerequisites | Several nested conditions |
| Evidence-linked answers | Attach support to each claim | Similar but irrelevant passages |
| Unanswerable cases | Request missing information | Absent amendments or schedules |
| Correction sequences | Revise a claim after new evidence | Superseded contract language |
The initial experiment should compare a strong prompted baseline, a retrieval-assisted baseline, and an adapted model under the same evidence conditions. Hold out entire document families. Keep amendments connected to their parent agreements when creating splits. Measure field accuracy, evidence support, exception coverage, and the cost of expert correction. This makes a gain interpretable and reduces the chance that repeated templates dominate the result.
Reasoning over long and changing context
Long context changes the system design space. More source material can be presented directly, yet available context capacity does not guarantee reliable use of every relevant fact. The Lost in the Middle study examined how the position of relevant information affected performance on its evaluated tasks. Its findings motivate position-sensitive testing for each deployed model and workload. Read the study.
We distinguish finding evidence, combining evidence, and applying a rule to evidence. A model may retrieve a date correctly while applying the wrong eligibility rule. It may answer a single-document question yet fail when an amendment changes the original terms. Evaluations should therefore vary evidence distance, distractor similarity, version order, and the number of necessary reasoning steps independently.
Context also evolves over time. An enterprise agent may accumulate tool outputs, user corrections, and intermediate conclusions. The model needs a reliable account of which observations are current and which claims remain unresolved. Research into context compression should measure the preservation of decision-relevant detail. A shorter memory that drops an exception can degrade the entire workflow even when its summary sounds accurate.
Evaluation as a research instrument
Our evaluation approach begins with a task contract. The contract describes admissible inputs, expected outputs, relevant evidence, unacceptable errors, and conditions requiring review. It then identifies measurable properties. This prevents broad claims such as “better reasoning” from substituting for a defined improvement in a particular setting.
A useful scorecard includes task completion, factual support, uncertainty handling, robustness, resource consumption, and behavior under distribution shift. Results should be disaggregated by task difficulty and failure severity. A model that performs well on frequent easy examples can still be unsuitable when a rare error carries a large consequence.
Human judgment has an important role, especially when several answers are defensible. Reviewers need written rubrics, calibration examples, and a process for disagreement. Model-based evaluators can accelerate screening, but their judgments should be checked against expert review and simple deterministic tests. Output schema validation, numerical reconciliation, and citation existence are often more reliable when implemented directly.
Open questions and limits
A persistent research challenge is disentangling knowledge from behavior. An adapted model may learn a response convention without improving the underlying decision. Conversely, it may improve extraction accuracy while losing useful general abilities. Measuring both target gains and broader regressions is essential when deciding whether an adaptation is ready for production.
Another challenge is evidence about causality. An explanation generated after an answer can be useful for communication without faithfully describing the computation that produced it. We avoid treating polished explanations as proof of internal reasoning. Observable checks, controlled interventions, and independently verified calculations provide a firmer basis for confidence.
Language models also operate within data and compute constraints. Limited representative training data, incomplete source documents, and restricted access can place a ceiling on performance. A responsible research agenda identifies those constraints and makes them visible in the system design. Increasing model size cannot recover facts that were never available to the workflow.
From research to enterprise systems
The enterprise implication is a modular approach to capability. Learned behavior, retrieval, deterministic computation, and human review each serve distinct purposes. Models can interpret language and propose actions. Authoritative systems can provide current facts. Validators can enforce schemas and arithmetic. People can resolve ambiguity where judgment or accountability matters.
NYIC connects language-model research to these deployment boundaries. Research questions become experiments with clear baselines. Findings become architecture decisions and release criteria. Production observations become candidates for new evaluation examples. This feedback cycle allows the research program to remain grounded in the behavior that users and institutions actually depend on.