Cornell Tech Frontiers of AI Symposium

Abstract. Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc look ups over past tokens. Consequently, they lack an inherent incentive to compress history into compact latent states with consistent transition rules. This often leads to learning solutions that generalize poorly. We introduce Next-Latent Prediction (NextLat), which extends standard next-token training with self-supervised predictions in the latent space. Specifically, NextLat trains a transformer to learn latent representations that are predictive of its next latent state given the next output token. Theoretically, we show that these latents provably converge to belief states, compressed information of the history necessary to predict the future. This simple auxiliary objective injects a recurrent inductive bias into transformers, while leaving their architecture, parallel training, and inference unchanged. NextLat effectively encourages the transformer to form compact internal world models with its own belief states and transition dynamics — a crucial property absent in standard next-token prediction transformers. Empirically, across benchmarks in world modeling, reasoning, planning, and language modeling, NextLat demonstrates significant gains over standard next-token training in downstream accuracy, representation compression, and lookahead planning. Furthermore, NextLat enables variable-length self-speculative decoding, accelerating inference by up to 3.3× in the language domain. NextLat stands as a simple and efficient paradigm for shaping transformer representations toward stronger generalization.

Abstract. Synthetic data has been a effective, if boring set of techniques: prompt some language model to restructure your corpus to match some downstream task, with occasionally some distillation. In this talk, we will take a more expansive view of synthetic data as a general algorithmic tool for generative modeling, arguing that the design space and possibilities of synthetic data are much bigger than it might seem. Through a few recent works, we will show that synthetic data has major benefits beyond transforming the data — improving in-domain perplexities, and enabling unique algorithmic primitives, such as neighborhood smoothing and concatenated ‘mega’ documents. With this broader view, we will point towards a nascent but interesting possibility of treating data itself as an algorithmic object to be engineered and optimized end-to-end.

Abstract. Today's LLMs are powerful, but I argue in this talk that they are still flawed in two ways. First, they are deployed as monolithic systems: even narrowly scoped tasks require a massive full model. Second, they are not native enough: in fact, text abstractions themselves may be unnecessary. In this talk, I will present two recent works that address these issues.

First, we focus on mixture-of-experts (MoE) models, a dominant architecture in LLMs. While MoEs appear to be modular, we show that in practice they are not: restricting inference to a subset of experts causes severe degradation, and this is intrinsic to how they are trained. We show, however, that it is possible to train an MoE such that modularity emerges naturally, without imposing human priors. Our model, EMO, enables selective use of expert subsets — down to 12.5% with minimal performance loss — while naturally organizing experts by domain.

In the second part, I argue for removing text abstractions altogether: humans perceive the world visually, and models should operate directly in pixel space. While ambitious, recent advances in VLMs make this increasingly feasible. I will present PixelRAG, a retrieval-augmented generation model that retrieves web information directly in pixel space. By eliminating complex and lossy HTML parsing, PixelRAG simplifies the pipeline while outperforming text-based RAG, even on text-centric benchmarks like SimpleQA and NQ, and also introduces a new efficiency lever through image compression.

Abstract. Pretraining LLMs at scale is reaching its limits — not in raw benchmark performance, but in the flexibility of what we can do with the resulting model. In this talk, I will argue that the path forward requires rethinking pretraining itself, including the optimizer, the architecture, and the objective. First, I will present a surprising finding: more pretraining can make models worse downstream, harder to finetune and more fragile under quantization. We trace this catastrophic overtraining to a simple culprit: sensitivity to perturbation, which grows steadily over the course of pretraining. Targeting sensitivity directly, through interventions like data curriculum and sharpness-aware optimization, yields up to 40% better downstream performance across both finetuning and quantization.

Next, I will turn to unlearning, which has emerged as one of the central problems for privacy, copyright, and safe deployment of LLMs, and has proven remarkably resistant to post-hoc fixes. I will show why: standard training entangles memorization with generalization in the same neurons. We introduce Memorization Sinks, which exploit learning dynamics to disentangle the two by design. The result is the first natively unlearnable language models, where each of millions of training sources can be cleanly removed by deactivating a small set of neurons, matching a model retrained from scratch without that source.

Finally, I will discuss how to enable creativity in tasks that require a far-sighted leap of thought, like scientific discovery, and argue that next-token prediction is the wrong default for the generative flexibility these tasks demand.

Abstract. Although large language models (LLMs) have shown promise in biomolecule optimization problems, they incur heavy computational costs and struggle to satisfy precise constraints. On the other hand, specialized solvers like LaMBO-2 offer efficiency and fine-grained control but require more domain expertise. Comparing these approaches is challenging due to expensive laboratory validation and inadequate synthetic benchmarks. We address this by introducing Ehrlich functions, a synthetic test suite that captures the geometric structure of biophysical sequence optimization problems. With prompting alone, off-the-shelf LLMs struggle to optimize Ehrlich functions. In response, we propose LLOME (Language Model Optimization with Margin Expectation), a bilevel optimization routine for online black-box optimization. When combined with a novel preference learning loss, we find LLOME can not only learn to solve some Ehrlich functions, but can even perform as well as or better than LaMBO-2 on moderately difficult Ehrlich variants. However, LLMs also exhibit some likelihood-reward miscalibration and struggle without explicit rewards. Our results indicate LLMs can occasionally provide significant benefits, but specialized solvers are still competitive and incur less overhead.

Abstract. Scientific discovery depends on knowledge that is rarely observed directly. Some of it lives in human experts: tacit knowledge and experience that often never make it into papers. Other knowledge is embedded in high-dimensional observations: images, spectra, time series, video, and instrument outputs whose scientific meaning is not immediately visible. In both cases, the challenge is to surface hidden structure and turn it into knowledge that can guide future discovery.

This talk will cover progress toward turning this hidden, or “dark,” knowledge into reusable artifacts. From high-dimensional observations, scientific foundation models can learn representations that reveal structure not apparent in the raw data, such as fine-grained behavior in video or latent variables behind indirect measurements. From human experts, AI systems can begin to recover the tacit expertise behind scientific judgment: what researchers try, what they reject, and how they revise their understanding. Taken together, these efforts point toward a broader ambition: AI systems that make the dark knowledge of science more visible, computable, and useful for discovery.

Abstract. Leveraging large-scale data and computing accelerator systems, statistical learning has led to significant paradigm shifts in many scientific disciplines. Grand challenges in science have been tackled with exciting synergy between disciplinary science, physics-based simulations via high-performance computing, and powerful learning methods.

In this talk, I will describe several vignettes of our research on modeling complex dynamical systems characterized by partial differential equations with turbulent solutions. I will also demonstrate how machine-learning technologies, especially advances in generative AI, are effectively applied to address the computational and modeling challenges in such systems, exemplified by their successful applications to weather forecasting and climate projection. I will also discuss the new challenges and opportunities that future machine-learning research faces.