Mike Beaumier

Some recent, exciting news on compressed tokenizers and how individual tokens modify images in encoder-decoder models. I hope this opens up a new avenue of research with smaller models across frameworks (perhaps JEPA?). https://lnkd.in/evFw8WBu

69

2 Comments

“Ambiguity is genAI’s kryptonite.” -- Dave McComb, CEO of Semantic Arts Most enterprises are stuck in AI proof-of-concept purgatory—not because their models aren't working, but because their data infrastructure wasn't built for AI-scale operations. Travis Van (InfoWorld) synthesizes insights from leading data infrastructure and AI experts on the critical gap between AI experimentation and production deployment. Key Voices: Dave McComb (Semantic Arts) - CEO Philip Rathle (Neo4j) - CTO Grant Miller (Replicated) - CEO Brian Gruttadauria (HPE) - CTO of Hybrid Cloud Anthony J. Annunziata (IBM) - Director of AI and Open Innovation Miqdad Jaffer (OpenAI) - Product Lead Three barriers blocking AI production deployment: Data Accessibility: Legacy systems create silos where AI agents can't reach the context they need—traditional data catalogs weren't designed for the real-time context retrieval AI agents require Semantic Integration: McComb and Rathle emphasize knowledge graphs as critical infrastructure for connecting disparate data sources with business meaning and relationships at scale Infrastructure Complexity: Gruttadauria, Miller, and Annunziata highlight how enterprises struggle operationalizing AI workloads across hybrid environments—the infrastructure layer itself becomes a bottleneck The shift happening now: Organizations are realizing they need a new layer—not replacing existing data warehouses or lakes, but adding an AI-native context layer that bridges the gap between where data lives and where AI agents need to access it. Jaffer's insights from OpenAI emphasize rethinking data architecture around agent workflows rather than forcing AI to work within human-designed query patterns. The companies winning at AI aren't necessarily those with the best models—they're the ones who solved the data access problem first. At Immersa, we are working diligently at solving this precise problem. #EnterpriseAI #DataInfrastructure #AIAdoption #ContextEngineering #DataArchitecture #AgenticAI #KnowledgeGraphs #DataStrategy

8

We recently released a preprint paper demonstrating how a consensus group of individual prompts with evolved contexts by large language models (LLMs) can significantly enhance diversity and overall performance in challenging agentic tasks. You can read more about our findings at https://lnkd.in/gmMw-d2W. We call it C-Evolve (Concensus Evolve), and it is done without needing to train LLMs' weights. #c-evovle #CompoundAISystems #ContextOptimization

23

If you do functional programming (like in Wolfram Language) you've probably used lots of pure functions, or lambdas. But what are lambdas like in the wild? Things I'm doing in CS, bio and ML converged to make me curious to find out... And as seems to happen whenever I go exploring in the computational universe ... they surprised me ... https://lnkd.in/eEY2euBD

735

23 Comments

In my previous post I wrote about fine-tuning an in-house LLM for travel recommendations. This post is about why owning the model stack turns latency into an advantage. (previous post - https://lnkd.in/dZS6934P) One of our recent papers, “Speed Without Sacrifice” (link in comments 👇), won the Industry Track Best Paper Award at ACL 2025. For me, the most interesting part is not the award (though kudos to our amazing team including Moran Beladev, Manos Stergiadis, Ilya Gusev and Eran Fainman), but what it says about building GenAI systems at scale. When you rely on third-party models, latency is a fixed parameter. You pick a model. You pick a tier. You live with the response time. With open-source models, latency becomes something you can actively design for. Together with our long-time partners at AWS, Daniel Zagyva, Laurens van der Maas, and Aleksandra Dokic, we explored how owning the full stack lets you optimize not just prompts or infrastructure, but the model architecture itself. Huge thanks to them (and the AWS team including our partner in crime Eran Bachar) for an outstanding collaboration over the last three to four years. This paper shows how we did that in practice. We used knowledge distillation to transfer quality from very large teacher models into much smaller students, preserving task performance while dramatically reducing cost and memory footprint. We then applied speculative decoding, using Medusa-style heads to parallelize token generation. Instead of treating inference as a black box, we changed how decoding works under the hood, achieving large speedups without sacrificing output quality. The key insight is that these techniques compound. Distillation makes models smaller and cheaper. Speculative decoding makes them faster. Combined, they delivered 10–20x latency improvements in real production systems like AI Trip Planner, Smart Filters, and large-scale content generation. The bigger takeaway is not a single optimization. Over time, we have built an internal recipe for latency control across open-source LLMs. Distillation. Speculative decoding. Quantization. Task-specific training. KV caching. Careful serving. Once you own this toolbox, you can apply it to almost any task, and tune the speed-quality-cost tradeoff intentionally instead of accepting it as given. This is why open-source models are so powerful in practice. Latency is no longer a constraint you work around. It is a dimension you control. And it is yet another great example of the kind of deep, hands-on GenAI work the team is doing to build fast, reliable, AI-first travel experiences for our customers. (and yes we are hiring in TLV and AMS - ping Moran Beladev for open roles)

45

1 Comment

Thanks to the resurgence of RL, LLMs are finally able to reliably coordinate tools and reasoning to do high-precision retrieval. Companies like Happenstance, Clado, and Mintlify have already shifted to agentic search, and it's only a matter of time until anything less feels broken to users. Link to full blog in comments.

20

2 Comments

Wrote a second blog post! This time it's about using LLMs for imputing missing data. I've seen a few papers about imputation with LLMs, but most seem to either train custom imputation transformers or horserace flagship models against standard methods. Mert Demirer and I recently played with an alternative idea which instead uses LLMs as part of an ensemble imputation approach, relying on their "knowledge" of the world to provide additional prediction signal. I haven't seen a paper about it yet, so we thought it'd be fun to make a short post: https://lnkd.in/gMzC6iaP

232

8 Comments

Check out our new blog post “Prompt Trees: Training-time Prefix Caching,” by the research team at Scaled Cognition. https://lnkd.in/dDYF2VXz TL;DR: Training speedups of up to 70x on tree-structured data. Not 70%. _70x_. How do we do this? Let's consider the case of RL rollouts for a conversational agent (this is just one possible example of many). You typically have many rollouts at each turn - this is a tree. In standard transformer training, you need to duplicate the prompt prefix for each rollout, compute gradients, and take a step (e.g., trl does this). That's a lot of duplicate encoding, especially for long conversations or many rollouts. How do we remove the duplicate encoding? The key insight is that token encoding only depends on other tokens by means of the position IDs and attention mask - if we can set those appropriately for a tree structure, we can use standard transformers to encode trees. It turns out you can use pytorch's flex attention to construct a tree-structured attention mask for a prompt tree. It's very efficient if you use a binary encoding of a node's path to compute whether one token is in the prompt prefix of another token. With this attention mask and correctly-offset position ids, the encodings that you get (and gradients that you compute) are _exactly_ the same as if you had encoded each path through the tree as a separate linear prompt (modulo numerical stability). How well does this work? That depends on a lot of things, but the biggest factor is how dense your prompt trees are. If your data is intentionally constructed to make good use of prompt trees, the speedup can be very large. We ran timing experiments on our cluster, measuring the relative speedup as a function of how dense the prompt tree is. The speedup is roughly linear in the proportion of duplicated tokens in the tree, with empirical speedups of up to 70x. It's not exactly linear, for various reasons we discuss in the post, but it's empirically nearly so. Note that this is a speedup for the gradient computation only, and it's one that potentially changes optimization dynamics (e.g., you're packing more information into each batch) As we're coming out of stealth, we're excited to be sharing more with the community! (We'll be starting with projects which are at the periphery of our tech for now and saving our core agentic modeling tech for later.) We'll be at NeurIPS, happy to chat, and we're hiring 🙂

35

The more I experiment with RAG, the more I realize smarter chunking matters far more than fancy models. One thing that surprised me recently is how much chunking influences retrieval quality. Most of the issues we blame on embeddings or model choice or prompt actually start with how we split our documents. Sometimes chunking improves precision. Other times it breaks valuable context and hurts performance. My takeaway from hands-on learning: 🔹 Chunk when documents are long, multi-topic, or noisy 🔹 Avoid chunking when documents already fit cleanly into the model’s context window This small shift made my RAG experiments noticeably more accurate and easier to debug. #RAG #LLM #AIEngineering #VectorSearch #EnterpriseAI #AI

29

3 Comments

Decentralized (model parallel) training, to this date, has been considered infeasible, primarily because of the extreme sensitivity of large-scale model training to communication latency. Training any serious model involves a very long sequence of forward and backward passes, where activations and gradients must be exchanged rapidly across devices. Normally, this is only feasible because the GPUs are co-located in tightly synchronized clusters with high-speed interconnects. Even small bottlenecks such as 100ms delays between consumer devices on Wi-Fi can be really catastrophic. Today we share a new paper that shows, for the first time, the ability to train billion-scale models over the open internet using low-end GPUs. Our compression algorithm can compress both activations and gradients by over 100x with no degradation in convergence or final performance. For the first time, this allows billion parameter models to be trained across globally distributed low end GPUs, connected only via standard internet, and match the performance of centralized training using high-bandwidth interconnects. In the coming days, we’ll launch the first ever fully volunteer-driven decentralized model training run, where anyone around the world can join and co-train a foundation model over internet. Preprint: https://lnkd.in/gUryp3Bs Register for the run → https://lnkd.in/dEKVEesZ

57

12 Comments

📄 It's paper Friday! 🧠 Is there an underlying semantic structure common to all word embeddings? A recent paper from a group of researchers at Cornell University makes the case that the "Platonic Representation Hypothesis" (the idea that embeddings have a common underlying representation space) holds true. Closer Look: The Challenge * Given two embedding models, the corresponding vector spaces are usually completely incompatible with each other * Even if they are both encoding the same text, you can't just match them vector-to-vector or apply tools designed for one model's embeddings to another model's embeddings The Idea: Universal Latent Representation Inspired by a paper in image processing that argues that "representations in AI models, particularly deep networks, are converging", the authors propose a Strong Platonic Representation Hypothesis where there's an underlying universal semantic structure that all text embedding models converge toward and build an adversarial network (a machine learning method where two networks compete) that allows translating unknown embeddings so that information extraction tools designed for known embedding spaces can be applied to them. The results are very telling: Their method achieves 0.92 cosine similarity and perfect matching on over 8,000 shuffled embeddings. A Call for Some Concern: Security Implications An adversary with access to just a vector database could use this method to extract sensitive information about the original documents - like medical records or private emails - without even knowing which model created the embeddings. Though the authors don't dive deep into the security implications, this opens up some questions about vector database security. My Take When I first read this I thought about Chomsky's universal grammar - the idea that despite surface differences, all human languages share deep structural principles. Could embeddings be revealing something similar about how meaning itself is structured? What are your thoughts? Let me know in the comments! #AI #MachineLearning #LLM #Research

11

2 Comments

Hugely important issue covered here, especially as teams often perceive surveys as the cheap or ‘easy’ method. If you can’t trust the quality of the data you are gathering, all of the time, money (and political capital) invested into the study is wasted. Screening & recruitment are some of the most important parts of the research process - part of the reason I favour qualitative moderated methods when it aligns with the team’s objectives.

42

3 Comments

In Anthropic's Agentic Coding Trends Report, they mention "perhaps the most valuable capability developments in 2026 will be agents learning when to ask for help, rather than blindly attempting every task, and humans stepping into the loop only when required." That's why we are releasing our latest research at Scale AI: Long Horizon Augmented Workflows (LHAW). LHAW is a synthetic data generation pipeline for creating underspecification on *any* dataset and evaluating how agents react. LHAW transforms well-specified long-horizon tasks into controllably underspecified variants using a three-phase pipeline: segment extraction, candidate generation and empirical validation. We generate & validate 285 ambiguous task variants across MCP-Atlas, TAC, and SWE-Bench Pro Finding #1: Clarification recovers meaningful performance, but not fully. Access to a simulated user significantly improves success on underspecified tasks (+31% Pass@3 for Opus4.5 on MCP-Atlas), yet agents are not able to fully recover original performance. Finding #2: Models vary widely in clarification strategy: GPT-5.2 spams, Gemini models underask. Some models extract high value information per question. Others ask far more frequently, achieving gains but with lower value per interaction. We measure this with with Gain/Question Finding #3: Clarification behavior adapts to cost. As expected, when interaction is “cheap”, agents ask more but gain less per question. When interaction is “expensive”, agents ask less but extract more value per question at higher risk of failure. Finding #4: Clarification failure-modes vary from widespread to model-specific. Certain failure-modes like poor question quality, underclarification, and question targeting apply across models. Some models show particularly bad tendencies to overclarify or misinterpret a response. As agents take on longer tasks, we want to know how they act under uncertainty and how much they burden us with their questions :) LHAW provides a way to create these tasks, evaluate clarification strategies, and (soon) train agents for reliability under real-world ambiguity. This work was led by George Pu and Mike Lee with contributions from Udari Madhushani Sehwag, David Lee, Bryan Zhu, Yash Maurya, Mohit Raghavendra, and Yuan (Emily) Xue Blog: https://lnkd.in/gp768At9 Full Paper: https://lnkd.in/gVTjemmv Dataset: Hugging Face https://lnkd.in/gTjVrszU

154

5 Comments

An important question I get asked is, why is RL important and is supervised finetuning enough for LLMs? I spent the weekend putting together a substack and a youtube video explaining the difference between RL and SFT, and the theoretical foundations for why RL works in challenging domains. I highly recommend this if you are working with LLMs or just curious to understand how they are trained. You can also find the substack version here: https://lnkd.in/eKewpccT https://lnkd.in/eNjYbWzh

7

2 Comments

Anthropic's recent blogpost discusses the important shift from evaluating simple LLMs ---> testing complex AI agents.  It highlights why standard benchmarks fall short for agentic workflows and offers a practical roadmap for building robust, multi-turn evaluations." Packed with insights, but couple of takeaways that stood out to me:  🎯 1. 𝐂𝐨𝐦𝐩𝐨𝐮𝐧𝐝𝐢𝐧𝐠 𝐯𝐚𝐥𝐮𝐞 𝐨𝐟 𝐭𝐡𝐞𝐬𝐞 𝐬𝐲𝐬𝐭𝐞𝐦𝐬: where costs are visible upfront but their benefits accumulate over time. (quicker iterations, faster deployment) 🎯 𝟐. 𝐓𝐡𝐞 𝐂𝐨𝐧𝐬𝐢𝐬𝐭𝐞𝐧𝐜𝐲 𝐅𝐢𝐱 / 𝐃𝐞𝐭𝐞𝐫𝐦𝐢𝐧𝐢𝐬𝐦 𝐢𝐧 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧𝐬: Regardless of agent type, agent behavior varies between runs. Sometimes, we want to measure how often (what proportion of the trials) an agent succeeds for a task, while for others, we want to measure if Agent succeeds atleast once. 🚀 Single success is enough across runs ? Use pass@k. 🚨 Need reliability across runs? Use pass^k. The post contains many more valuable insights worth exploring. Read more here: https://lnkd.in/gwCxqhDj

59

2 Comments

Icymi: neat new analysis from Charles Yang uses ORCID profiles and github links, along with public Cluade Code commits, to track scientific adoption of Claude code. It's clearly incomplete, but as a first indication of early adopters, he finds a 2% adoption rate of Claude Code. There is a U-shaped adoption curve, and economists seem to be adopting at roughly double the rate of other scientists. (Link in comment.) These are neat findings, and I look forward to more and better data sources that allow us to track agentic coding option. Right now, the conversation is heavily driven by social media. The adopters and those who see large benefits are undoubtedly the largest. Less represented are those who haven't adopted because of up-front costs (time, money, or both), organizational constraints or organizational policies, or other reasons. Moreover, it's a reasonable hypothesis that there are strong complementarities between preexisting coding skills and use of Claude code, in which case the earliest adopters may have the highest returns. More evidence is urgently needed!

31

1 Comment

New: Databricks is launching a startup accelerator for seed and pre-seed companies, offering up to $250,000 to startups in the form of funding and credits https://lnkd.in/eHjG-9T8

33

3 Comments