Max K.

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

We developed an open-source AI tool to design concrete mixes that are stronger, more sustainable, and ready to build with faster—speeding up construction while reducing environmental impact. https://lnkd.in/gPCk8tCM But the impact of this AI tool is not just hypothetical! Amrize used Meta’s AI-based technologies to design a new low-carbon mix, and successfully deployed it in an at-scale slab-on-grade application at Meta's new data center in Rosemont, MN. Compared to the legacy mix, this new AI-designed mix is: 🦁 Stronger ⏱ Faster 🍃 Lower carbon ⏱ ️The ideal set time All this was achieved without needing any new materials, nor special equipment. Best of all, the AI is open-sourced. https://lnkd.in/g2KA7KZW This work was featured in a Meta engineering blog article published today! https://lnkd.in/gBU9HY8H

98

9 Comments

There is this paper from Stanford that posits that the path to AGI runs through LLMs - I think the more interesting part it discusses is LLMs as System-1 substrate, and Agentic Systems as the System-2 coordination layer that selects, constrains and binds these LLMs to perform meaningful tasks. They highlight three areas of focus: 1/ Semantic Anchoring - binding the LLMs through examples, metaphors, RAG, tuning etc. 2/ Socratic Filtering - use multiple models to ‘debate’ to pass the reasonableness test as part of the reasoning process 3/ Independent judges - AI judges anchored in the domain that block out weak responses that are not aligned to the overall goal 4/ Transactional memory - that persists states over time. While all these are part of most Agentic Systems that we build today, there are some interesting ideas under each of them that can enrich Systems design. One interesting concept in the paper is ‘Reasoning as a Phased Transition’ where a tiny amount of context can override the pre-trained context, producing an abrupt flip in behavior. Something we all observe empirically in AI systems today- the paper develops a formal definition for identifying the threshold and provides an important insight: small boosts in the above-listed areas can shift results from generic to targeted, anchored in the domain context. This is not new, we see this in many physical and biological systems where smooth changes in a control variable yield sharp changes in the system. Lots of interesting ideas with direct applications to building Enterprise AI systems.  https://lnkd.in/gvCDXMtg

27

8 Comments

NVIDIA TensorRT LLM provides a high-level Python LLM API, with its PyTorch-native architecture enabling developers to experiment with the runtime or extend functionality. Learn how these latest TensorRT-LLM optimizations boost reasoning inference performance in our recent blog post. 🖇️ https://lnkd.in/gx52zwTA #PyTorch #OpenSourceAI #AI #Inference #Innovation

94

2 Comments

Positional encoding and persistent homology have been powerful tools for graph neural networks. This very recent ICML paper combines the two methodologies to leverage both algorithm strengths. It's an interesting new direction for TDA + neural network architectures. https://lnkd.in/eskZZfsW

92

2 Comments

Two regimes. Two platforms. Two different shapes. I've been testing my Two State Ontology (TSO) framework on quantum hardware — IBM Marrakesh (superconducting qubits) and Quandela (photonic). TSO predicts that the shape of decoherence depends on whether a system is being actively driven through a critical interaction threshold (Γ_c). Below Γ_c (idle qubits, no interaction): standard exponential decay. At Γ_c (driven dynamics, active coupling): sigmoid/tanh transition. This week's results: • IBM Marrakesh idle decay: Both superposition (X2) and entanglement (X1) circuits decay exponentially. ΔAIC > 30 — decisive. This is what TSO predicts when no path rotation is occurring. • Photonic bunching cascade (Quandela SLOS): tanh preferred over exponential by ΔAIC > 25. Measured κ = 1.325 against TSO's prediction of 4/3 = 1.333 — a 0.6% deviation. • Kim et al. Rydberg re-analysis (129,791 shots): tanh beats exponential 2.75×. The pattern: standard QM below the threshold, TSO-specific behavior at it. Same relationship statistical mechanics has with thermodynamics — ordinary behavior in most regimes, critical phenomena only at phase transitions. TSO has zero free parameters. All constants derived from established physics. The decisive test remains: Rydberg atom arrays with tunable interaction strength through Γ_c, where TSO predicts ~35% deviation from exponential. All notebooks public. All failures documented (including 3+ killed versions). The framework evolved from first principles before any hardware testing. https://incapp.org/TSO/ #QuantumComputing #Physics #QuantumFoundations #OpenScience

Read Chaim Rand's practical walkthrough for optimizing PyTorch inference. Learn about batched inference, memory formats, IPEX, and graph compilation on Intel Corporation Xeon.

5

This is a fantastic article on using Claude Code for complex #computationalphysics problems. The typical way of using #GenAI for #research is basically you sitting there, holding the agent’s hand through every single step and then #validate. But now that models have gotten a lot better at long, complex tasks, there’s this sort of new paradigm emerging — you define the high-level goal, set a team of agents loose, and go grab several coffees throughout the days of the experiment. The article walks through a concrete example: a researcher at Anthropic (not even a cosmologist) using Claude Opus to build a differentiable Boltzmann solver — essentially code that models the #physics of the early universe right after the Big Bang. The secret sauce is basically a handful of #engineering patterns that is worth sharing: 1. A clear “mission file” (CLAUDE.md) that lives in the project root and tells the agent exactly what it needs to build, what the success criteria are, and any important design decisions. The agent can also update this file as it learns things. 2. A progress/changelog file that acts as long-term #memory across sessions — what worked, what didn’t, why certain approaches were abandoned. Without this, every new session may end up repeating the same dead ends. 3. A test oracle — in this case, the reference implementation (the Cosmic Linear Anisotropy Solving System, a well-established #cosmology code) — so the agent always has a way to check whether it’s actually making progress or just pretending. 4. Using #Git as a coordination mechanism: commit and push after every meaningful chunk of work, so we have a full #recoverable history and can check progress remotely. 5. The “Ralph loop” — a wrapper that pings the agent when it claims it’s done and asks “are you really done?” This fights a known #failure mode where agents give up a bit early on complex tasks. The result? A sub-percent #accuracy against the reference implementation, running on an HPC cluster over a few days. Not production-grade, but remarkably close for something built autonomously with minimal steering from someone outside the field. It includes the #repository with code and details of the implementation, similar in style of Andrej Karpathy, reporting his experiments during the last few weeks. https://lnkd.in/gsD4ESDx

59

2 Comments

It had been on my list for a while to read the SID-1 technical report after seeing a post about how using OpenAI-style messages in RL can be surprisingly dangerous in multi-turn settings with many tool calls. The first insight is about the messages abstraction. Converting a raw token stream into messages and then back into tokens is lossy because it changes how the exact byte sequence is tokenized. A concrete example is when the model generates a sequence of bytes that the tokenizer splits into two very common tokens with normal probabilities. After parsing and reformatting through the chat template, those same bytes can be re-serialized in a way that the tokenizer now maps to a single, very rare token that the model almost never produces. The reward is computed on the original tokens the model actually generated, but the log-probs used for the update correspond to this new token that was never sampled. Since this token has an extremely low probability, its log-prob has a large magnitude, and in policy gradient this translates into a disproportionately large gradient. That single artificial token can end up dominating the update, creating a feedback loop that gradually destabilizes training and eventually leads to collapse. There is a second, distinct effect happening at the same time. Malformed tool calls or slightly wrong formatting can get “repaired” by the parser and chat template, so bad rollouts end up looking syntactically valid before they ever reach the trainer. The environment sees a correct tool call, the reward is good, and the trainer also sees a correct sequence. The model never receives signal that it actually produced something malformed. Stability is preserved, but the model does not learn tool correctness because the fixing layer steals the learning signal. Their fix is conceptually simple. Use a strict Tokens-In Tokens-Out pipeline, where the trainer sees exactly the token sequence the model generated. No parsing, no chat templates, no message abstraction in between. Another very interesting part of the report is their analysis of the “length debiasing” proposed in Dr. GRPO and similar works, where the per-token advantage is no longer normalized by the rollout length. This assumes that rollout length and quality are unrelated. In tool-use and reasoning-heavy environments this is not true. Bad rollouts tend to be longer than good ones. In that case, removing the length bias makes the average per-token advantage negative. Over long runs, this slowly pushes down the logits of all sampled tokens while pushing up the logits of tokens the model never uses, until the model starts emitting garbage or out-of-vocabulary tokens and collapses.

7

2 Comments

Transformer attention layers operate by projecting high-dimensional token representations into low-rank subspaces defined by per-head key/query/value projections. This inherently places a rank constraint on how relationships between items can be represented, and it forces the model to approximate rich interactions via combinations of rank-limited components. Our paper “Orthogonalization of data via Gromov-Wasserstein type feedback for clustering and visualization” provides a theoretical and algorithmic framework for dealing with full and low-rank structure in relational data representations by adapting transition probabilities and iteratively refining cluster orthogonality via Gromov-Wasserstein-inspired feedback. It shows how a low-rank Markov transition representation can be refined to better reflect true structure and interpretability, and how such refinements converge to meaningful solutions with spectral gap enhancement. This connects directly to issues in multi-head attention: -Both involve representing pairwise relationships (attention scores vs. transition affinities) in a full-rank or approximately low-rank form. -Low-rank factorization can produce spurious grouping or similarity structure that doesn’t reflect the true geometry, unless guided by careful feedback or regularization. -The GW-feedback orthogonalization that we exemplify with shows some ideas how to improve, stabilize, and interpret the geometry of such low-rank relational structures, offering insight into how optimization can go astray in ill-conditioned, low-rank parameter spaces, but also how the implication of the approximation can be assessed. Give it a read: Low-rank constraints are not just computational conveniences — they have real geometric and optimization implications, and without appropriate mechanisms (feedback, orthogonalization, spectral refinement), learned low-rank representations (like attention maps or transition matrices) can be misleading or unstable. I’d like to hear your thoughts on this. Is it needed? Should we care? What’s a good future architecture? What should I read? Merry Christmas!

10

⚡ 10s AI News ⚡ Soumith Chintala Chintala, co-creator of PyTorch, is now CTO of Mira Murati‘s Thinking Machines Lab. This move pairs 11 years of Meta infrastructure expertise with OpenAI’s product legacy to solve AI scaling limits. With PyTorch used in over 80% of research, Chintala will drive vertical integration between frameworks and frontier models. This consolidation of elite talent aims to optimize compute efficiency as the startup targets a $50 billion valuation. 🚀 #AI #PyTorch #TechLeadership #ThinkingMachineLabs #AIMarketNews #WeeklyVentures

1

In this episode, we discuss Reinforcement Learning for Reasoning in Large Language Models with One Training Example by Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, Yelong Shen. The paper demonstrates that reinforcement learning with verifiable reward using only one or two training examples (1-shot RLVR) substantially improves mathematical reasoning in large language models, nearly doubling performance on benchmarks like MATH500. This method generalizes across different models, algorithms, and examples, showing unique phenomena such as post-saturation generalization and the importance of policy gradient loss and exploration encouragement. The authors provide open-source code and data, highlighting the potential for more data-efficient RLVR approaches in improving LLM capabilities.

4

This paper investigates how reinforcement learning (RL) enhances large language models’ (LLMs) ability to recall and navigate structured knowledge hierarchies (such as medical‐codes). The authors show that RL‐enhanced models outperform both base and supervised fine‐tuned (SFT) versions on hierarchical recall tasks. Crucially, they argue the improvement arises not from new factual memorization but from better procedural skill in knowledge traversal: SFT models, when given structured prompting to guide hierarchical navigation, recover much of the gap (from ~24pp to ~7pp). They further analyze internal model activations to reveal that RL modifies how the model queries knowledge rather than what it knows: factual‐representation activations stay similar, but query‐representations diverge. https://lnkd.in/ggNa4nqD

1

New paper on improving LLMs' native capabilities of retrieval/ranking, to be presented in NeurIPS25: https://lnkd.in/eSX8vKPm Can we give LLM a list of items in context and ask it to retrieve/rank relevant ones? How does LLM solve such a problem? Is attention doing something similar to embedding-based retrieval? How to make it efficient with long context? Can we leverage embedding learning techniques to improve quality?

301

5 Comments

This is the fourth post in my series on PTC 2026. The last post outlined the architecture contrast between centralized training campuses and distributed inference networks. This post covers the toolchain changes that make distributed inference practical. Three themes came up repeatedly. First, disaggregated serving. Multiple teams described separating prefill and decode across different GPU pools. These phases have different resource profiles, so treating them as one monolithic job leaves performance on the table. The NVIDIA Dynamo came framework is making this operational. Second, KV cache mobility. This was the topic I found most interesting. Engineers described work to externalize context state so it can move between nodes or sites. The goal is failover without full recomputation. Projects like LMCache and Mooncake act as the glue layers enabling this across inference engines like vLLM and SGLang. Third, cache aware routing. Instead of naive load balancing, inference requests route to nodes that already hold relevant context. It’s the difference between treating inference as stateless compute versus a service with session affinity. The consistency across conversations was striking. The software to treat inference as a routed, recoverable service is maturing fast. That changes who shows up to build. In the next post, I will talk about the ecosystem assembling around distributed inference and the conversations that made it feel real at PTC.

8