ngrok

"How hard can that be?" That's what Sam Rose asked about the replica set controller. Its whole job: if you ask for four of a pod, keep four of that pod running. The answer turned out to be several thousand lines of code. Sam ported the replica set and deployment controllers to webernetes, his Kubernetes cluster that runs entirely in the browser. In part 4 of the series, he walks through the code and shows: - why maintaining "four of this pod" takes several thousand lines of code - why the deployment controller isn't "just controlling replica sets under the hood" but "its own beast," with all the rollout logic - what the status field is actually for: "a place for the controller to communicate with you, the user, and with itself later on" - how a TypeScript class stands in for a container, with an exec function, exit codes, and its own in-memory file system - how the simulated cluster network handles DNS, services, and load balancing Watch part 4 → https://lnkd.in/gF8GdZsG

The main reason people leave ngrok endpoints unprotected is cost. Good news: Traffic Policy is now 10x cheaper. 100k TPUs used to cost you a $1, and now that's down to $0.10. Here's where you'll actually feel the biggest difference: → JWT validation and IP restrictions run 10 TPUs per request. A million requests used to cost $100 in TPUs; now it's $10, before volume discounts. → The OWASP WAF rules for blocking scrapers and bots: same 10x drop. → Rate limiting is 1 TPU per request, so a million rate-limited requests now cost $1. → OAuth, OIDC, and basic auth didn't change. They've always been free. Also in this round: custom domains now bill only for hours they actually receive traffic ($0.01 per active hour, idle domains are free), and Endpoint Pooling now bills per active endpoint. All of these new rates apply at the start of your next billing cycle. Full details from Sam Crowell Richard ↓ https://lnkd.in/g3Y-yd4A

Sam Rose's next interactive essay is about Kubernetes probes. Even starting it took… 90,000 lines of TypeScript. The research is a full rebuild of Kubernetes that runs entirely in the browser, a project now called webernetes. In episode 3 of the build log, it gives its first proper demo: a working cluster in a browser tab. Where things stand: → 90,000 lines of TypeScript, up from 40,000 one episode ago → The whole cluster ships as about 300KB of JavaScript, less than most cookie consent banners → Kubelet hell: escaped → Deployment controllers: due in 3 days, when Sam's summer holiday is supposed to start The episode walks through the demo and the trade-offs that get a cluster down to 300KB. Watch over on YouTube: https://lnkd.in/gGveWuTf

ngrok is absolutely superb service no-one is talking about

There's a gap nobody talks about in AI gateways. Every one of them routes to the hosted providers, OpenAI, Anthropic, and the rest, but none can reach a model you run yourself. Local models, fine-tuned models, private models. None of them are reachable through a normal gateway. For example, last month we fine-tuned our own model on our domain data. It beat the hosted model we were paying for. But we couldn't ship it. It lived on our own hardware, and the gateway had no way to reach it. That's not the gateway's fault. It was built for hosted providers. For a gateway, "every model" has always meant the ones someone else hosts. Not the one running on your own machine. So we tried ngrok. ngrok AI is the one gateway that reaches every model, including the local, fine-tuned, and private ones, because it routes through the ngrok agent. That agent is the same piece that has always tunneled into localhost and private networks, so it reaches a model on your own box the way it reaches anything else you run. We pointed our OpenAI SDK at it and changed a single line, the base URL. Now the same gateway reaches all of it: > Our fine-tuned model on our own GPU > A local Ollama model on localhost > The hosted models, still in one failover chain To be clear, ngrok AI isn't hosting the model for us. We still run and maintain our own inference server. It only makes that server reachable through the same gateway as everything else. The failover surprised me. When our box went down mid-deploy, that same call fell back to a hosted model on its own. When a gateway promises every model, that should mean every model. Whether it's an Ollama build on localhost or a fine-tuned behemoth on a neocloud. You just read how this works. Now go try it already. Link in the comments. *partnerpost

ngrok's AI Gateway can now route to a model running on your own hardware, privately, without opening a port or putting it on the public internet. To your app, that self-hosted model is just another entry in your fallback list, right next to OpenAI and Anthropic. Fine-tuned, open-weight, or just cheaper on hardware you already own, and the data those models see never leaves your network. That's the part no other gateway does. The rest is what you'd expect: one hosted gateway in front of every model your apps call, your own and the public providers alike, behind a single URL. We've spent a decade routing traffic into private networks, so pointing a gateway at a model on your own hardware is the natural next step, and the piece only ngrok can ship. From that same baseUrl (https://⁠gateway⁠.⁠ngrok⁠.⁠ai) you also get: → routing and failover that reroutes a slow or failing model to a healthy one, with automatic retries, before users notice → scoped access keys, so each app only reaches the providers and models you allow → observability that breaks tokens, latency, errors, and spend down by app, dev, and model, instead of one lump sum on a provider invoice → bring-your-own-keys, so you keep your negotiated rates and pay providers directly It works with the SDKs you already use (OpenAI, Anthropic, Vercel AI SDK) and it's fully programmable through our API, so you can drive it from Terraform, a CLI, or your coding agent. One flat fee: $0.05 per million tokens, inference billed on top. Buy credits, no subscription. Start building → https://app.ngrok.ai Full-story blog post and docs in the comments ↓

"ngrok? That's the tool my engineers use to demo localhost." Accurate. Also about 10% of what it's doing in production at any given moment. → Teams reach internal services tucked inside a customer's network. No VPN, no hardware appliance, no quarterly "can someone please open this port" ticket. → Platform teams put one front door in front of every ingress and handle auth, rate limiting, and observability with Traffic Policy at the edge, instead of wiring it up service by service. → One gateway handles devices, APIs, and LLM endpoints, with the same controls on each. The localhost tunnel was always the wedge. What's underneath is developer infrastructure built for production. Worth six minutes if you've only ever met the tunnel ↓

Our very own Sam Rose ported Kubernetes to the browser. Like a real flippin' cluster with pod lifecycles, cluster DNS, the scheduler, and a simulated pod network. It's ~100,000 lines of TypeScript, almost all written by LLMs, but with every line reviewed by hand. So here’s how he sure it isn’t slop: He ported a working slice of Kubernetes to the browser (as in real pod lifecycles, cluster networking, the scheduler, the works), then proved it behaves like the real thing by running one test suite against both a live k3s cluster and his port. If a test passes on k3s and fails on the port, that’s a bug to go fix. 204 of them pass today. There’s a playable cluster right in the post, plus an honest accounting of where LLMs helped and where they fell flat on their face (porting code, it turns out, is not their strong suit). Blog post → https://lnkd.in/gMiavF4w The fully-fledged demo → https://lnkd.in/gJPUJVp8

Sam Rose is a wild one for this. We want to teach the invisible layers of modern software in ways grounded in reality, not pure simulations or throwaway demos. Things that'll last for a long, long time. Sometimes that means 100k lines of TypeScript just to earn the right to start telling the story at all. That ambition led Sam to build a browser-based Kubernetes simulator from scratch, just so future content could be built on something real. He just open sourced it, and worked with Ryan Blunden on a video about what that build actually felt like. Both are worth your time. Repo + demo: https://lnkd.in/gV-9VCdq Behind the build: https://lnkd.in/gcSXnRBi