Dustin Vannoy

Databricks cut our nightly pipeline from 9 hours to 47 minutes. But the real win wasn’t speed—it was trust. We inherited a zoo of ad‑hoc Spark jobs and brittle Airflow DAGs. Moving to the Lakehouse, we rebuilt with Delta Live Tables and Unity Catalog so every dataset had lineage, ownership, and quality gates. What worked: - Design bronze → silver → gold, and treat gold tables like products with SLAs. - Centralize permissions and tags in Unity Catalog; lock PII by default. - Put expectations in DLT close to the data, and alert on failures—not just on job success. Bonus: small, reusable notebook functions beat giant “do‑everything” pipelines. If you’re starting or scaling on Databricks, which guardrails did you put in first, and what do you wish you’d done earlier? #Databricks #DeltaLake #Lakehouse

4

Curious how our consultants are actually using AI? They're testing it, breaking it and figuring out what it can actually do. This post is a great example!

8

The gap in Databricks is tribal knowledge.    Not the platform. The context around it.    We've spent years helping clients wrangle  → Data pipelines  → ML workflows,  → Agentic systems    Same problem every time.    Great platform, too much institutional  knowledge needed to move fast.    That changed when I started working with  the Databricks AI Dev Kit.    It gives AI coding agents actual  Databricks context.    What it gives your agent:    50+ live tools.  19 skill files packed with platform patterns.  An MCP server that lets your agent act,  not just suggest.    Claude Code, Cursor, whatever you use.    Pair that with Genie Code and the loop  from idea to working pipeline  tightens dramatically.    At FrameData AI, this is exactly  what lets us show up to client  conversations with something built.    Not a deck.  Not a POC roadmap.  Something real.    No context loss. Faster value.  That's the job.    If you're building data and AI systems  on Databricks, what's your agent stack? 

13

1 Comment

Databricks CEO Just Dropped The Most Honest Advice About The Future Of AI Data company leaders aren’t interested in histrionics and hand-waving Joe Procopio Joe Procopio Following 5 min read · Feb 23, 2026 1,926 57 Press enter or click to view image in full size Here’s a little secret for you. The next wave of AI success is going to be completely dependent on structured data. Maybe that’s a no-brainer to you. Maybe I’m telling you that the next time you take a drink of water, that water will be wet. But I believe that prediction, and the fact that it’s kinda clear already, is the reason why you get a certain sense of calm when the CEOs of these massive data companies talk about the future of AI. They’re confident, but thoughtful. Open-minded, but not susceptible to hyperbole, histrionics, and hand-waving. They don’t say things like all software developers will be replaced by AI (the deadline for which has since passed) or all tech employees should become baristas and bartenders (the dream is over, nerds) or don’t worry, we’ve hammered out all the unintended consequences for letting users create their own AI porn (what could go wrong?). Instead, the data CEOs say common sense things like ““Why would you move your system of record? You know, it’s hard to move it.’” Holy crap. I think I’m in love. SaaS Goes Into the Woodchipper Become a member to read this story, and all of Medium. Joe Procopio put this story behind our paywall, so it’s only available to read with a paid Medium membership, which comes with a host of benefits: Access all member-only stories on Medium Read everything on Medium, including member-only stories Support the writers you read most Help build an ad-free, independent reading and writing platform https://lnkd.in/gy_9dFK2

1

Snowflake just made a move that changes everything about its strategy. And most people missed it. Cortex Code CLI now works with dbt and Airflow, tools that live OUTSIDE the Snowflake ecosystem. This is bigger than it sounds. Here's why: For years, the pattern has been: → Cloud vendors build AI tools that only work within their walls → You get amazing features... but only if you go all-in → The second you need dbt or Airflow, you're on your own Snowflake just broke that pattern. THE OLD WALLED GARDEN: "Here's an amazing AI coding assistant!" *only works in Snowflake worksheets* Dev: "But my dbt models live in VS Code..." Vendor: "🤷" Result: AI stays in the demo, real work stays manual THE NEW OPEN APPROACH: Cortex Code CLI works where you actually work: → dbt projects in your local repo → Airflow DAGs in your pipeline code → Streamlit apps in your development environment → Git workflows, bash commands, everything And here's the kicker: 🔥 FIRST STANDALONE SUBSCRIPTION MODEL You don't need Snowflake compute to use it. You don't need a Snowflake deployment. You just... use it. 30-day free trial → paid subscription. That's it. What this unlocks: ✅ Teams using BigQuery + dbt can use Cortex Code ✅ Data engineers on Databricks + Airflow benefit ✅ AI assistance follows you across your entire stack ✅ No vendor lock-in to access the AI layer The shift happening: 2020: "AI tools are product features" 2023: "AI assistants are walled-garden add-ons" 2026: "AI agents are standalone infrastructure that work anywhere" We analyzed: → Complete feature breakdown → How the dbt/Airflow integration actually works → Pricing model vs alternatives → Strategic implications for the data stack 📖 Full breakdown on Substack - https://lnkd.in/gsCiZsrY Real talk: Is this the future of data tooling? AI agents that work across vendors? Nick Akincilar Anand Jha Aria Attar Jennifer Daniell Belissent, PhD? #Snowflake #DataEngineering #OpenSource #dbt

6

If you are working with real world use cases, you know that different LLMs can be useful on different projects. What matters is the ability to quickly test which works best for your usecase. Well, there is an abundance of choice on Databricks. You don't need to spend time building an API integration. Top of the like models are available by default.

13

If you've added indexes, partitioned tables, and tuned autovacuum, but your dashboards still go red, you might not be doing anything wrong. We see this all the time. Here are 6 signs that Postgres tuning won't fix your performance problems, and if 4 or more apply to you, the full write-up (shared in the comments) shows you what to do about it

37

3 Comments

As more data teams move off Redshift to control costs, simplify performance, and support AI workloads, Databricks’ new guide lays out a clear, phased approach to executing a migration to lakehouse architecture: • Inventory and prioritize workloads to reduce migration risk • Translate schemas thoughtfully to avoid long-term performance issues • Leverage tools like Datafold’s Migration Agent to handle end-to-end migration—from scoping to validation If your team is planning—or even just considering—a Redshift migration, this field guide is a great place to start. https://lnkd.in/gdak_duZ

10

Multimodal data can be difficult to analyze together. At #SnowflakeBUILD, Senior Developer Advocate James Cha-Earley will show you how Snowflake Cortex AISQL solves this by letting you analyze text, audio, images, and documents, all with SQL. In this hands-on lab, you’ll learn to build a production-grade pipeline to power AI use cases from customer service data: - Multimodal Processing: Use Cortex AISQL functions for transcription, document parsing, classification, and entity extraction to process voice calls, PDFs, chat logs, and support tickets. - Intelligent Workflows: Create pipelines that surface actionable intelligence for both customers and support agents. Register today: https://lnkd.in/gaB-wgcv

43

1 Comment

I built ADVNT's replication cost model the fast way first: stack deltas on a baseline. Add a sprint, add the delta. Clean, incremental, efficient. The problem: the base was frozen at an early system state. Every delta was mathematically correct. The foundation it sat on hadn't been recomputed in over a year. By the time we ran it from scratch, the stacked-delta floor was 52% below what a bottom-up count of the live system actually produced. So we rebuilt the model the same way we rebuild everything here — derive it from what the system is, not from what it used to be. The current recompute: 498 development-months to replicate. At $250K per developer-year, that's $10.4M raw. Realistic figure — accounting for production validation, institutional knowledge transfer, and calendar sequencing — is $15.6M. The floor is $6.5M: 65% of raw, derived by rule and applied to a live count. Not an estimate. A formula applied to ground truth. Wall clock: 18 months minimum. Production validation is calendar-bound, not headcount-bound. 24+ months is the realistic timeline regardless of team size. The deeper issue wasn't the numbers. It was the method. Accretion on a frozen base always drifts. The only fix is a rule that forces recomputation from source — and a system that can produce source truth on demand. How many valuation models in AI companies right now are still running on a baseline that hasn't been touched since the system was half this size? #SystemsThinking #PlatformEngineering #BuildInPublic #advnt

2

1 Comment

Where does the semantic layer belong? With Fivetran and dbt merging today, we are back to three classic (post-modern?) pillars of the data stack: ETL, CDW, and BI. Every one of them now offers some semantic modeling functionality: e.g., semantic views in Snowflake, Metricflow in dbt, and Cube SL in BI. We have so many semantic layers that we have even started building semantic interchange infrastructure - OSI. But why do you need a semantic layer in the first place? A semantic layer enables data consumption. In BI, you need it for self-serve analytics so users can safely point-and-click in the UI and build the analysis they want. In embedded analytics, you need it to avoid SQL spaghetti code generation and support multi-tenant data access (that is where Cube started!). Finally, with AI now, you need a semantic layer to provide context about your data to AI (Snowflake built semantic views for Cortex Analyst). It is always about data consumption. With AI taking over or augmenting the data consumption layer in the next several years, we will see every vendor in that space introducing their own semantic layers. With that proliferation of semantic layers, where should we build and keep one? The main data consumption layer seems to be the perfect place for that - you want tight integration with the UI and the features of this layer. The other consideration is developer experience - code-first, version control, integration with CI/CD. And finally, the interoperability with the rest of the stack. You're never going to have a single data consumption layer, so you need to make sure your primary one with the semantic layer integrates well with the rest. This way, semantic models built there can be reused across other data consumption tools.

56

9 Comments

jOOQ 3.21.0 with Databricks, Spanner support, cloud dialect upgrades, redacted columns, meta dependencies, many UDT and DDL improvements, and much more https://lnkd.in/eBpTv4p6

30

What if prompts and agents could target named, governed datasets and not all your files? How would results change? @Dataset Targeting lets teams scope RAG to reusable, access-controlled datasets you can reference in prompts or agents with simple @ notation (e.g., @Policies_2025). Result:  less manual file-picking, fewer off-topic hits, and audit-ready evidence trails by default. This is an add-on tool that overlays your existing stack (no rip-and-replace). In our survey, developers and customers reported cleaner answers and lower inference spend when they bind agents to curated datasets. It also supports fine-grained RBAC and multi-agent workflows across projects for governed AI and compliance needs. We also published a buyer’s guide comparing @Dataset targeting across 15 AI platforms plus a review checklist for this capability. https://lnkd.in/gR7yGriM Curious how you’re handling dataset scoping today? What’s worked, what hasn’t? Drop your approach below or DM for the checklist. #GovernedAI #RegTech #compliance #GRC #RiskManagement #OpenSourceAI  #Audit #GenerativeAI #inteligemslabs

34

2 Comments

Databricks supports Apache Iceberg v3 as a storage format now. This is an indirect result of the Tabular acquisition in 2024. Tabular was founded by the original creators of Apache Iceberg (Netflix alumni Ryan Blue, Daniel Weeks and Jason Reid). Their platform enhances interoperability between open-source table formats like Iceberg and Delta Lake. This helps enterprises unify data storage and querying across diverse systems. Some people wonder if Databricks is now "moving" to Iceberg. We believe not. Databricks is all about "democratizing data". One prerequisite of that is that they must be able to run whatever storage format you bring, all while opening the door to a broader ecosystem of engines and tooling (think of Trino for example). We will post the official announcement in the comments!

13

1 Comment

You can now use Immuta to manage access to Databricks #Lakebase tables, views, materialized views, and partitioned tables. Access controls defined in Immuta through subscription #policies are automatically enforced in Databricks Lakebase. This integration enables users to provision and manage access to Databricks Lakebase data dynamically and at scale, reducing manual overhead for administrative teams and ensuring consistent access control across Databricks Lakebase and other platforms that Immuta supports. https://lnkd.in/gPVx536e

30

1 Comment