It is the ‘Week of Agents’ at Databricks, and this has inspired me to create a blog around leveraging agents to help dissect and analyze complex energy markets. Like I mentioned in my previous blog post, I’ve been intrigued by energy markets as of late and I am trying to identify ways to keep up to date with what is happening within North America and across the world when it comes to energy markets. The more I dig in, the more I appreciate just how fragmented the information landscape is.

An energy market analyst on any given Wednesday morning is watching the EIA petroleum status report drop, tracking overnight geopolitical developments out of the Middle East, monitoring ERCOT load forecasts ahead of a Texas heat wave, and trying to synthesize all of it into a coherent view across commodities. These signals don’t live in the same place, don’t update on the same cadence, and require genuine domain knowledge to interpret together.

I also wanted to look at this as an opportunity to get more hands-on experience with novel cutting-edge technology like agents. Instead of a trader switching between five tabs and three data providers, what if a single conversational interface could pull from all those sources as they sit in your data lake like live inventory data from the EIA, latest available public data on ERCOT curtailment risk, breaking news with AI-classified sentiment and synthesize a coherent answer on demand?

This post walks through how it was built using LangGraph, Databricks Genie, and Databricks Model Serving.

What Has Been Built

The research agent routes questions to specialised sub-agents, each backed by a Genie space with domain-specific data, and synthesises the results into a single answer.

For a single-domain question (“What’s the WTI price?”), the agent calls one tool. For a cross-domain question (“Give me a full market brief”), it calls all three and synthesises. The routing decision is made by the LLM and it reads the tool descriptions and decides what’s needed.

The Data Foundation

Before there’s an agent, there’s a lakehouse. This project runs 7 Lakeflow Spark Declarative Pipelines on Databricks serverless, ingesting from public energy APIs into Unity Catalog:

All pipelines are deployed and scheduled via a Databricks Asset Bundle, with two scheduled jobs running on weekday evenings and Wednesday mornings to keep the data fresh. On top of this lakehouse sit three Genie spaces, each a natural language SQL interface scoped to a specific domain:

• Market Data: prices, EIA inventory/production, ERCOT hub prices and spark spreads
• ERCOT 4CP Analysis: historical peak demand, 2024–2025 curtailment risk model, weather data
• News & Sentiment: rolling ~4-week news window with AI event classification and sentiment scores

Genie handles the SQL generation and execution. The agent’s job is to know which Genie to ask.

Building the Agent

Step 1: Wrap Each Genie as a LangChain Tool

Each tool is a thin wrapper around the Databricks SDK’s Genie Conversation API. We start a conversation, poll until it completes, and extract the text response and SQL result rows.

from databricks.sdk import WorkspaceClient
from langchain_core.tools import tool

GENIE_SPACES = {
    "market_data":    "01f120a563181069924649e4eadfb75e",
    "ercot_4cp":      "01f1227bc4c2113bbb95bcccd6dd634c",
    "news_sentiment": "01f12d623a761d688dd9ab160d273a94",
}

def _query_genie(space_id: str, question: str, timeout: int = 120) -> str:
    w = WorkspaceClient()
    resp = w.genie.start_conversation(space_id=space_id, content=question)

    deadline = time.time() + timeout
    while time.time() < deadline:
        msg = w.genie.get_message(
            space_id=space_id,
            conversation_id=resp.conversation_id,
            message_id=resp.message_id,
        )
        status = msg.status.value if hasattr(msg.status, "value") else str(msg.status)

        if status == "COMPLETED":
            # Extract natural language response + formatted SQL result rows
            parts = []
            for att in msg.attachments or []:
                if hasattr(att, "text") and att.text:
                    parts.append(att.text.content)
                if hasattr(att, "query") and att.query:
                    cols = [c.name for c in att.query.result.statement_response.manifest.schema.columns]
                    rows = att.query.result.statement_response.result.data_array or []
                    parts.append("\n".join([" | ".join(cols)] + [" | ".join(str(v) for v in r) for r in rows[:25]]))
            return "\n\n".join(parts) or "No data returned."

        elif status in ("FAILED", "QUERY_RESULT_EXPIRED"):
            return f"Genie query failed: {status}"

        time.sleep(2)

    return "Genie query timed out."


@tool
def query_market_data(question: str) -> str:
    """Query live commodity prices and EIA supply/demand fundamentals.
    Covers: WTI crude, Brent, Henry Hub, RBOB, heating oil prices;
    US petroleum inventory; natural gas storage; EIA drilling productivity;
    ERCOT hub prices, load actuals, and spark spreads."""
    return _query_genie(GENIE_SPACES["market_data"], question)


@tool
def query_ercot_4cp(question: str) -> str:
    """Query ERCOT 4CP curtailment risk and peak demand data.
    Covers: historical Four Coincident Peaks 1998-2020; 2024-2025 curtailment risk
    model with enhanced_risk_score and curtailment_alert flags; weather (HDD/CDD)
    by ERCOT load zone; projected load thresholds vs actuals."""
    return _query_genie(GENIE_SPACES["ercot_4cp"], question)


@tool
def query_news_sentiment(question: str) -> str:
    """Query recent energy market news with AI sentiment analysis.
    Covers: rolling ~4-week news window; AI-classified event types (supply disruption,
    geopolitical event, weather event, regulatory change, etc.); sentiment labels
    (positive, negative, neutral, mixed); commodity and country tags."""
    return _query_genie(GENIE_SPACES["news_sentiment"], question)

The tool docstrings are really important here, this is how the LLM decides which tool to call. The more precise the description, the better the routing.

Step 2: Wire It Together with LangGraph

With the tools defined, we can wire them together in LangGraph:

from langchain.agents import create_agent
from databricks_langchain import ChatDatabricks

llm = ChatDatabricks(
    endpoint="databricks-meta-llama-3-3-70b-instruct",
    temperature=0,
    max_tokens=4096,
)

agent = create_agent(llm, tools=[query_market_data, query_ercot_4cp, query_news_sentiment],
                     system_prompt=SYSTEM_PROMPT)

create_agent is the LangChain v1 API for building agents, replacing the deprecated langgraph.prebuilt.create_react_agent. It builds a ReAct graph where the LLM reasons about which tool to call, calls it, observes the result, and repeats until it has enough information to answer. ChatDatabricks is provided by the databricks-langchain package and handles authentication and routing to any Databricks Model Serving endpoint.

The system prompt defines the agent’s identity and reinforces the routing logic:

You are an expert energy market research assistant...

Tool routing:
- query_market_data    → prices, EIA supply/demand, ERCOT power prices/load/spark spreads
- query_ercot_4cp      → 4CP peaks, curtailment risk scores, weather, load forecasting
- query_news_sentiment → recent headlines, event classification, sentiment trends

For multi-domain questions, call multiple tools and synthesise the results into a single answer.

Step 3: Deploy to Databricks Model Serving

We wrap the agent in an MLflow PythonModel so it can be deployed as a REST endpoint and queried from the playground, notebooks, or any HTTP client.

class EnergyResearchAgent(mlflow.pyfunc.PythonModel):

    def load_context(self, context):
        _configure_credentials()  # inject DATABRICKS_HOST from env
        self.agent = _build_graph()

    def predict(self, context, model_input, params=None):
        if hasattr(model_input, "to_dict"):
            model_input = model_input.to_dict(orient="records")[0]

        messages = [HumanMessage(content=m["content"])
                    for m in model_input.get("messages", [])
                    if m["role"] == "user"]

        result = self.agent.invoke({"messages": messages})

        return {
            "choices": [{"message": {"role": "assistant",
                                     "content": result["messages"][-1].content},
                         "finish_reason": "stop", "index": 0}],
            "object": "chat.completion",
        }

Logging and registering to Unity Catalog:

mlflow.set_registry_uri("databricks-uc")

with mlflow.start_run():
    mlflow.pyfunc.log_model(
        name="energy_research_agent",
        python_model=EnergyResearchAgent(),
        code_paths=["agents/energy_research_agent/agent.py"],
        pip_requirements=["langgraph>=1.0", "langchain>=1.0",
                          "databricks-langchain", "databricks-sdk>=0.40"],
        registered_model_name="energy_research_agent",
    )

One important detail: the serving container has no default Databricks credentials. The fix is to inject them via secrets when creating the endpoint:

ServedEntityInput(
    entity_name="energy_research_agent",
    entity_version="3",
    workload_size="Small",
    scale_to_zero_enabled=True,
    environment_vars={
        "DATABRICKS_HOST": "https://<your-workspace>.cloud.databricks.com",
        "DATABRICKS_TOKEN": "{{secrets/<your-scope>/databricks_pat}}",
    },
)

Seeing Inside: MLflow Tracing

Deploying the agent is one thing. Understanding what it’s actually doing is another. With mlflow.langchain.autolog, every run produces a full execution trace. Tool calls, LLM steps, inputs, outputs, and latency, all viewable in the Experiments UI.

mlflow.langchain.autolog(log_traces=True)

with mlflow.start_run(run_name="full market brief"):
    result = agent.invoke({"messages": [HumanMessage(content=
        "Give me a full energy market brief: WTI price, ERCOT curtailment risk, "
        "and any recent news that could move prices."
    )]})

Opening the trace for the cross-domain question, you can see the full ReAct loop:

1. LLM thinks: “This question spans prices, ERCOT risk, and news. I need all three tools.”
2. Calls query_market_data: asks Genie for latest WTI and nat gas prices; gets back SQL results
3. Calls query_ercot_4cp: asks Genie for the current risk score and latest curtailment alert
4. Calls query_news_sentiment: asks Genie for recent articles and aggregate sentiment on crude
5. LLM synthesises: writes a coherent market brief grounded in the actual data values

Each step shows the exact question sent to Genie, the SQL it generated, the raw result rows, and how long each step took. For a question that hits all three Genie spaces, total latency is typically 30–60 seconds which is dominated by Genie query time, not LLM reasoning.

Tuning Your Genie Sub-Agents: From Instructions to Certified SQL

I had to do a ton of fine-tuning on the Genie spaces to get the correct responses back.

There are three tiers of guidance you can give a Genie space, in increasing order of reliability:

The News Genie Problem: Wrong Join

Out of the box, when asked “What geopolitical events are affecting energy markets?”, the news Genie returned this:

“The only geopolitical event type affecting energy markets in the data is geopolitical_supply_risk, which is referenced in 50 articles.”

Technically correct. Completely useless. Genie took the path of least resistance and it counted rows in silver_news_sentiment and called it done. The actual article titles, descriptions, and publication dates sitting in bronze_news_articles were never touched.

Instructions fixed this. Adding explicit guidance to always JOIN the two tables and never return counts when the user wants headlines produced the response you’d actually want:

“Recent geopolitical events affecting energy markets include ongoing conflicts in the Middle East, particularly the U.S.-Israeli war with Iran, which has led to supply disruptions and volatility in oil prices. Notable data points include:

• Strait of Hormuz closure impacting global oil and gas transit
• Middle East volatility causing oil price swings
• EU postponing Russian oil ban due to Iran war
• Big Oil seeking new resources as infrastructure is damaged”

The key instructions for the news Genie space:

Always JOIN bronze_news_articles with silver_news_sentiment on
bronze_news_articles.article_id = silver_news_sentiment.article_id when answering
questions about news events, headlines, or market developments.

Never return only counts or category summaries — show actual article titles
and descriptions. ORDER BY published_at DESC.

For geopolitical questions, filter WHERE event_type IN
('geopolitical_event', 'geopolitical_supply_risk') and return the full
article titles and descriptions joined from bronze_news_articles.

The Resulting Agent Response

With both fixes in place, a cross-domain question that previously returned stale data and category counts now produces:

Question: “What are the most recent energy market headlines and is overall sentiment bullish or bearish on crude oil, as well as the current prices?”

Agent response:

Based on the recent energy market headlines, the overall sentiment on crude oil is leaning towards neutral and positive, with 5 neutral and 3 positive headlines out of 8 total headlines. The current prices of crude oil are $111.54 for WTI and $109.03 for Brent, as of April 2, 2026. These prices reflect the market’s response to the geopolitical events and other factors mentioned in the headlines, such as the record highs in Brent crude oil price and the halt in crude oil production by Kuwait.

Sentiment from news articles should probably be treated as directional, not predictive.

This is the agent firing both tools, query_market_data for current prices (now correctly hitting silver_commodity_ohlcv via the certified SQL example) and query_news_sentiment for headlines and scores. Before the Genie tuning, this same question returned a stale March price and a single line counting news categories. After tuning, it returns a grounded, multi-signal market view.

Here is what the MlFlow trace looks like as well:

Genie Tuning Principles

1. Start with instructions for JOIN and output format issues. They’re easy to write and usually fix Genie picking the wrong path when the correct answer requires combining two tables.
2. Use certified SQL examples for table selection issues. When Genie consistently chooses the wrong table despite instructions, example_question_sqls is the reliable fix. It bypasses Genie’s SQL generation entirely for known question patterns.
3. Clarify column semantics. Series codes like WCRSTUS1 are opaque. Spelling out what they mean (crude stocks, production, imports) helps Genie pick the right series.
4. Write instructions that address known failure modes directly. “Never return counts when the user asks about specific events” is more actionable than “be detailed”.

This investment in Genie quality pays compounding dividends. Every query that flows through the agent benefits, whether it comes from a user in the Genie UI, from the LangGraph agent, or from a scheduled notebook. Time spent tuning the sub-agents is time saved debugging the orchestration layer.

What Makes This Pattern Powerful

A few things stand out about this approach compared to alternatives:

Genie as a sub-agent is genuinely useful. Genie handles all the SQL complexity, schema understanding, joins, aggregations, formatting. The LangGraph agent just needs to ask it a question in plain English and get back a structured result. You get the flexibility of natural language querying without exposing raw database access to the agent.

Scoped Genie spaces dramatically improve accuracy. I have found a single Genie covering all tables writes worse SQL on domain-specific questions because the schema is too broad. Three tightly-scoped Genie spaces write better SQL on their respective domains, and the agent router compensates for the added complexity.

Tool descriptions drive routing quality. The LLM’s routing decisions are entirely based on the tool docstrings. Writing precise, example-rich descriptions, what the tool covers, what it doesn’t, when to use it, is the highest-leverage thing you can do to improve agent behaviour. No fine-tuning required.

MLflow Tracing makes debugging tractable. Without tracing, a wrong or incomplete answer is hard to diagnose. Was it the routing? The Genie SQL? The LLM synthesis? With traces, you can see exactly where things went wrong.

What’s Next

This was a lot of fun to build and I think as I continue bringing this data into the lakehouse there will continue to be interesting analytical and AI opportunities.

A few natural extensions from here:

• Databricks App: wrap the endpoint in a Streamlit chat UI so the agent is accessible to non-technical users without an API client
• Memory: use Lakebase to persist conversation history and user preferences across sessions, enabling multi-session research threads
• Evaluation: use MLflow’s genai.evaluate() to score routing accuracy and answer quality against a golden set of questions

Thanks for reading 😀!

Note: All prices and spark spreads shown here are derived from ERCOT HB_NORTH real-time settlement point prices (15-minute intervals), Henry Hub front-month proxy (NG=F continuous futures), and publicly available weather data, all ingested and modeled in Databricks for this project. Spark spreads are calculated using assumed heat rates (7.0 MMBtu/MWh for CCGT, 10.0 for peakers) and represent fuel-cost margins only, not total plant profitability. Values are based on my processed dataset and may differ from official ERCOT settlements, nodal prices, or plant-specific economics. HB_NORTH broadly represents the Dallas/Fort Worth region.

Over the past few months, I’ve been intrigued by energy markets, especially since I’ve been working more with the trading team at Plains on use cases like crude blending. To get a better feel for the data energy traders analyze daily, I pulled in data from ERCOT, the U.S. Energy Information Administration, and other sources into Databricks, focusing on the Texas market. This also gave me a chance to get hands-on with Lakeflow Spark Declarative Pipelines. While I wasn’t the biggest fan of its predecessor, DLT, I’ve heard along with the new branding there are significant improvements to the product as well.

The idea I had to start was to look at these various data sources to analyze how extreme weather events impact gas and power prices.

On January 26, 2026, a cold front pushed into Dallas and temperatures plummeted more than 30 degrees below the seasonal average. Over the next three days, something remarkable happened to energy markets:

Gas prices jumped sharply. Power prices more than quintupled. And then, as temperatures recovered on January 29, the spread turned negative and while January 30 showed some residual elevation as the grid normalized, the event was effectively over within two days.

The spark spread captures the core economics in a single number.

The two charts above tell the same story from different angles. The top chart tracks two lines: average power price (what electricity sold for at the HB_NORTH hub) and the CCGT spark spread (the profit margin for a gas plant generating that electricity). On January 21 both lines are flat and low, a mild day with no stress on the grid. Then the cold front arrives on January 26 and both lines spike sharply upward, the power price climbing to $168/MWh and the spread jumping to $120. They stay elevated through January 28 before collapsing back to near zero on January 29 the moment temperatures recovered.

The bottom chart shows why. Dallas average temperature is plotted across the same date range. The V-shaped dip, bottoming out at an average of 21°F on January 26, is the cold snap. The visual alignment between the temperature dip and the price spike in the top chart is the entire point: weather drove demand, demand drove prices, and prices drove the spread. The mechanism isn’t abstract, you can see it happen in real time across two charts stacked on top of each other.

The fact that both lines in the top chart move together matters too. In a summer heat wave, power prices spike but gas prices stay cheap, so the spread explodes far above the power price line. In this winter event, gas prices spiked alongside power prices. Gas infrastructure tightened as heating demand and power generation competed for the same fuel. The spread was still strongly positive, but it was more compressed than a typical August afternoon. That compression is the signature of a winter stress event versus a summer one.

What Is The Spark Spread?

The spark spread is a common metric for estimating the profitability of natural gas-fired electric generators. The spark spread is the profit margin a gas-fired power plant earns for generating one megawatt-hour of electricity. It answers one question: is it worth burning gas right now? They are calculated with the following equation:

Spark spread ($/MWh) = power price ($/MWh) – [natural gas price ($/mmBtu) × heat rate (mmBtu/MWh)]

Heat rate measures how many MMBtu of fuel a power plant must burn to generate 1 MWh of electricity. Lower heat rates mean higher efficiency.

• CCGT (combined cycle gas turbine, ~49% efficient): 7.0 MMBtu/MWh: the workhorses that run most of the day.
• Peaker (simple cycle, ~34% efficient): 10.0 MMBtu/MWh: expensive to run, called only when the grid is stressed.

A positive spread means the plant covers its fuel cost and makes money generating electricity. A negative spread means every megawatt-hour it produces loses money on fuel alone.

On January 28, with power at $200/MWh and gas at $7.46/MMBtu, the CCGT spread was $200 − (7.0 × $7.46) = $147.67. Many gas plants were earning extremely high margins. On January 21, with power at just $22.44 and a spread of -$11.69, few plants would choose to run purely for energy revenue.

This is the same plant, same fuel, a week apart.

Where Spark Spread Falls Short

The spark spread is widely used precisely because it is simple, but that simplicity comes with real limitations worth understanding before drawing conclusions from it.

In Energy Trading & Investing, David W. Edwards notes that spark spread can overestimate actual plant profitability by 20–30% once operational constraints are accounted for. The core criticism is that spark spread models are reactive. They assume a generator simply turns on when the spread is positive and turns off when it isn’t. Real plant dispatch doesn’t work that way.

A few of the key gaps:

• Ramp time: CCGTs take 1–4 hours to reach full output; peakers 10–30 minutes. A price spike that resolves before a cold plant can respond means the realized margin is lower than the spread implies. The optimal dispatch schedule has to be anticipated, not just observed.
• Start-up costs: Lighting off a plant burns fuel and causes mechanical wear. A brief positive spread may not justify the cost of starting, especially if the window is short.
• Fixed costs excluded: Spark spread covers fuel cost only. It does not account for pipeline costs, fuel-related finance charges, variable O&M, taxes, or fixed expenses. A plant can show a positive spark spread and still lose money overall.
• Hub vs. nodal prices: HB_NORTH is a trading hub. Actual plant economics depend on their nodal price, which can diverge significantly due to transmission congestion.

None of this makes spark spread useless. It remains one of the most widely used signals for understanding generator economics and grid stress. But as the EIA puts it, spark spread is an indicator of market conditions and not necessarily an exact measure of profitability for any one specific generator. Read it as directional, not precise.

Two Types of Grid Stress (And They Look Completely Different)

The Texas grid experiences stress in two different ways and the spark spread behaves differently in each.

Summer Heat Waves: When Gas Plants Win

August is often the best month to own a gas plant in Texas. August has the highest average CCGT spread at $19.82, making it the most profitable month to own a gas plant in Texas. The next highest months are May ($18.58) and January ($13.14), but August stands out as the clear leader. Temperatures regularly exceed 100°F, air conditioning load pushes demand to annual peaks, and gas prices stay low in summer because heating demand is absent. As a result, power prices spike while fuel costs stay cheap.

On August 20, 2024, the average power price at HB_NORTH hit $226/MWh with gas at $2.20/MMBtu. The CCGT spread that day was $210/MWh. The maximum interval price hit $4,853/MWh. In the entire month of August 2024, there was not a single day with a negative spark spread in my dataset, and according to the dataset I generated, CCGT plants were profitable 88% of all 15-minute intervals.

Shoulder Season: When Gas Plants Lose Money Every Day

Flip the calendar to late February or early March and the picture reverses completely. Temperatures are mild, demand is low, but gas prices remain elevated from winter. The spread collapses.

These are what we call the dead days, when the spread is so negative that not a single 15-minute interval is profitable:

On February 26, 2025, not one 15-minute interval across the entire day covered a CCGT plant’s fuel cost.

The Seasonal Calendar

The following statistics are calculated from my dataset covering Dec 2023–Mar 2026.

Zoom out, and the pattern holds true year over year. The table below shows multi-year averages at HB_NORTH (Dec 2023–Mar 2026):

March has the most negative-spread days of any month, as winter gas prices linger while mild temperatures keep power demand low. December is close behind, cold enough to spike gas, but often not cold enough to push power prices high enough to compensate. The rare exception: when a proper freeze arrives, like January 2026.

The charts show the economics: August dominates with $20/MWh CCGT spreads, 86% profitable intervals. Shoulder months dip negative. Peakers struggle everywhere except summer peaks.

Peakers burn 43% more gas per MWh, so they need much higher prices to break even. They exist because August profits cover the dead months. The spark spread tells you exactly when the grid is stressed and how much plants are making (or losing).

The Demand Side Mirror: What 4CP Has to Do With It

Gas plant operators aren’t the only ones watching the spark spread. Every large industrial customer in Texas, such as refineries, petrochemical plants, data centers, and aluminum smelters, has someone whose entire job in summer is monitoring ERCOT load in real time.

The reason is something called Four Coincident Peaks (4CP). Each year, ERCOT identifies the 4 highest 15-minute demand intervals during June through September. Those four moments determine each large customer’s share of transmission charges for the following 12 months. For a major industrial facility, the bill can run into millions of dollars. Getting caught at full load during a 4CP event or curtailing prematurely on a false alarm has significant financial consequences.

In a blog I read, they highlight just how extreme 4CP costs can be. For example, a CenterPoint customer facing a $56.51/kW 4CP rate would pay roughly $2.8M annually for a 50 MW load. This is a cost determined entirely by their demand during just four 15-minute intervals across the summer. In effect, a single hour of peak demand can drive millions in yearly charges.

In one of the tables we have generated we have a column called economic_ratio_ccgt that represents the ratio of intervals where the spark spread for a CCGT (Combined Cycle Gas Turbine) plant is economically positive compared to the total intervals in a day. On the hottest summer days in our dataset, that ratio hits 1.0, meaning all 96 intervals were profitable, power prices stayed elevated all day, and the grid was running tight from open to close. Those are the days when 4CP risk is highest.

The same signal that tells a gas plant operator “run hard all day” tells an industrial energy manager “this might be the one.” Generators are sprinting. Industrials are watching every interval. Both are responding to the same underlying condition and that is scarcity on the Texas grid

Building the Data Platform on Databricks

Analyzing spark spreads across 700+ settlement nodes, 96 intervals per day, and multiple years of history requires more than a few CSVs and notebooks. To make this analysis repeatable, I built a small data platform to continuously ingest, clean, and model power, gas, and weather data.

The platform is built on Databricks using Lakeflow Spark Declarative Pipelines (SDP). At a high level, it follows a medallion-style architecture:

• Bronze: Raw ingestion of ERCOT hub prices, Henry Hub gas data, and weather feeds
• Silver: Cleaned and aligned time series across all sources
• Gold: Derived metrics like spark spreads, profitability ratios, and daily aggregates

This structure makes it easy to move from raw market data to something analytically useful without constantly rewriting transformation logic.

SDP (formerly Delta Live Tables) handles much of the pipeline orchestration and data quality enforcement. One area where it stands out is how naturally it supports things like slowly changing dimensions and built-in expectations (for example, dropping or flagging bad data at ingestion time). You can read more about the concepts here and expectations here.

That said, I still prefer the flexibility of writing custom PySpark when I need tighter control or easier local testing.

Overall, the goal wasn’t to build a perfect platform, but a fast, flexible one that makes it easy to explore questions like:

• How do spark spreads behave during extreme weather?
• When do gas prices compress generator margins?
• What does “grid stress” actually look like in the data?

The diagram below shows how the pieces fit together. From raw ERCOT prices and gas data to a daily spark spread signal:

Note: I’m experimenting with animated diagrams here, let me know what you think!

Conclusion

It’s cool how much insight can come from combining a simple market concept with the right data infrastructure. The spark spread is a straightforward calculation, but when you view it across thousands of settlement points, weather events, fuel markets, and seasonal demand patterns, it becomes a powerful lens into how the Texas grid behaves under stress.

This project also turned into a great excuse to experiment with a modern data stack. Using Databricks SDP, I was able to ingest and model data from multiple sources including ERCOT market data, natural gas prices, weather feeds, and EIA datasets. Bringing these together in a medallion-style architecture made it easy to iterate quickly, test new ideas, and explore the relationships between weather, fuel costs, and electricity prices.

Another surprisingly helpful part of this journey has been working with Claude Code and a few MCP servers. Having an AI assistant available while exploring a new domain made it much easier to understand unfamiliar energy market concepts, locate useful datasets, and quickly prototype analysis pipelines. It has felt less like searching for information and more like collaborating with a research assistant while building the platform.

This is still very much the beginning of the exploration. I’m currently experimenting with bringing in additional datasets and expanding the platform to look at other ERCOT signals such as nodal congestion, real-time price spikes, and transmission constraints. There is a lot of interesting structure in these markets, and the combination of modern data platforms and AI-assisted development makes it easier than ever to dig into it.

Thanks for reading 😊!

Working with the folks over at Collide in Texas inspired me to build this project to learn the concepts and tooling they were leveraging. They are doing lots of amazing work in the oil and gas space with AI and process automation. If you’re interested in learning more about the business of oil and gas and how tech intersects with the energy industry, check out their podcast: Collide Podcast on YouTube.

Introduction

I have been having a blast over the last few weekends, vibe‑coding and leveraging Claude Code with Sonnet 4.6 and the ai-dev-kit from Databricks to build some interesting applications geared towards midstream oil and gas.

I had been relatively skeptical of vibe coding up until a few weeks ago, and, self‑admittedly, have not done a ton of it. I have been a bit underwhelmed by tools like GitHub Copilot, and I don’t seem to be the only one. Scrolling through Reddit, I found a post as recent as 4 months ago. The author claimed that Copilot’s code quality had “significantly dropped” over the past few months, even when using Claude Sonnet 4.5 inside VS Code.

And that sentiment isn’t isolated. There are multiple threads where developers describe Copilot as excellent for boilerplate and quick inline suggestions, but inconsistent when it comes to deeper reasoning, multi-file refactors, or nuanced debugging.

But my skepticism around vibe coding largely stemmed from those early Copilot experiences. The suggestions often felt like autocomplete on steroids rather than a true thinking partner. If that’s your baseline experience, it’s easy to dismiss the whole category.

At Plains, we’re collaborating with commercial teams to unlock value from our data lake datasets. I wanted to prototype something visually compelling. A geospatial app that would demonstrate the power of combining structured data, unstructured contracts, and maps. And who doesn’t love geospatial data?!

I thought to leverage Claude Code and the new Databricks ai-dev-kit to generalize what we have started building at Plains for the purpose of this blog post. I also wanted to take the opportunity to test out ai-dev-kit and generate synthetic data for features and functionality that are both aspirational and that we have actually built out internally to demonstrate what we can do once we have all the data landed and modeled within our data lake. I appreciate the synthetic data is not perfectly accurate to what we would see in the field, but it is not bad given it was generated using Claude and the Databricks MCP server.

We have many of our datasets landed in the data lake, so although the data itself is proprietary, the patterns and tooling we are building are broadly applicable. For this post I generalized it into a synthetic Permian Basin dataset for things like wells, crude and gas pipelines, acreage dedications, processing plants, and compressor/pump stations and built a full interactive mapping application on top of it, all running on Databricks.

The Use Case

We are seeing a large amount of data being locked away in unstructured contracts. Like I mentioned in a previous post, the ‘oil and gas industry is burdened by complex contracts that are often lengthy, unstructured, and scattered across various formats’, and we have built several capabilities internally at Plains to OCR, extract structured information, and query said data at scale and in one recent example exceeding 25,000 contracts. Contract data in isolation is useful for compliance and record keeping. Contract data connected to a geospatial context becomes a strategic asset, especially when you pair it with other geospatial datasets to visualize pipelines, wells, and in the unstructured contract example, acreage dedications which are inherently spatial. They define where volumes are contractually committed across counties, basins, and lease blocks.

A commercial analyst may know things like the contract start and end date, the operator’s name, and the minimum volume commitment (MVC), but other questions might be a bit more difficult. Things like:

• Where does that acreage physically sit relative to our pipelines?
• Is production in that block trending up or declining?

The goal was to generate synthetic data like data sources I have seen in the energy industry like Enverus, pipeline data modelled in PODS, acreage dedication contracts etc. and demonstrate how we geospatially visualize this data in an easy to use and understand map interface, allowing the commercial team to answer tough questions like:

‘What does this contract mean spatially, operationally, and competitively?’

What Was Built

The app was built to directly answer those use case questions. An analyst can pull up the map, click on an acreage dedication to see its spatial footprint relative to pipelines, check whether nearby wells are producing strongly or declining, open the linked contract PDF, and query across all of it in plain English, all without leaving Databricks. Specifically:

• “Where does that acreage sit relative to our pipelines?” → acreage dedication and pipeline layers are overlaid on the same map, making spatial relationships immediately visible.
• “Is production trending up or declining?” → clicking any well surfaces a BBL/MCF time-series chart pulled live from Unity Catalog.
• “What does this contract actually say?” → clicking an acreage block opens the linked PDF contract directly in the app.

A lot of the boilerplate and foundational software components were in place before vibe coding began. I had already decided on a language and framework (React + Node.js), a mapping library (Mapbox GL), where I was going to host it (Databricks Apps), and how the app would authenticate and query data (Unity Catalog via a SQL warehouse). Having those decisions locked in upfront meant Claude wasn’t being asked to make architectural choices and instead it was being asked to build features within a well-defined structure. That’s a meaningful distinction. Vibe coding works best when the AI is filling in the what, not deciding the how.

The app is a React + Mapbox GL frontend served by an Express.js backend, deployed as a Databricks App. All the operational data lives in Unity Catalog and is queried at runtime through a SQL warehouse. The data being represented on the map is coming from a mix of structured and unstructured datasets. I have learned over the past 6 months that people really seem to like viewing data on a map. Geospatial data can be incredibly powerful and can help tell a cool story.

The choice of React + Mapbox GL was deliberate. Mapbox is the closest thing to an industry standard for interactive geospatial work outside of enterprise tools like Esri. It handles GeoJSON layers, dynamic filters, and camera animations natively, and react-map-gl gives you a clean component wrapper around it.

Express.js was chosen because it lets you serve both the React single-page app and all the API routes from a single Node.js process, which maps cleanly to how Databricks Apps works. One command, one process, no separate infra.

Bringing the app to the data (rather than extracting data to an external host) was a conscious decision: the Express server runs inside Databricks, which means it authenticates to Unity Catalog natively using the injected OAuth token with no data leaving the platform.

Every layer of the map can be interacted with, including:

• Wells
• Crude and gas pipelines
• Acreage dedications
• Compressor/pump stations
• Processing plants

Results are cached in-memory for 5 minutes so navigating around the map doesn’t re-query the warehouse on every interaction.

Similar to a dataset like Enverus, we can view well production history in a time-series chart to get an idea for BBL and MCF being produced from a given well.

We also integrated the geospatial experience with Genie, where a user can ask questions like:

‘Which acreage blocks have no active contracts, show their locations?’

Genie will interact with the tables in Unity Catalog and once a response is returned, the corresponding acreages will be highlighted on the map for exploration:

Getting the results from Genie to highlight on the map was a fun challenge. Genie generates SQL against Unity Catalog, the server re-executes that query via the warehouse, extracts the matching well or acreage IDs from the result, and passes them back to the frontend so Mapbox can highlight just those features while dimming everything else on the map.

Another cool query I was playing around with:

‘Show me active oil wells in Reeves or Loving County with an IP30 above 500 BOPD that have no active dedication and production numbers look strong. Include well name, operator, formation, IP30, and cumulative oil’

I leveraged the ai-dev-kit (i.e. Databricks MCP server) heavily to create objects within Databricks. Things like Catalogs, Schema, and Tables were all created and populated using natural language in Claude Code. I asked Claude Code to do things like:

‘Create some dummy PDF contracts, upload them to a Databricks volume in the Permian catalog, and link to them in the app when selecting an acreage dedication, and open the PDF within the app’

…which allowed me to quickly generate dummy data and link it directly into the interface I was building:

Opening one of those associated contracts reveals some pretty convincing dummy data:

The dummy data generated is not perfect and you will notice in the above screenshots there are things like overlapping acreage dedications, but I think overall Claude and its integration with the Databricks MCP server did a solid job.

Conclusion

My eyes have been opened leveraging tools like Claude Code and the new ai-dev-kit from Databricks to quickly prototype applications. So long as you understand how the Databricks platform works and the sequence of steps you need to orchestrate, you can guide Claude to build some cool stuff on Databricks.

Lessons Learned

I attempted to leverage Claude to deploy my JavaScript application to Databricks Apps and it did not work very well.

The deployment failure was the most instructive moment of the whole project. I asked Claude to deploy the Node.js app to Databricks Apps before I fully understood the steps myself. After several failed attempts I had to step in, manually work out the correct deployment sequence, and only then delegate the automation of that process back to Claude to create a GitHub Actions pipeline to automate the process of deploying the app going forward. That pattern generalized well: Claude is a force multiplier once you understand the problem space, but it will struggle when you don’t understand the platform well enough to verify its output.

A few other things I’d keep in mind for next time:

• The MCP server excels at data operations: Creating catalogs, schemas, tables, uploading data to volumes, but is less reliable for platform infrastructure tasks like app deployment, at least right now.
• Decide on your stack before vibe coding begins: Having already chosen React + Mapbox GL + Express freed Claude to focus on building features rather than making architectural decisions.
• Synthetic data is awesome for prototyping: Being able to ask Claude to generate realistic contracts and well data and load them directly into Unity Catalog via MCP collapsed what would have been days of data prep into minutes.

What’s Next

The synthetic Permian dataset served its purpose for this post, but the real value unlocks when this runs on actual data. As more of our datasets land in the data lake, swapping out the synthetic layer is a straightforward schema change. A few things are on the roadmap beyond that:

• Pass-through authentication: so the app respects Unity Catalog row-level permissions, a user should only see the contracts and acreage dedications they’re entitled to.
• Extending to other basins: the Permian is the proof of concept, but the same architecture applies anywhere we have pipeline and contract data.
• Tighter Genie integration: to support more complex, multi-step analytical workflows directly from the map.
• Contracts tab: to quickly view and search contract details and filter by operator, county, and AI‑generated contract summaries.

I think this application is a great example of the power of Databricks Apps running on the Databricks platform. There is huge value in bringing the apps to the data, instead of the data to the apps.

Thanks for reading 😀!

Disclaimer:
Hidden Markov Models were new territory for me, and understanding the underlying math required building the forward, backward, and Viterbi procedures from scratch in NumPy. While I’ve developed a much stronger intuition for how the parameters λ = (A, B, π) drive inference, I’m still refining my understanding. This post reflects what worked in our environment and what I’ve learned so far.

This post is a follow-up to 🛢️🤖 Why Detecting O&G Pipeline Anomalies Is So Hard, where I discussed the technical complexity we encountered while building a capability to ingest PI historian data into Databricks for further processing.

Use Case

The technical work is driven by one core objective: reduce unnecessary pipeline shutdowns caused by false leak alarms. In many cases, these shutdowns are not caused by true failures, but by conservative safety mechanisms reacting to limited visibility into the true state of the pipeline and incomplete or uncertain system context. A common pain point is how operational slack is assessed. Controllers often must manually estimate slack conditions in real time, without a fully observable view of packing or unpacking behavior across the line. It’s error-prone and easy to miss during high-pressure operations. When slack is misjudged (or missed entirely), monitoring systems can trigger false leak or deviation alarms, leading to avoidable shutdowns and unnecessary investigations.

To address these challenges, we developed the Deviation Counter Tool to work hand-in-hand with our pipeline state detection. When the ML model infers that the pipeline has entered a specific state, such as unpacking (i.e., shutting down), the deviation counter logic is automatically activated. This tool continuously monitors for deviations, removing the need for manual slack estimation and reducing the risk of false alarms.

Key Terms: Slack, Pack, and Unpack

• Slack: The presence of unpressurized or low-pressure sections in a pipeline, often due to imbalances between inflow and outflow. Slack can lead to inaccurate flow measurements and complicate leak detection.
• Pack (Packing): The process of increasing pressure in the pipeline by introducing more product (inflow exceeds outflow), typically during startup or ramp-up operations.
• Unpack (Unpacking): The process of decreasing pressure in the pipeline by reducing inflow or increasing outflow (outflow exceeds inflow), often during shutdown or ramp-down operations.

Data Ingestion

Everything starts with data engineering. None of the reporting, analytics, or ML happens without it. We are leveraging the PI Web API to pull a subset of PI tags from our meters. Specifically:

• Inflow (product entering a meter)
• Outflow (product leaving a meter)
• Over/short (outflow minus inflow, where a negative value indicates potential loss or leak)

We’re ingesting 1-minute data every 3–5 minutes into our data lake and exposing the tables in Unity Catalog for dashboarding and machine learning.

A lot of effort has gone into hardening this ingestion pipeline.

We’re also running a data quality engine during ingestion. For example, we explicitly check for stale SCADA records. If a DQ issue is detected, we flag it and ignore the data in our ML model for state prediction. We don’t want to generate predictions off poor-quality historian data.

Machine Learning Approach

In the previous blog, I talked about potentially approaching this use case as a classification problem. We are still considering that approach, potentially leveraging something like a nearest neighbor algorithm, and modeling different pipeline states (i.e., normal, shutdown, leak). However, we have had some recent success using a Hidden Markov Model (HMM). In fact, there’s relevant academic work that inspired this direction from the College of Science, Engineering, and Technology in Houston. In ‘Hidden Markov Models for Pipeline Damage Detection Using Piezoelectric Transducers’, researchers applied an HMM-based method to detect pipeline leaks and crack conditions by mapping different damage conditions to distinct Markov states and using statistical signal features as the observable emissions. Their method showed that a Gaussian mixture model HMM (GMM-HMM) could successfully recognize whether a pipeline had a leak, and in some cases, locate it under time-varying conditions, despite noisy measurements.

HMMs can be useful for handling sequential data where the underlying process isn’t directly visible, but you can infer it from patterns in the observations. They typically excel in noisy, time-series environments because they account for uncertainty and temporal dependencies. The HMM is based on a Markov chain, which is a model that tells us something about the probabilities of sequences of random variables (states), each of which can take on values from some set. Hidden Markov Models are designed specifically for sequential data where the true state is not directly observable. That description fits pipeline operations almost perfectly.

From Weather to Pipelines: A Markov Perspective

The classic example for HMMs is the “weather example”:

In this diagram:

• Hidden states: Weather conditions (Rainy, Cloudy, Sunny)
• Observations: Emotions (Happy, Neutral, Sad)
• Green arrows: Transition probabilities, the likelihood the weather changes from one state to another each day
• Red arrows: Emission probabilities, the likelihood of observing a particular emotion given the current weather

We only see the emotions (observations), not the weather (hidden states). The HMM helps infer the most likely sequence of hidden states behind those observations.

The same idea applies to pipelines.

Observable data (what we measure):

• Inflow
• Outflow
• Over/short

Hidden states (what we actually care about):

• Normal operation
• Line packing (starting up)
• Line unpacking (shutting down)
• Leak
• Shutdown
• Transition states

An HMM asks a more realistic question:

“Given everything we’ve observed up to now, what state are we most likely in, and how likely is it to transition to another state?”

That temporal dependency is critical because pipeline states persist. A leak doesn’t appear and disappear randomly minute to minute. A line doesn’t pack instantly. Modeling persistence reduces noise-driven false positives.

This model is a multivariate Gaussian HMM (one Gaussian per state) with 6 hidden states and diagonal covariance, trained unsupervised on 9 standardized, flow-derived features (inflow, outflow, over/short, ratios, deltas, and operational flags).

The HMM was trained and versioned within Unity Catalog, and it took nine iterations to align the transition structure and feature engineering with operational reality. I’m sure many more iterations are ahead as we collect more data, observe edge cases, and continue refining how the model reflects real pipeline behavior.

We leveraged tools like StandardScaler to account for unit variance, and engineered features for flow relationships (e.g., inflow/outflow ratio), temporal dynamics (e.g., delta inflows & outflows based on a lag window), and operational flags (e.g., binary flag for inflow < 5.0 (shutdown detection)).

Orchestration & Deployment

All our artifacts in Databricks from ingestion of the PI data to predicting pipeline state via batch inference leveraging the HMM, are deployed via asset bundles and orchestrated in workflows.

We are also leaning heavily on Unity Catalog to version our HMM, so we have version history for training iterations.

Conclusion

One challenge we’ve run into is temporal resolution.

We’re currently ingesting 1-minute data every 3–5 minutes. That sounds reasonable until you see something like this:

At first glance, everything looks within range… and then it drops to zero.

We pulled second-by-second data for that same window to see what happened in those 60 seconds.

Apparently, a lot can happen in 60 seconds in pipeline operations.

We may need to increase ingestion frequency because pulling at a minute granularity can miss critical context. When you’re modeling state transitions, that missing context matters.

There is still a lot to learn, but we’re getting closer to something that reflects operational reality rather than just reacting to noisy signals.

Big shoutout to Shawn for helping sharpen how we framed the operational problem, and to Mark for leading the data engineering effort that makes any of this possible.

Thanks for reading! 😀

Who doesn’t love talking about data quality?

It may not be the most exciting topic right now (or ever), but it certainly is very important. I am sure we have all seen the meme below circling around online, but it is so true and it demonstrates the importance of high-quality data.

Mark van der Linden and I have had many conversations about this and he always says, ‘never use data unless you understand its quality’, and that goes for the full stack. Reporting, analytics, and especially all kinds of AI.

Looking back, I’ve jumped headfirst into use cases more times than I’d like to admit without truly grasping the data’s quality upfront. Had I prioritized that early, I would’ve avoided a ton of rework, set more realistic expectations with stakeholders, and delivered better outcomes faster.

Data quality issues often tell a compelling story. When you surface gaps or inconsistencies to business owners and collaborate on root-cause fixes, like process changes that improve capture at the source, it turns a technical problem into a shared opportunity for improvement. Stakeholders get bought in, and the data gets better.

Full disclosure, I haven’t gone deep into dedicated data quality tooling yet, but I’ve always been intrigued by enterprise solutions like Informatica Data Quality.

That said, this ties directly into the philosophy Mark and I explored in our recent post The Art of Keeping Things Simple. Heavy tools bring specialized skills, complex licensing, and yet another vendor to manage, exactly the kind of over-engineering we argued against.

So, why not keep things simple? We’re experimenting with embedding a lightweight, SQL-based data quality engine directly into Data Weaver on Databricks. The goal: stay within a single, unified framework (Databricks + SQL/Python) for ingestion, transformation, orchestration, and quality checks, no extra layers, no context-switching.

Use Case

I always like to ground technical solutions in business value. It’s easy for us as technologists to forget why these systems exist in the first place. Let’s talk about the use case we are looking at to build some DQ rules.

In a recent post, I talked a bit about getting data out of mainframe for a series of commercial focused use cases we are working on. Mostly, we’re dealing with volumetric receipt and delivery data at the contract level (i.e., what are our contractual volumetric commitments). We also pull in data from our scheduling system to verify if those committed volumes have been scheduled (i.e., ensuring logistics are in place for timely and accurate scheduling of oil and gas deliveries), as well as our crude oil accounting system for the volumetric actuals.

To expand this further, we’re integrating third-party data from Enverus, a leading provider of energy analytics. Enverus supplies granular insights into production forecasts, well performance, and market trends, which we layer onto our internal datasets. This helps us validate contractual commitments against external benchmarks, like regional production averages or competitor activity. For instance, we can cross-check our volumetric data against Enverus’ well-level production estimates to spot discrepancies early, such as over- or under-estimated reserves that might impact contract viability.

A key area where this shines is in handling acreage dedications. In the oil and gas space, acreage dedications lock in specific land areas (often measured in acres or sections) to a midstream contract, committing all production from that acreage to a particular pipeline, gatherer, or processor. Our data quality engine scrutinizes these dedications by validating spatial and volumetric alignments:

• Are the dedicated acres accurately mapped to contracts?
• Do production actuals from those acres match the dedicated commitments?

We flag issues like overlapping dedications (e.g., the same acreage tied to multiple contracts) or mismatches between dedicated volumes and historical output, which could lead to legal or operational risks.

This ties directly into right-sizing contracts based on historical actuals. By analyzing trends in actual deliveries from our accounting system, spanning months or years, we can recommend adjustments to contractual minimum volume commitments (MVCs) or take-or-pay clauses. For example, if historical actuals show consistent under-delivery on a dedicated acreage block (perhaps due to declining well productivity, as corroborated by Enverus data), we can simulate scenarios to “right-size” the contract:

• Reduce MVCs to avoid penalties
• Renegotiate terms for flexibility
• Even release underperforming acreage from dedications

This not only optimizes costs but also aligns commitments with real-world performance, reducing exposure to shortfall fees.

Full disclosure: much of the above is still aspirational. We’ve built pieces of this today, but not the entire end-to-end vision yet. What matters most at this stage isn’t the completeness of the technology, it’s deeply understanding the business of scheduling, marketing, and contract management. Once you truly understand how volumes flow, how commitments are made, and where real-world constraints live, it becomes much clearer what is and isn’t possible from a technical standpoint. The technology follows the business, not the other way around.

SQL Based Data Quality Engine

Now let me put my propellor hat back on and talk about some tech!

Rather than introducing a standalone data quality platform, we opted for a lightweight, SQL-first data quality engine embedded directly into our Databricks workflows. The guiding principle is simple: data quality should live where the data already lives, using tools the team already understands.

At its core, the engine follows a three-table design inspired by dimensional modeling patterns. The goal is to separate what we are checking from how often issues occur and where they occur, enabling both high-level monitoring and deep operational investigation.

Three-table design:

• dq_dim_rule (Dimension Table)
  • Defines each data quality rule with metadata
  • Fields: RuleId, Area, ShortName, LongName
• dq_fact_rule_summary (Fact Table - Aggregated)
  • Stores daily aggregate counts per rule
  • Fields: Id (auto-increment), RuleId, Count, Date
  • Provides high-level metrics for dashboards and trending
• dq_fact_rule_detail (Fact Table - Granular)
  • Stores individual DQ violations per rule
  • Fields: Id (auto-increment), RuleId, ContractId, Date
  • Enables drill-down investigation of specific problem data.

The Databricks notebook runs daily to:

• Delete existing detail records for each rule and current date
• Calculate violations by querying source tables (receipt_data, delivery_data, scheduling_data)
• Insert both summary counts and detailed contract IDs for each rule
• Track data quality trends over time with the date dimension

This is where data engineers, analysts, and business users go when they want to understand which contracts are problematic and why.

Execution Pattern

A daily notebook in Databricks orchestrates all checks and follows a consistent, idempotent pattern:

1. Remove existing records for each rule and the current processing date
2. Execute SQL logic to identify violations across source systems:
   • Receipt data
   • Delivery data
   • Scheduling data
3. Insert:
   • Aggregated violation counts into the summary table
   • Individual contract violations into the detail table
4. Persist results with a date dimension to enable historical trending

This approach ensures the job can be safely re-run, supports incremental rule additions, and keeps the entire framework transparent and auditable.

We’ve implemented several foundational rules so far, such as:

• Contracts missing receipt or delivery volumes
• Spot deals with incomplete or missing date ranges
• Mismatches between scheduled and actual volumes

Today, these results are surfaced through Databricks AI/BI dashboards, giving stakeholders real-time visibility into how data quality evolves over time, rather than treating it as a one-off validation exercise.

Conclusion

What this approach has reinforced for us is that meaningful data quality work doesn’t require heavyweight platforms, complex rule engines, or yet another vendor to manage. In many cases, it simply requires:

• Clear business context
• Well-understood data
• And a disciplined, repeatable way to surface issues

By embedding data quality checks directly into our Databricks pipelines using SQL and Python, we’ve kept quality close to the data, close to the engineers, and most importantly close to the business questions that matter.

This approach won’t replace enterprise data quality platforms in every scenario, nor is it meant to. But for teams already operating in a modern lakehouse environment, it offers a pragmatic, low-friction way to start treating data quality as a first-class citizen, without over-engineering the solution.

In a future blog we will talk about data profiling, specifically a tool in Data Weaver that provides the ability to rapidly profile data to quickly get a feel for the quality of the data.

Thanks for reading 😀

Mainframe technology has always been intriguing but has largely been a mystery to me how it worked or what ‘mainframe’ even meant.

Working at Plains, many of our commercial applications for crude oil accounting, scheduling, volumetric data (noms and actuals) and contract capture all run largely on the mainframe. Over the past few months, I have had the opportunity to learn from and work with a few mainframe developers and SMEs on the commercial side of our business.

As mentioned above, my biggest challenge learning all of this was that I did not really understand what mainframe meant. So, to get my hands dirty, I leveraged some open-source technology to spin up an extremely barebones mainframe on my computer to get a feel for various mainframe concepts.

In this post, I’ll walk through what a mainframe is, how I emulated one, and how data moves from mainframe to Databricks. I hope this post demystifies some mainframe concepts for you!

What is Mainframe?

Talking with friends in other industries, mainframes are still very much the backbone of enterprise computing for industries like banking, airlines, and energy, where transaction integrity and speed are critical. Several IBM studies indicate the continued use of mainframes is widespread and will continue well into the future.

Mainframes handle almost 70% of the world’s production IT workloads, according to one of the studies.

Mainframes are enterprise data servers engineered to handle massive, mission-critical workloads, and it is not uncommon for them to process transactions in the trillions per second. They are built to run the world’s most important data systems without interruption, even under extreme load.

The term mainframe initially referred to the large cabinet or ‘main frame’ that held the central processing unit (CPU) of early computer systems.

Relevance to Plains

The role I have at Plains has inspired me to get hands-on with mainframe technologies since many of our commercial applications run on an IBM z/Architecture (or something very similar). With my role focused on data engineering/ML, getting data out of mainframe into our Lakehouse is critical.

To have more informed conversations with that team, I wanted to get hands-on with the technology. I am a big believer in learning by doing.

After a bunch of research, I found an open-source software implementation of the mainframe System/370 and ESA/390 architectures, in addition to the latest 64-bit z/Architecture. While infinitely less complex than the mainframe environment we have at Plains, it has still allowed me to get my hands dirty with some mainframe concepts. I also validated the environment with our internal mainframe team to make sure I was on the right track with my learning.

The software is called Hercules, and it has been incredibly helpful to wrap my brain around mainframe technology.

Getting Hands-On with Mainframe

To be honest, I had not even really seen what a mainframe looked like prior to experimenting with this emulator, so this was a fun experience. After logging in, you are greeted with what is called an Interactive System Productivity Facility (ISPF) menu that provides a way to interact with the mainframe.

There are a wide variety of options here, most of which I have not touched. Most of my time has been spent in the UTILITIES option. Selecting option 3 will take us there.

From there, we have several more options to choose from. I really wanted to get hands-on with some COBOL programming, so I spent most of my time there.

Once inside the Utilities menu, I quickly discovered that everything in the mainframe world revolves around datasets. These are essentially the mainframe’s version of files, but they’re far more structured and rigid.

• Some datasets behave like simple text files.
• Others act more like folders, holding multiple “members” such as JCL scripts, COBOL programs, or configuration files.

JCL (Job Control Language) was cool to learn about, and it is basically an instruction sheet that tells the mainframe what work to do. A JCL script defines:

• Which datasets to read from
• Which programs to run
• Where to put the output
• What system resources to allocate

When you submit a job, the mainframe reads your JCL, locates the datasets you reference, runs the COBOL program you point to, and routes the output into spool datasets that you can view through SDSF (System Display and Search Facility) or another similar job log viewer.

Selecting option 4 or DSLIST allows me to search for the dataset I want. In this case, I am searching SYS2, which is used for datasets that belong to the operating system, utilities, or shared components, not to individual users.

This emulator I have loaded has lots of prebuilt COBOL scripts that I can experiment with to get a feel for how it works. All those examples are contained within the dataset SYS2.JCLLIB. Pressing enter brings up the following screen where I can select my dataset by using the arrow keys and entering V to view the contents.

After pressing enter, we are brought to another screen where we can see a variety of different things, COBOL scripts being one of them. I stumbled upon a basic ‘Hello World’ application in COBOL to familiarize myself with the syntax and get a feel for submitting jobs on mainframe.

Scrolling down to that file and entering E allows us to view the syntax of that COBOL application.

This COBOL script just prints Hello World. We can submit this job by entering the submit command in the Command prompt.

Doing so returns this:

I was trying to identify where I could see the output of this job, but could not figure it out 😂

Talking with the mainframe team at Plains, they leverage a tool called System Display and Search Facility (SDSF) to monitor, manage, and view system activity, job queues, and job output in real-time. I do not have access to that on my emulator, and when I tried to view the logs in the OUTLIST utility, which basically displays JOB output, I could not find it.

I can see there is something called SYSOUT where printed records are stored, but I could not figure out how to access that. In theory, I should have been able to view the ‘Hello World’ output there.

Extracting Data from Mainframe

One of the most important aspects of mainframe systems is their use of specialized databases. Unlike modern relational databases, many mainframe databases such as IBM’s IMS (Information Management System) are hierarchical in structure. This means data is organized in a tree-like format, with parent and child records, rather than tables with rows and columns. I have yet to dig into these databases at Plains but I hope to in the coming months.

As things stand now (and as far as I understand), we are leveraging scheduled jobs via JCL that call mostly COBOL code on mainframe. These jobs export data from various commercial apps that run on mainframe, into a tilde separated format. These tilde separated files are often times SFTP’d between file shares for consumption.

There are also .NET client applications built for deal capture that leverage a SQL database where data is copied from mainframe into these client SQL databases on a scheduled basis.

We are tying into a mix of the tilde separated files and the client app SQL servers to pull volumetric, deal information, and lease supply data and using Data Weaver to ingest and process this data in Databricks.

While we are not connecting directly into the mainframe IMS databases today, depending on our end users requirements, we may need to explore that to pull that data on a more regular interval.

I suspect this will not be a problem for now since the jobs on mainframe to extract the data are scheduled to run in accordance with crude oil accounting requirements and operational workdays. In a midstream context, a “workday” is not a calendar day, but a contract-defined 24-hour operating period used for measurement, allocation, inventory reconciliation, and settlement.

Aligning extract jobs to these workday boundaries ensures that volumetric data, deal activity, and lease-level movements remain consistent with how the business measures and settles crude. As long as downstream ingestion and processing remain in sync with these accounting cutoffs, the data we land in Databricks should reconcile cleanly with commercial and accounting systems, even if the underlying source systems and technologies differ significantly.

Conclusion

I had fun trying to get all this running on my laptop and I learned a ton in the process. What the mainframe has done and is doing at Plains is amazing, and I am excited to see what data and information we can unlock out of the commercial apps that run on the IBM mainframe using some cutting edge technology in Databricks.

I hope those that have not interacted with a mainframe (like me) found this blog helpful. I am excited to continue working with the team at Plains to learn more about this technology.

Thanks for reading! 😀

Managing data lakes, especially before modern tooling, was a painful, time-consuming undertaking. There are many aspects to managing data lakes, but I wanted to focus this blog on the data security aspect.

Data security is the bane of my existence and is a consistent issue with ETL workloads as you are effectively copying data from a source system with its own access control into another system, probably Azure Data Lake Storage (ADLS), that has its own set of permissions and access controls. I have never seen security control mapping from source systems to data lakes done well. Most organizations unintentionally expose data far more broadly than they should.

I have worked in a few industries, each with varying degrees of security requirements. I have worked in airlines where we dealt with passenger records and PII information that needed special care from a security standpoint. I, however, have spent most of my time in energy and oil & gas and I have found that keeping data security as simple as possible helps keep work moving quickly.

At Plains, we follow an ‘open by default and closed by exception’ policy where each data set that is landed in our lake is by default ‘open’ to all, and we restrict access to data sets that are identified as sensitive on ingestion. If you work in a more sensitive industry like banking, that may not be a reality for you.

In this blog, I wanted to share a few stories and experiences I have had over the years securing data lakes and the pain of doing it.

As I was thinking about these previous experiences, I also want to talk about the amazing benefits Unity Catalog provides and how it could have saved me a lot of pain 5-6 years ago.

The Days of Data Lake Gen1

My first couple of years in the data space were purely focused on DevOps and integrations. Basically, making the release process for the data engineers more efficient and integrating all the many services we used.

It has been a while, but I believe we had an architecture like this:

I have annotated the above diagram with numbers to explain how the data moves between these services:

1. ADF leveraging the integration runtime pulls data from on-premises systems
2. Data is dropped into bronze as parquet
3. Databricks picks up the data from bronze and converts it into delta parquet moving into silver
4. ADF picks up the data from silver and puts it into ‘gold’ which is either analysis services or Synapse depending on the use case
   • Fact/dimension tables loaded from gold zone or Synapse
   • Analysis services built the semantic model used by Power BI
5. Power BI for reporting

As mentioned earlier, my sole job at the time was purely to manage integrations and the DevOps processes that supported the release of artifacts into production.

The industry has made leaps and bounds in progress to simplify this process but at the time this was incredibly complicated. There was no unified CI/CD pipeline across services, ADF’s JSON ARM templates were painful, AAS DevOps was extremely manual and the identity and access model were incredibly siloed across these services.

Each of these challenges could have individual blogs written about them, so to keep this blog focused, I want to move on to talking about the complexities around managing security in data lakes.

The above architecture references Data Lake Gen1, which I believe has been marked as out of support by Microsoft, but even Data Lake Gen2, I would argue, has the same technical challenges I am about to lay out.

The Pain of Managing Data Lake Security

If I knew what I know now back in 2019, I would have saved Conner back then a lot of pain. Now I want to talk about how managing data lake permissions back then was difficult and set up the blog to talk about how Unity Catalog addressed many of these challenges. Let’s start by talking about how POSIX-style permissions are limiting.

POSIX ACL Limitations

An Access Control List (ACL) is a list of permissions attached to an object (such as a file or folder) that specifies which users or groups can access the object and what operations they can perform.

This is a huge oversimplification of the technology, but Azure Data Lake Storage Gen1 and Gen2 implement a hierarchical file system with POSIX-style permissions and ACLs, similar to those found in Linux file systems. This means access control is primarily managed through three permission types: Read, Write, and Execute.

• Read (r):
  • For files: Allows viewing of the file’s contents.
  • For directories: Allows listing the contents of the directory (e.g., using ls).
• Write (w):
  • For files: Allows modifying or deleting the file.
  • For directories: Allows creating, deleting, or renaming files within the directory.
• Execute (x):
  • For files: Allows running the file as a program or script.
  • For directories: Allows entering or accessing subdirectories within the directory.

Note: Given the amount of permissions Unity Catalog exposes, it becomes very clear that RWX permissions are incredibly limited.

ACL Inheritance Wasn’t Reliable

There were lots of issues around setting permissions as well if you were not mindful in setting them up prior to ingesting data.

Let me explain this in an example, imagine you are streaming data in from a source. In my case, when I was helping maintain these services, we were bringing in PNRs and DCS data (passenger records and departure control records) from a large airline.

If you imagine all the flights taking off and landing, there was a high volume of data coming in. The integration pattern looked something like the below. We had an application publish messages to an event hub, and we had Databricks read from the event hub queues and drop the messages into parquet format in our data lake.

It was exciting to see this data flowing into our lake after all the hard work building ARM templates and making sure everything was integrated nicely. However, I remember my excitement quickly dwindled when one of the data engineers came by and asked ‘Hey, can I get access to the PNR data please’?

I remember clicking through the UI in the Azure Portal and attempting to add permissions, but I made one bad assumption. I assumed there would be a way to recursively set the POSIX permissions on the PNR folder. Basically, I needed to set permissions on every single object (file and folder) within the PNR folder. Otherwise, the data engineer requesting this could get access denied reading a parquet file buried in a child folder somewhere.

You cannot recursively set permissions in the portal and you need to write some code to do this.

In our case, we wrote a .NET application to apply new users to these folders. Since the streaming data had been coming in for a few weeks, the number of files and folders in the PNR folder grew into the hundreds of thousands.

Running a script made this easier, but it was prone to failure due to network or timeout issues. For example, the script could get a good portion done assigning the permissions and it would fail, meaning some of the folders would not get the right permissions set. Data engineers would often report unauthorized error messages when trying to read those parquet files. Since the script had no form of state and could not pick up where it left off, we had no choice but to rerun the script from the beginning and continue running it until the data engineer no longer received that error.

As time went on, our code evolved to include things like ‘state/checkpointing’ so we could rerun the script from where it left off, but it really felt like we were building some pretty intricate software and it was just another thing to add to the long list of things we had to maintain.

ACL Limits

Another oversight we did not realize at the time was that there were hard limits as to how many ACLs you can set on a file or folder. The maximum number of access ACLs per file or folder is 32. This means only 32 objects can have access to a file/folder at a time. Our script was assigning individual users access to these folders 🤦‍♂️, which did not scale and we hit that limit very quickly. The obvious solution here (hindsight is always 20/20) was to assign Microsoft Entra ID groups access to the data lake objects. This was going to require our .NET software to become more sophisticated, creating groups and assigning them on demand.

The ‘Solution’

Full disclosure, all of this happened a number of years ago, so I am a bit fuzzy on the details.

After all that pain, we learned some important lessons and were ready to revise our approach. We spent a bunch of time sitting with vendors and updating our .NET code to roll out a new strategy for managing security in our Data Lake. The cleanup of existing permissions is a whole other story, but we basically had to write more code to recursively remove all existing permissions and apply the new security pattern, while taking into account the disruption this would cause. It was a gong show!

The Model That “Finally Worked”… for a While

The model we settled on was straightforward. Every source system got a Microsoft Entra ID group for readers and another for writers, and we applied those groups directly to the parent folder in Azure Data Lake Storage Gen2 before any data landed. Because Azure Data Lake Storage uses inherited POSIX-style ACLs, anything that arrived beneath automatically picked up the same permissions. For the more complex relational systems, we expanded this pattern so we could secure both at the source level and at the individual table level. It still didn’t give us anything close to column-level or row-level access control, but compared to where we started, it was a massive improvement.

This setup actually worked well for a while. We weren’t constantly hunting down folders that lost inheritance or ACL entries that exceeded the 32-item limit. Permissions became something we only had to think about once during source onboarding. Users were simply added to the appropriate Entra ID groups and everything just flowed from there. For a moment, it felt like we had finally tamed the complexity of data lake security.

Then we were blindsided by something we didn’t expect at all: Azure authentication itself.

How Azure Authentication Works

This is where understanding how Azure handles authentication becomes important. When a user or service accesses something like Azure Data Lake Storage, Azure issues a JSON Web Token, or JWT. It’s just a small, signed package of information that gets passed along with each request. Inside that token is all the context Azure needs to authorize the user, who they are, which tenant they belong to, and, most importantly for us, which security groups they’re in. Azure Data Lake Storage checks the object IDs of those groups against the ACLs on the folders and files and decides whether the request should be allowed.

A simplified JWT looks like this:

{
  "upn": "user@company.com",
  "oid": "1234-5678",
  "groups": [
    "guid1",
    "guid2",
    "guid3",
    ...
  ]
}

These group GUIDs are how Azure Data Lake Storage (and anything using POSIX ACLs) decides whether a user has read/write access to a file or folder.

The challenge is large enterprises have employees who belong to hundreds or even thousands of Microsoft Entra ID groups.

However, JWTs have size limits. When the group list becomes too large, Azure cannot fit them all into the token. So Azure switches to what’s called the “overage claim” mode.

Instead of including the group list, the token contains:

"_claim_names": { "groups": "src1" },
"_claim_sources": {
  "src1": {
    "endpoint": "https://graph.microsoft.com/v1.0/me/getMemberObjects"
  }
}

This tells the resource:

“I can’t fit the groups in the token, go query Microsoft Entra ID to figure out what groups this user actually belongs to.”

Authorization could now take hundreds of milliseconds or more since the groups were no longer embedded in the JWTs. We ended up seeing some pretty significant latencies to storage reads and writes.

Long story short, we had to engage several engineers across different product groups at Microsoft to try to get this resolved.

I believe a hotfix had to be released from one of the product groups to address this issue, but that was after weeks of sending/analyzing logs and lots of meetings.

Enter Unity Catalog

After years of wrestling with the limitations of POSIX ACLs, recursive permission headaches, and Azure authentication quirks, Unity Catalog came along and seriously simplified data lake security. Unity Catalog centralizes data governance, enabling fine-grained access controls and simplified permission management, all natively integrated with Databricks. This modern approach addresses many of the pain points from legacy architectures, making it far easier to secure sensitive data, manage user access, and ensure compliance across the organization. Unity Catalog basically proxies access to your data lake:

With Unity Catalog, organizations gain the ability to define access policies at the table, row, and column levels, which extends far beyond the limitations of traditional RWX permissions. Permissions can be managed through a unified interface, making onboarding simpler and reducing the risk of misconfiguration. Additionally, Unity Catalog enables auditing of data access and changes, supporting compliance and security investigations. Integration with existing identity providers ensures seamless management of users and groups. Here is a side by side comparison of POSIX style permissions and Unity Catalog:

With Unity Catalog, so many folks will be able to avoid the grief of managing ACLs in data lakes.

Conclusion

Managing data lakes has historically been a complex and error-prone process, fraught with technical limitations and operational headaches. 2019 Conner really wishes he had Unity Catalog 😂 Unity Catalog represents a significant leap forward, offering centralized, fine-grained, and auditable data governance that simplifies security management and helps organizations meet their compliance goals. By adopting modern tools like Unity Catalog, teams can spend less time fighting with permissions and more time unlocking the value of their data.

Thanks for reading! 😀

Disclaimer:
PI data, pipeline sensor data, and the operational use cases around them are areas I’m actively learning. There are experts at Plains and across the industry who know this inside out. I’m approaching this from a technical perspective, and it’s a complex topic. This blog is meant to capture ideas and lessons from others while helping me solidify my own understanding. I’ve done my best to ensure accuracy, but please treat this as a learning perspective rather than authoritative guidance.

👋 Introduction

Over the past few months, I’ve been diving into the world of pipeline operations and the rich data that comes from them. I have grown a huge appreciation for how complex these systems are. The sensors along these pipelines generate massive amounts of data, tracking flow, pressure, temperature, vibration, and more, every second of every day. Interpreting this data is far from straightforward and understanding how to extract meaningful insights is both fascinating and challenging.

I recently had some great conversations with Scott McKean on the complexities of operational anomaly detection in pipelines. We also have an incredibly knowledgeable OT team at Plains who have been patient in explaining topics like over/short, inhibits, triggers, and more. It has been and will continue to be a humbling learning experience.

This space sits at the intersection of engineering, operations, and data science. There are some interesting data and AI problems here and I’m excited to continue exploring this space and sharing insights along the way.

🧩 Anomaly Detection Complexity

There is a lot that happens on an oil and gas pipeline, and it is far more complicated than just monitoring a few pressure and flow readings. Pipeline data is never static, and the pipeline system is constantly responding to external factors like temperature and pressure changes, and various operational decisions. So, while anomaly detection in many industries looks for outliers, in pipelines the definition of “normal” is always shifting. Sifting through ‘noise’ (weak outliers) and ‘anomalies’ (strong outliers) can be very difficult.

From what I have learned thus far, here are a few of the major factors that make the task of detecting anomalies so difficult.

Side note:
Scott has a great blog on anomaly detection and is definitely worth checking out.

🌡️ The Pipeline System is Always Changing

Hydrocarbons inside a pipeline expand and contract with temperature and pressure, and this stored compression is known as linepack. When a pipeline shuts down, pumps stop, pressure bleeds off, and the product settles. The measured volume can suddenly look different from what the system says should be there. This temporary difference is often called an over/short imbalance, and it can look almost identical to other operational issues like leaks or metering errors. Without accounting for linepack and how pressure and temperature shifts affect volume, even a good anomaly detection model will throw constant false positives.

This is why basic volume imbalance checks are unreliable the moment temperature swings, shutdowns, or other transient events happen.

📘 Why Linepack and Transients Break Simple Imbalance Detectors

Pipelines do not behave like static tanks. They act more like long, flexible storage vessels. As pressure increases, the pipe walls and the product inside compress, storing additional volume as linepack. During real operations, this is always changing as pumps ramp, temperatures shift, valves move, and product batches travel down the line. Because of this, the actual volume inside a pipeline at any moment is not fixed and can’t be inferred from flow meters alone.

Transient events make this even more obvious. Startup, shutdown, pump trips, valve changes, and temperature swings all shift linepack faster than simple models expect. A basic imbalance detector that just looks at flow in versus flow out ends up interpreting these changes as apparent losses or gains. To a naive model, those look like leaks. In reality, they are perfectly normal behaviors. This is why modern leak detection systems and anomaly detection approaches need to understand operating context, pressure and temperature behavior, and the physical dynamics of the pipeline rather than relying on static thresholds.

🔀 Multiple Pumps and Split Flows Add Signal Complexity

Pipelines are not linear “point A to point B” systems, and they often include multiple pumping stations, branch lines that split the product into different paths and potential recombination points downstream.

Each pump also introduces changes in the pipeline like pressure, flow, vibrations and transient behavior like shutdowns.

If flow in ≠ flow out → anomaly will generate a false positive every time.
These pipeline systems are incredibly complex.

🔔 Triggers and Inhibits Make the Operational Context Dynamic

Pipeline safety systems rely on automated logic and there are a few terms I have learned about in the past few weeks:

• Triggers: Conditions that automatically generate alerts to the control center for operator review and decision-making (e.g., high pressure)
• Inhibits: Temporary overrides that prevent those alerts when abnormal readings are expected (e.g., during startup or maintenance)

During startup, shutdown, pigging, and maintenance, inhibits are often active and from an operations standpoint we are saying ‘Yes, conditions look abnormal, but we expect that, so don’t respond.’

For anomaly detection models, this is incredibly hard. The model must also know when to ignore anomalies (i.e., the noise). This means the key is not just analyzing the sensor values but understanding the operating state at that moment.

💸 False Positives Can be Expensive

If an anomaly detection system signals a leak incorrectly and triggers or pressures teams to shut down, and if the system cries wolf too often, it simply gets ignored, which is worse than having no system at all. An incorrect flag could lead teams to shut down a pipeline segment:

• Product stops moving across large distances
• Field crews may be dispatched to remote terrain
• Downstream contracts and storage logistics are interrupted
• The cost can escalate into hundreds of thousands of dollars per hour

🛠️ Our Initial Approach

Before having a good understanding of all the above factors, we originally started tackling this as an unsupervised learning problem, specifically leveraging isolation forest and this seemed very appealing since we did not need to label events, and it did a decent job at detecting anomalies. We ran into some stark limitations right off the bat. Isolation forest only tells you that an anomaly occurred, not what changed or what the anomaly could be. It produces an anomaly score, but it doesn’t reveal which signal, feature, or condition triggered that score.

Isolation forest is not great since all you get is if an anomaly is detected or not and not what was detected. Being able to explain to the operational teams ‘why’ an alert was triggered is important to build trust and continuously improve the model through the practice of MLOps.

📐 Recognizing the Multi-Dimensional Nature of the Problem

The next thing I learned came from taking a step back and looking at the data from a physical standpoint.

A single flow rate or pressure signal cannot tell the whole story. Pipelines are multi-dimensional systems that are influenced by many variables including things like flow rate (volume movement), pressure (linepack, pump load, blockages, restrictions), vibration (pump condition and mechanical signatures), amongst other things. The behavior of any one signal only makes sense in context with the others. After understanding this, we moved away from isolation forest to an ensemble model to help us tackle the multi-dimensional nature of this problem.

📊 Shifting to Classification Models (e.g., XGBoost)

Once we understood the operating patterns, we moved toward supervised and semi-supervised classification, where the model learns to identify operating states and anomaly types, not just differences. This is where classification models (like XGBoost) became a much better fit.

Instead of asking, “Is this point different from normal?” (Isolation Forest), we instead ask, does this data pattern match one of the following:

• normal operation,
• shutdown conditions, or
• a potential leak?

This framing is much more aligned with how pipeline operators think.

🧪 Using Synthetic Data to Train the Classifier

To wrap my head around how to technically do this, I thought of generating some synthetic pipeline data to learn how to do this.

We generated these states by sampling from realistic ranges of flow rate, pressure, temperature, vibration, pump speed, and valve position, each representing a different part of pipeline behavior.

We validated how well (or not) the feature engineering went by leveraging EDA techniques to understand the relationship between the features. As I understand it, the less overlap between features, the better. In the below example, you can see some plot overlaps between normal and shutdown which could lead to false positives. Also, in retrospect, I wish I did not display balanced (i.e., normal) flow in red 😂.

⏭️ Next Steps

The ML model is just one piece of broader operational architecture. The output needs to be trusted, repeatable, and actionable across hundreds of pipeline systems with different equipment, environmental conditions, and product characteristics.

We’re approaching the next stage in three main areas:

🤝 Pairing ML Models with a Rule-Based Engine

While the classification model helps interpret multi-sensor patterns, pipeline operations still rely on deterministic safety logic that cannot be replaced. So instead of choosing between ML or rules, the goal is to combine them:

This approach reduces false positives while maintaining operational trust, something purely ML-driven anomaly detection rarely gets right on the first try.

🗂️ Model Governance and Lifecycle Management

To move beyond proof-of-concept, we need consistent versioning, reproducibility, and centralized governance:

🚦 Data & Pipeline Orchestration at Scale

The data pipeline itself matters as much as the model:

• We ingest PI tag data (and related historian telemetry).
• We cleanse, align timestamps, and compute features.
• We classify operating state + evaluate for anomalies.
• We push outputs to dashboards and alerting systems.

To manage this at operational scale, we are evaluating Lakeflow Declarative Pipelines to orchestrate:

Ingestion → Tag Aggregation → Feature Computation → Model Inference → Alert Routing

And last, but certainly not least, continue working with our OT teams to encode operational playbooks into the context layer.

Thanks for reading 😊

The amount of adoption and hype around Model Context Protocol (MCP) has pleasantly surprised me lately. While I was skeptical at the start regarding its value proposition and how widely (or not) it would be adopted, I’m now seeing how it’s coming together.

The most important aspect of agents are their tools, as they give them access to your enterprise context as well as the outside world. I’m starting to see how MCP will enable more seamless agent integration with the outside world.

At Plains, we’ve fully embraced the Lakehouse architecture, and the possibilities expand dramatically when you have data from your critical applications centralized in one place. This presents some fascinating opportunities for MCP, which I’ll explore in detail below.

Databricks has been releasing lots of exciting features during their ‘Week of Agents’ campaign and I wanted to cover their recent release around the MCP catalog and marketplace. I wanted to put together a short blog post on how to get started and where I see some opportunities.

📢 Announcements

There were a few exciting announcements around MCP during this release, but it really boils down to discovery, governance, and actionability of MCP endpoints. On the discovery side, you can now explore external MCP servers via the marketplace:

There are some neat MCP integrations to S&P and Nasdaq and there are a few free ones to get started with. I will cover the easiest way to explore external MCP shortly.

You can also more easily discover MCP servers hosted within your Databricks environment. If you think in terms of data products, discoverability is an important attribute of them, and I see MCP servers becoming products within your data and AI platform that will start to take on a lifecycle of their own and enable agents within your environment to interact with your enterprise data.

You can now explore MCP servers within your environment in the ‘Agents’ tab:

Lastly, the Multi-Agent supervisor now supports connecting to external MCP servers, giving agents access to AI-ready external data. I fully expect the offerings in this catalog to expand in the future.

🚀 Getting Started

I would say the easiest way to get started with testing some of these new features is to select one of the free external MCP servers. I suggest the Tavily MCP that allows agents to hook into the web.

1. Navigate to Tavily and create an account. You get 1000 free API calls a month to test out the functionality!

2. Next, navigate to the Databricks Marketplace and find the Tavily MCP server and select ‘Install’.

   

3. Finally, place your API key from Tavily into the bearer token input and click ‘Install’.

   

4. You should now be able to see your newly created Tavily MCP server in the catalog:

   

5. Now the fun part begins and we can create a multi-agent supervisor and give it access to the new MCP server:

   

6. Asking the agent in the playground, what tools do you have access to? is a good first test to make sure the agent has access to be able to interact with the Tavily MCP server:

   

7. Let’s give the agent a run for its money and ask a tough question:

   For companies in the midstream oil & gas sector, what are the top three real-world use cases where multi-agent systems have reached production? Include provider names, architecture patterns, and ROI details if available.

   The agent seems to spin for about a minute, but no answer is returned:

   

   The trace also does not seem to indicate that the Tavily MCP server was called; however, I can see the calls on my Tavily subscription.

   

8. Let’s try another question!

   How does the Model Context Protocol (MCP) differ from OpenAI Function Calling?

   This one worked really well. I can see in the MLflow trace that the Tavily MCP server was called multiple times to generate a file response:

   

   

   I am also impressed with the answer. For reference, you can view the agents response to my second question and see how it did.

Note: I have converted the response to markdown for better readability.

💡 Use Case(s)

I’ll admit, some of the MCP use cases in the market right now still feel a bit aspirational. But I think that’s the point. The value comes before the use case. Getting ahead of the curve on how systems will be accessed by agents and ensuring your data is structured, cataloged, and centralized is what positions you to move fast when the patterns become clear.

Instead of waiting for the “perfect” GenAI use case to fall into our lap, the strategy is to get the prerequisites in place now:

• Data centralized
• Metadata and governance
• Services exposed consistently through something like MCP

Then iterate on use cases as the opportunities show up.

One area where this does feel tangible today is with the sensor data we collect from our pipelines. We are already well underway on a build out of an anomaly detection layer (both a rules-based engine and an ML classifier, e.g., XGBoost). But once an anomaly is flagged, an operator still has to go and validate whether it is real or explainable. This is a highly manual workflow that often requires switching between SCADA, historian data, Maximo, and inspection/maintenance logs.

An MCP-connected agent could replicate the exact steps an experienced operator takes to triage the anomaly but do it in seconds instead of minutes or hours.

The Agent’s Workflow

Once an anomaly is detected, the agent would:

1. Automatically trigger from the alert
   • Input comes from the classifier/rule engine
   • Pass along signal metadata (location, severity, timestamp, sensor type)
2. Check maintenance history via Maximo
   • “Is there a work order already open for the affected equipment?”
   • If yes → annotate the alert as operationally explained
3. Query SCADA / historian for recent operating patterns
   • Compare pressures/flows/temperatures before and after anomaly
   • Detect step changes vs gradual drift vs sensor noise patterns
4. Look at nearby related assets
   • Sometimes upstream equipment explains downstream anomalies
   • Check if other components in the same block also changed state
5. Retrieve relevant inspection or ILI records
   • Determine if this area has prior corrosion, fatigue, or coating damage history

6. Return a structured human friendly report

   Example output:

   Anomaly at Station 12 Compressor Train B. No active work order. Upstream suction pressure dropped 8% at the same timestamp, possibly due to upstream supply swings. Last ILI shows no integrity red flags. Recommend monitoring; no immediate action required.

The cool part with this potential use case is MCP standardizes how agents access enterprise systems, and with MCP servers discoverable via the MCP catalog, other agent builders can tie into existing functionality.

🎯 Conclusion

The introduction of the MCP catalog and marketplace represents a significant step forward in making enterprise AI agents more accessible and operational. By providing a centralized hub for discovering, managing, and utilizing MCP servers, Databricks is addressing key challenges in enterprise agent development. As the ecosystem grows and more organizations contribute their MCP servers, we’ll likely see an acceleration in the adoption of agent-based solutions across industries.

It would be really neat if eventually these MCP servers were also exposed in Databricks One for the business to discover agents and the tools they can access.

These features are still in beta, so it is fully expected to run into the issue we saw in the ‘Getting Started’ section where the agent did not reply at all. I am excited to see where this goes next.

Thanks for reading! 🙂

I have found myself in a rollercoaster of emotions over the past two years during this AI Hype cycle. There is a visual model called the ‘Gartner Hype Cycle’ that has been catching on, namely in the AI community. It articulates the various stages of hype as a given technology matures.

I have been guilty in the past of riding the hype cycle and inflating my expectations of what the technology can do.

I have found myself in the ‘trough of disillusionment’ a few times over the past year, especially as I have tried to roll out customer-facing agents and have run into a whole host of issues getting them into production.

Is it because the technology is limited? Or is it because I have not figured out the best practices to follow to implement this in an optimal way?

I think as we near the ‘Plateau of Productivity’ the industry will start figuring out the ‘optimal way’ to leverage this technology going forward.

In this blog, I wanted to share where I am deriving the most value from Generative AI and the use cases we have been working on.

🏔️ The Generative AI Value Pyramid

Databricks has a great visual to articulate the value pyramid for generative AI:

Many teams (and I am totally guilty of this too!), start by building fancy UI’s and chatbots first to start leveraging generative AI.

I have made the joke in the past to friends and fellow colleagues that if I have to build one more chatbot, I may go crazy 🤪.

You will notice in the pyramid visual above, Databricks mentions that ‘batch inference delivers real value, fast!’

But what is batch inference?

🤷‍♂️ What is Batch Inference?

Batch inference is when you run an AI (LLM) or machine learning model on a large set of data all at once instead of processing it piece by piece. This usually happens on a schedule, like a nightly job, using automated workflows or pipelines. The model processes the data, and the results, such as predictions, categories, or summaries, are saved back into a data system where they can be used for reports, applications, or decision making.

In practice, this means you’re enhancing your existing ETL or data engineering pipelines by adding AI as another transformation step. So instead of just cleaning, joining, and reshaping data, you can now enrich it with intelligence, like classifying documents, detecting anomalies, or generating insights at scale.

🧱 How Does Batch Inference Work in Databricks?

Databricks has the concept of AI Functions, and AI Functions allow you to ‘apply AI, like text translation or sentiment analysis, on your data that is stored on Databricks’.

The cool part here is that these functions are invokable via code and are easily integrated into your pipelines.

We will talk about some of the use cases I am seeing below, but let’s talk about concepts a bit more to make sure we have a good understanding of how they work.

Databricks exposes a number of task-specific AI functions, that can do a variety of things like analyze sentiment, classify input text according to labels you provide, amongst other things.

There is also a function called ai_query() which enables you to invoke models hosted in the Mosaic Model Serving platform. It basically calls LLM or ML models via their REST API’s.

This is incredibly powerful and we have used this to invoke foundational models, custom agents, and agents created via Agent Bricks.

Let’s talk about a few use cases we are working on to cement these concepts.

🤖 Use Cases

There are a few use cases we are working on, some are in production, while others are still in development.

🏷️ Classification

We are pulling incidents out of our EAM (Enterprise Asset Management System) into our Lakehouse. Inside the incidents table is a free text description field where operators in the Permian Basin manually record details whenever an incident occurs. These incidents can range from minor events, like a worker tripping on site, to more serious safety or equipment issues.

We applied the ai_classify() function to that free text description field to enable an LLM to categorize it. Our SQL function looked similar to the example below:

SELECT
    description,
    ai_classify(
        description,
        ARRAY(
            'personal safety',
            'equipment/mechanical failure',
            'process/operational upset',
            'environmental',
            'vehicle/transportation',
            'fire/explosion',
            'regulatory/compliance'
        )
    ) AS category
FROM
    incidents
LIMIT 10;

We then created a new column called ai_generated_category and worked with the end users to define what ‘good’ looked like.

This data was presented in a PowerBI report and provided efficiency gains when generating reports on incident trends at our facilities.

📝 Summarization

Plains Midstream is going through a large divestiture effort right now and there is a lot of data that needs to be analyzed in an incredibly short window of time.

Amongst other things, contracts dating back to the 80’s must be reviewed and bucketed into categories of ‘review’, ‘divest’, or ‘retain’.

There are several data points that need to be evaluated to make an informed choice as to what bucket to put that contract into.

For example:

• Does that contract contain a sold or retained asset?
• What legal entity is mentioned in the contract?
• Are there any clauses like right of first refusal or change of control defined in the contract?

To help with the analysis and indexing of 20,000+ contracts, we decided to leverage Agent Bricks Custom LLM agent.

Leveraging that agent allows us to guide it by specifying fields like:

• ‘Identify ROFR Clause if it exists. Make sure to identify the section where it exists.’
• ‘Identify if any sold assets are mentioned. Leverage the list below to identify the sold assets:’
• ‘Identify legal entity if it exists. Make sure to identify the section where it exists.’

Once the agent is created, we can invoke this ‘at scale’ across all 20,000+ contracts via a SQL statement like so:

SELECT
  file_name,
  ocr_text,
  ai_query(
    't2t-1234-endpoint',
    ocr_text,
    failOnError => false
  ).result AS ai_summary
FROM catalog.schema.table

This enables us to expose a summary of the contract so the lawyers can easily glance at the salient information for the divestiture.

We then leveraged Databricks Apps to create a custom front-end application where the lawyers could select the original PDF and contrast it with the AI-generated summary to ensure we are providing the source file that the LLM used to generate the summary.

🧩 Structuring Data

The last use case I want to talk about is adding structure to unstructured data. We leverage the AI extract function called ai_extract(). Providing OCR’d text to this function lets you extract structured fields from it.

We parsed this structured data downstream, wrote it to a delta table, and then set up a Genie Space where an LLM can write SQL queries for you based on natural language input.

🥳 Conclusion

Generative AI is still evolving, and while the hype is real, the most tangible value I’ve seen so far comes from integrating AI into existing data workflows, especially through batch inference. Whether it’s classifying incidents, summarizing contracts, or structuring messy data, the impact is clear when AI is applied thoughtfully and with the end user in mind. There’s still plenty to learn, but focusing on practical use cases is where the real progress happens.

Thanks for reading! 😀