Building Unbiased Benchmarks with Machine Learning Oil and Gas Modeling: URTeC 2020 Novi Paper Summary

90% of American drivers say they’re better than average, and 90% of Shale focused Operators have “peer-leading” breakevens, returns and well production forecasts. Suuuure they do! How do you cut through Investor Relations fluff to identify top-performing operators to learn from, or underperformers to acquire? We will use machine learning to build unbiased benchmarks for oil and gas financial modeling.

Simply looking at raw production numbers without the context of acreage quality or some other measure of expectation isn’t enough; the operator getting average production with some of the worst rock in the basin may have more to teach you than the operator with top 90-day cums that stumbled into the Core of the Core with a legacy minerals position that was ignored from 1985 through 2014.

Establishing a proper baseline is the subject of Novi’s upcoming URTeC 2020 paper, Benchmarking Operator Performance in the Williston Basin using a Predictive Machine Learning Model. In that paper, we propose two different, unbiased benchmarks that come out of our machine learning oil and gas models. First, the model’s estimate of the total rock quality (we call this geoSHAP). Second, we look for consistent over or underperformance relative to our model’s predictions — using the “known unknowns” of the model to our advantage, identifying who’s doing well on the decisions (such as choke management or lift type) unknown to the oil and gas model.

Oil & Gas Model Benchmark 1: geoSHAP

Our machine learning models learn what geologic variables lead to high production, just like they learn from completions or spacing parameters. To train this model, we used stratigraphic tops, log measurements, and mudlog geochemistry from the NDIC to generate hundreds of derived geologic properties, which we then ran through a principal components analysis to reduce down to five features that were fed to the model.

GeoSHAP does a good job predicting actual production. Most operators follow a simple trend of increasing average production as their average geoSHAP increases. A few small operators (<5 wells) deviate from the trend, which can be attributable to natural well variation over small sample sizes, but one large operator stands head and shoulders over the rest with similar rock quality: EOG.

Digging deeper into EOG’s outperformance in the Bakken

Let’s dig a little deeper into EOG’s outperformance by looking at how it changes through time. In 2013, EOG was crushing the competition, producing several times per well higher than their “peers.” In 2015, they fall back to earth as they go to lower-quality rock, and by 2017, the rest of the pack has caught up with them.

We can see how their drilled rock quality changes through time. In 2013, they were drilling some of the highest-quality rock in the basin in the Parshall-Sanish area and along the east side of the Nesson Anticline. As the years went on, EOG moved to lower-quality rock outside the Parshall core.

The other component of EOG’s stellar performance? Large jobs. EOG deployed completions with high proppant loading way before their competitors in the basin, putting away over 1300 #/ft on average in 2013 — much higher than most Williston operators use, even today.

Unbiased Benchmark 2: Model Prediction & Marathon Example

Proppant and geology are known to the oil and gas model — so if those are the main drivers of EOG’s outperformance, then EOG’s actual production should be close to the model prediction– and that’s exactly what we found.

Marathon, by contrast, falls near the bottom of their peers in actual production compared to model prediction. Here, we’ll use the model’s known unknowns to our advantage. As a reminder, the model has seen the gross completions variables (proppant volume, fluid volume, stage count), a range of geology variables, and spacing at the time of frac.

What’s not known to the oil and gas financial model? Post-frac operational choices (choke management, lift type), private completions choices (diverter, frac rate, etc.), and unknowable geological complexity that should average out over a large enough well count (natural fractures, small-scale stratigraphic variations, etc.).

Operational choices should have a changing impact over the life of the well — they should change the shape of the production curve. So, let’s look at how Marathon’s performance relative to model prediction varies through time.

Marathon starts off 22% above model prediction at IP30 and ends 29% below oil and gas modeling prediction for IP720 (both cum oil). As you can see from the above chart, the distance Marathon covers in actual-predicted space through time is larger than almost all other operators. Artificial lift type or choke management could definitely be the cause of this behavior — a spot check of the Brush 24-8H well (NDIC File #30550, shows similar behavior to the Marathon average) shows an IP test flowed at 55/64″ choke, and an ESP installed 10 weeks after the IP test–both possible contributors to high early-time production and low later-time production.

Conclusions: Use of Machine learning benchmarks for oil and gas financial modeling

• GeoSHAP captures the rock quality well and provides a strong unbiased benchmark for comparing operator performance for a range of tasks, including identifying underperformers for acquisiton
• EOG’s overperformance relative to geoSHAP is mostly a result of large completions designs
• Performance compared to the model’s prediction can benchmark operator performance on the known unknowns of the oil and gas valuation models: lift type, choke management, proprietary completions details
• Marathon’s early-time overperformance and late-time underperformance is likely the result of employing ESPs and/or large chokes

Have an operator to evaluate? Check their performance without using biased data. 

With our machine learning models, you can accurately pinpoint how operators compare and what drives their performance – above or below average. 

You can request a free demo here: novilabs.com/demo

Presentation Details ::

Benchmarking Operator Performance in the Williston Basin using a Predictive Machine Learning Model Control ID: 2750

URTeC 2020, Wednesday, July 22, Morning Session, Theme 11: Business of Unconventionals: Maximizing Value and Reliability II

Terms & meanings

Artificial lift type

This term is used in the oil and gas industry to describe the different methods used to increase the flow of liquids within a production well.

Usually, when a well is first drilled, there is sufficient pressure and volume to bring oil and gas to the surface, but eventually, the first production fades. That’s when artificial lift is typically installed to increase production rates. Some operators tend to install lift earlier than others, and some operators tend to install larger, more intense lifting mechanisms. Often, these are operators with a stronger time preference (goals such as minimizing payback months or maximizing NPV) 

Choke

In the oil and gas industry, chokes are a device used to regulate the flow of fluids created by wells.These are used to manage flow rates, especially early in the well life in order to manage reservoir pressure and performance. 

SHAP Values

SHAP Values (an acronym from SHapley Additive exPlanations) break down a prediction to show the impact of each feature. They use game theory mathematics, named after the Nobel Prize winner Lloyd Shapley

Ted Cross

Ted Cross is VP of Product Management at Novi, where he builds powerful machine learning data and analytics products to solve the toughest problems in unconventional oil and gas. Prior to Novi, Ted was a Senior Geologist with ConocoPhillips.