Public Data Models: Better Than Anyone Expected

We have all heard the saying “garbage in = garbage out” when it comes to creating any sort of statistical model. You may have even seen it here. In an ideal world, we would be able to create forecasting models with perfect data quality, representing the exact details of each well stimulation, and the exact amounts of daily oil, gas, and water produced from the well.  We do not live in an ideal world, and must deal with missing or incorrect data points to create a forecast, no matter which method we choose.

Texas Railroad Commission Allocated Data: Garbage In, Garbage Out?

In this post, we will deal with one source of inaccurate training data – lease level allocation in the state of Texas. The state agency (Texas Railroad Commission) does not report the production of oil, gas, or water from each individual well.  The state does report the sum total oil and gas production for leases, which can consist of anywhere from 1 to dozens of wells. This reporting regulation creates a data science problem in and of itself, which data providers have attempted to solve with varying degrees of success.

Many customers and prospects have asked, “without knowing exactly how much each well produces, how can we train a model? Garbage in = garbage out, right?”  Similarly, we must ask the question, “without knowing exactly how much each well produces, how can we test  a model”?  Luckily, we have just the dataset to answer this question. Over time, our customers have uploaded daily production data for more than 10,000 wells. In the Midland Basin, we have a dataset of ~5,000 wells we can use to train and test models. 

Establishing a Public Training set and Private Test Set

In this experiment, we wanted to answer the question “how bad is the model trained on monthly allocated data compared to the model trained on private daily data”?  We then started on the path of splitting the model into a test and training set, and evaluating the error. However, a model’s ability to forecast an allocated guess of production is less important than its ability to predict the true production – something that can only be known from private data.

Below, we compare the forecasts from a Midland Basin public (allocated production) trained model to two different test sets:

1. Public allocated monthly production
2. Private daily production

The following image illustrates the two production profiles in question, as well as a forecast for a test set well. The forecast is much closer to the true private production profile than the public allocated profile of the same well, despite the forecast model being trained on a public allocated production dataset. 

Forecasting Results vs Private Test Set

To create our test set, we use backtesting, a method commonly used in meteorology, to create a test set defined by the age of the well, rather than a random assignment. Using this approach, we can ensure that no data leaks from the future into the past, and we can ask the question “how well would this model have performed if it was trained on Jan 1, 2019?”. In order to create this basin-wide training set, we applied some outlier removing filters: constraining the dataset to between 1000 and 3000 lbs/ft and gals/ft, only considering the three major zones Wolfcamp A, Wolfcamp B, and Lower Spraberry, and enforcing a lateral length between 3000 and 20000 feet. This led to an overall model dataset of 9,750 wells, and a training set of 5,193 wells which came online before Jan 1 2019, and 7,163 wells which came online before Jan 1 2020.

First we examine the results of our forecasting model on the public allocated actuals from post-2019. For this analysis, we only consider test set wells where we also have private actuals, so that our comparison in both cases is against the same set of wells. We see a median absolute percent error (MAPE) around 23% against the public allocated actuals, with a well count starting at 1,407 at day 90, dropping to 585 at day 660. However, the goal of our forecasting model is not to predict public allocated production – it is to predict the true oil production from  each unconventional well. When we compare the same forecast results to private daily actuals, we see MAPE around 19%, despite the training set not including any private daily data.

What explains this surprising result? How can a public model be better at forecasting a test set composed of private data it has never seen before than public allocated data, which is sourced from the same algorithm as its training set? 

One factor may be the smoother profiles of the private actuals, which can represent a more natural decline curve than allocated production. Another factor is that random forest models, such as the one used here, create weighted averages of the training set production profiles, and this averaging may balance out the error distributed across the public training set profiles. 

No matter the exact explanation for the superior performance of these forecasting models when judged against private data, it does raise the question of how to evaluate a model. If an operator had trained this model with no access to private data, they would be left assuming that the model is much less accurate than it truly is. They may even be forced to further iterate, or scrap machine learning efforts altogether. In this example, public well production data did not stop us from making a good model, but allocated production data may have prevented us from understanding that we had made a good model. 

Conclusions

• Texas Railroad Commission public production data can be used to create accurate machine learning models
• Without access to a large private dataset of true well production, it is difficult to assess exactly the accuracy of the model
• Models trained on public allocated data are better at forecasting private per-well actuals than they are at forecasting allocated profiles
• If you are using Texas public data to create a forecasting model, your model may be better than you realize.

AI-driven oil well drilling optimization workflow

Building machine learning models can be challenging and incredibly time-consuming. Luckily, there are new solutions that can help.

At Novi Labs, we bring you the best oil well drilling optimization tools and processes—all in one place. 

Unlike other traditional forecasting & type curve methods, our well planning software creates accurate forecasts for well drilling optimization designs, multiple cube designs and optimizes capital allocation.

You can book a call with one of our experts, or read more about the platform here. 

Kiran Sathaye

Technical Sales Advisor at Novi Labs