How to Build a Physically Sensible Model in a World of Biased Data: Double Machine Learning

Note: This is the 3rd of a multi-part series on Novi’s approach to creating forecasting models which respond accurately and sensibly to downspacing and parent-child depletion.

In the previous posts, we discussed how confounding factors can make it difficult to create a useful model. Machine learning models can accurately interpolate over values that are present in the dataset, but because the dataset has not been evenly sampled, standard machine learning models may produce results which do not honor physical intuition. These modes will struggle to disentangle the impacts of highly correlated input variables, such as spacing and geology.

The Science Behind Isolating Variables

The best way to determine the impact of a single variable is to change that variable while holding others constant. In science, this is called an experiment. When outside factors are not controllable, you can instead randomly assign values of your interventional variables and hope that the impacts of the other features “average out” during the analysis. This is called a randomized controlled trial (RCT) RCTs are the standard in medical research.

However, it is sometimes not possible to run an experiment or controlled trial for financial, ethical, or physical reasons. In these cases, practitioners turn to a field called causal inference to estimate what an RCT would have revealed if it was possible to conduct one. In the second blog post, we saw one of these strategies, known as stratification. Separating data according to the levels of a confounding variable lets you see the impact of your variable of interest, holding the other variables constant. As a reservoir engineer, you probably already do this, identifying wells in a similar geological area and examining well productivity across different spacing and timing characteristics.

Stratification and Simpson’s Paradox

The observation that correlations can flip directions based on aggregation/disaggregation is known as Simpson’s paradox, and is resolved with prior knowledge of the causal relationship. In our case, our prior knowledge allows us to inform the model of intuitive directional relationships between parent-child and interwell spacing on horizontal well production.

However, stratification creates two additional questions for our use case. First, we need to prescribe in advance how many bins to use, and where the boundaries of each bin should be. Relying on quantiles is ineffective for non-uniform distributions, as we often see in well spacing and completion designs. On the other hand, relying on domain knowledge introduces further questions of its own. Should distance bins be the same for the Midland and the Eagle Ford with its extensively downspaced pad designs?

Second, we need to consider how many variables to include. If we leave out potential confounders, then we run the risk of incorrectly assigning impact to our variable of interest (or of missing that impact unfairly). However, as we include more variables, we increase the likelihood of any individual bin becoming small and providing unreliable estimates of the correctly de-confounded impact. 

Instead of binning across several continuous variables simultaneously, we would like to use our confounding variables directly to de-bias the relationships of interest. For the causal relationship between well spacing and oil production, I would like to have a way to remove the influence of rock quality from the amount of oil produced, and from the operator’s decision about how to space wells. Any relationship left between well spacing and production then could not be explained by the confounder of rock quality.

The Solution: Double Machine Learning (DML)

Here’s another way to think about this. The goal of stratification was to remove the impact of one variable on an outcome, while allowing another variable to still influence that outcome. But it’s not the only way to do this. If you have a prior expectation about how much oil a typical well will produce in your acreage, and you have a prior expectation about how operators space wells in your acreage, you can look for pads with unusual layouts and ask yourself if the wells on that pad are producing more or less than you expect.

Of course, we don’t know in advance how much a well will produce – that was the point of training the model in the first place. We also don’t know rock quality in advance – we construct our acreage tiers based on the importance our models ascribe to geology after they have been trained on a dataset. We have to train models so that we can train other models.

Specifically, we train three of them.

1. A model that attempts to forecast oil production using only geological information. If we don’t have this as an input feature from another model, we can provide a set of geological variables directly and let the model figure out where the good and bad parts of the basin are. We will then generate a forecast from this model, and calculate the model errors. These errors represent everything about oil production that we cannot not predict only from knowing rock quality.
2. A model that predicts pad layouts based on rock quality. Again, if we don’t have this as an available feature, we can use the geological characteristics of the basin directly to do this. This model can be something simple, like predicting wells per section, or complex and also predict things like number of parents and parent age. We will again generate a prediction from this model and calculate how wrong it was based on a set of known pads. The leftover signal here indicates which wells have unexpected spacing layouts relative to their location in the basin.

What we have done here is create two new variables – one representing unexpected production volumes and one representing unusual pad designs. Another way to think about this is that we have removed the shared information between spacing features in the dataset and our measurement of rock quality. We can now fit the final model:

• A model that learns some sort of mapping from unexpectedly narrow spacing to unexpected production shortfalls. This is known as Double Machine Learning, or DML.

However, this mapping will give us one function representing every well in the basin. It would predict that every well would experience the same decrease in expected oil production due to downspacing. This is better than where we were before, but not good enough. The amount of degradation experienced by the well should depend on how much communication there is with its neighbors, which is a function of distance, but also a function of geological parameters and completion intensity. 

Instead, what we would like to see is a mapping from interwell spacing to production change that is conditioned on other features in the dataset. But now we are back in the same situation we had before. How many variables do we use? How many bins should each variable have?

To accomplish this, we change the regression tree algorithm itself. Instead of choosing optimal splits by minimizing the summed differences between the mean of each node and its membership, our generalized random forest method allows the substitution of a different loss to consider for each leaf. In particular, we will be using the gradients between our input features (unusually-spaced wells) and output targets (unusual production values). We want the nodes, after splitting, to have two lines that fit the data inside their respective nodes better than the single line in the parent node did.

In an individual tree, this results in a gradient calculated from a subset of the dataset for each predicted well. Across the trees in the forest, this generates a gradient for each predicted well, from each well in the training data, but where those training wells have been weighted according to how likely they are to produce a similar response to changes in spacing. In practice, we find that this operation tends to base its weighting mostly on the completion intensity of the wells in question.

Conclusion: Key Takeaways for Causal Modeling

In summary, the causal model approach takes advantage of several facts: 

1. We know the general direction of the impact of spacing and timing on oil production
2. We have algorithms that can train predictive models
3. We can train models to quantify the bias in the dataset
4. We can use outputs of successive models to train a final model which understands the bias problem (double machine learning)

In our next post, we will examine results on forecasted pads in the Delaware basin, varying the spacing and timing of developments with forecasts generated by causal and purely associative randomized tree models. 

For more information about these methods, please see the following:

• https://arxiv.org/abs/1608.00060

• https://arxiv.org/abs/1610.01271

Key Takeaways

• Standard machine learning models can interpolate accurately but still violate physical intuition when datasets are unevenly sampled and features are highly correlated.
• Stratification isolates a variable’s impact by holding confounders constant, but correlation direction can flip under aggregation due to Simpson’s paradox.
• Stratification requires predefining bin counts and boundaries, and quantile binning performs poorly for non-uniform distributions such as well spacing.
• Adding more confounders in stratification increases de-biasing fidelity but can create small bins that yield unreliable impact estimates.
• Double Machine Learning (DML) de-biases spacing effects by modeling production-from-geology and pad-layout-from-geology, then regressing residual production on residual spacing.
• DML produces basin-wide average spacing impacts unless the final model conditions responses on features like geology and completion intensity.
• A generalized random forest can estimate conditional spacing impacts by splitting on a loss based on gradients between unusually spaced wells and unusual production values.
• Novi reports the generalized random forest weighting often depends primarily on completion intensity when identifying wells with similar spacing-response behavior.