Overfitting and Train-Test Splits Explained for Finance Professionals

A model that scores brilliantly on the data it was built from can still be useless on next month's data. That gap is called overfitting, and the standard defence is to keep some data back for testing, using a train-test split and, often, cross-validation. This guide explains both ideas for finance professionals who review, commission or sign off models rather than write them. We use the scikit-learn documentation, version 1.9.1, as our source for how these ideas work in practice. If you are new to the wider subject, start with our overview of machine learning for accountants and come back here for this specific angle.

What overfitting means

The scikit-learn user guide (cross-validation chapter) puts it plainly: learning the parameters of a prediction function and testing it on the same data is a methodological mistake. A model that simply repeated the labels of the samples it had just seen would get a perfect score but would fail to predict anything useful on yet-unseen data. The guide calls this situation overfitting.

For finance teams the practical meaning is simple. A model judged on the data it learned from tells you how well it memorised, not how well it will work on new invoices, new customers or new periods.

The train-test split

The documentation says it is common practice, when running a supervised machine learning experiment, to hold out part of the available data as a test set. In scikit-learn the helper function for this is train_test_split, described as splitting arrays or matrices into random train and test subsets. Its documented behaviour includes:

• test_size: the proportion (a number between 0.0 and 1.0) or absolute number of rows in the test split. If neither test_size nor train_size is set, the test proportion is set to 0.25.
• shuffle: the data is shuffled before splitting by default. The page says that if shuffle is switched off, stratify must be None.
• random_state: controls the shuffling; passing a whole number gives reproducible output across calls.
• stratify: if set, the data is split in a stratified fashion using the supplied class labels.

Reproducibility is worth insisting on. If a colleague cannot re-run your split and get the same test rows, nobody can check the result.

An illustrative example

Illustrative example, with made-up round numbers: suppose a team builds a model to flag invoices likely to be paid late, using 1,000 historical invoices. They train on all 1,000 and report that the model gets 98 of every 100 right. That figure says little, because the model has already seen every one of those invoices. If instead they set aside 250 invoices, train on the other 750 and score the model only on the 250 it never saw, the result might be far lower, say 70 out of every 100. The second number is the honest one, and the large gap between the two is the signature of overfitting. These numbers are invented to show the pattern; they are not results from any real model.

Why a single test set is not always enough

The scikit-learn guide notes a second risk. If you keep adjusting settings until the score on the test set looks best, knowledge about the test set can leak into the model, and the evaluation metrics no longer report on generalisation performance. The traditional fix is a third slice of data, a validation set, used for tuning, with the test set kept for a final check. The guide adds the drawback: splitting into three sets reduces the data available for learning, and results can depend on the particular random choice of sets.

Cross-validation

Cross-validation is the documented answer to that drawback. A test set is still held out for final evaluation, but the separate validation set is no longer needed. In the basic approach, called k-fold cross-validation, the training set is split into k smaller sets. For each fold a model is trained on the other k minus one folds, then validated on the remaining part. The performance figure reported is the average of the values computed in the loop. The guide says this can be computationally expensive but does not waste much data.

The simplest way to use it is the cross_val_score helper. The documentation's example reports scores for each of five folds together with their mean and standard deviation. In our view, a reviewer should look at the spread across folds as well as the average, because widely swinging fold scores are a prompt to ask questions.

Finance data has an order

Much finance data is time-ordered: monthly ledgers, daily prices, quarterly forecasts. A random shuffle can put later periods in the training data and earlier periods in the test data. The scikit-learn documentation for TimeSeriesSplit describes it as a cross-validator for time-ordered data, in which, unlike standard KFold, training data always precedes test data, so the model is not trained on future observations and evaluated on past ones. In its time-series approach, each training set is made of earlier data and the following block is used for testing, so successive training sets are supersets of those before them.

Our suggestion: whenever a model is built on dated records, ask which splitting method was used and whether any test row is earlier than a training row.

Data leakage through preparation steps

The guide also warns that preprocessing, such as standardising columns or selecting features, should be learned from the training set and then applied to held-out data. If the scaling is calculated on all the data first, information from the test rows has already crept in. The documentation notes that a Pipeline makes it easier to compose these steps and provides this behaviour under cross-validation, so that they are learned from the training part of each fold only.

Reading training and validation scores

The documentation's section on validation curves gives a compact rule of thumb. If the training score and the validation score are both low, the estimator is underfitting. If the training score is high and the validation score is low, it is overfitting. Otherwise it is working well. The guide also describes a learning curve, which shows training and validation scores for varying numbers of training samples, as a way to judge whether more data would help. It explains the underlying idea through bias (an estimator's average error across different training sets) and variance (how sensitive the estimator is to changes in the training set); in its polynomial example, degree 1 is too simple (high bias) and degree 15 generalises poorly (high variance). In its underfitting and overfitting example, a linear function (polynomial of degree 1) is not sufficient to fit the training samples, which the page calls underfitting, while at higher degrees the model overfits, meaning it learns the noise of the training data.

Questions to ask when a model is presented

These are our suggestions for non-technical reviewers, not scikit-learn requirements:

• How much data was held back, and was it untouched until the end?
• Was the split reproducible, and is the method written down?
• Were the data time-ordered, and if so, did the test period come after the training period?
• Was any preparation step fitted on the full dataset before splitting?
• What were the training and validation scores, and how large is the gap?
• Does the team understand what the model should do when it is wrong, and who reviews those cases?

Teams that code their own analysis can see how these helpers fit into everyday analysis in our guide to Python for accountants, and the broader finance context is in machine learning in finance. Library names, defaults and version numbers change over time, so check the current scikit-learn documentation before relying on any detail here.

Learnsignal's offices are off Baggot Street in Dublin, and we offer in-person training from there. For current details, see our CPD pages.

Frequently asked questions

What is the difference between overfitting and underfitting?

According to scikit-learn's documentation, if both the training score and the validation score are low the estimator is underfitting. If the training score is high and the validation score is low it is overfitting.

What default test size does train_test_split use?

The documentation says that if neither test_size nor train_size is set, the test proportion is 0.25. Always state your split explicitly so others can reproduce it.

Why not just use a bigger training set?

More data can help reduce variance, but the scikit-learn guide cautions that you should collect more only if the true pattern is too complex for a lower-variance model. A learning curve helps judge whether more data would benefit the model.

Is cross-validation always better than a single split?

The guide presents it as a way to avoid wasting data on a fixed validation set, at the cost of more computation. A test set should still be held out for final evaluation, and for time-ordered data a time-aware method is needed.

Does a good test score guarantee the model will work next year?

No. A test score only describes performance on data like the held-out sample. If the underlying business changes, the model needs to be re-checked.

In short: never judge a model on the data it learned from, keep a test set untouched, respect time order, and read the gap between training and validation scores.