Measuring COVID-19’s Impact on Your Business

Introduction

Apart from the humanitarian and personal problems associated with the novel coronavirus COVID-19, many business leaders are wondering what impact the virus will have on their companies. How will sales be impacted? Do we need to slow production? If so, how much? There are simple approaches to this, like looking at year-over-year differences or building projections based on recent trends. While those methods are technically data-driven, there are much more rigorous ways of judging the impact of the virus on your business. Traditional projections do not account for many things, including recent industry trends that are not directly related to COVID-19. For example, a year-over-year analysis would not account for industrial growth over the last year or new competitors entering the market and may greatly over- or under-estimate how much your business has been impacted, thus leading to faulty decision making. Here we will walk through a more modern analytical process that will lead to more accurate, and therefore actionable, estimates.

Approach

Data Set

The dataset being used here is from Google Trends and takes the difference in search volume between cruise booking and cruise cancellation as a proxy for revenue in the cruise industry. This is not a perfect approximation, but it realistically reflects the directional effects of COVID-19 on the industry.

In a non-simulated application of this approach, one would have direct access to revenue, cruise booking and cruise cancellation data, but this data will serve well for an illustration. We need a substantial amount of data to build reliable forecasts, so we have pulled and re-weighted data from January of 2016 to date.

Method

We will not be focusing on specific techniques here, as the focus of this post is on practical application instead of theory or implementation. However, it is important to note that there are many machine learning and statistical forecasting methods (such as the ones used here) that are relatively accessible in common programming languages.

Our goal is to construct a counterfactual model, one that answers the question “What would this KPI look like if recent events had been normal?”. Once a reliable counterfactual is in place, we can subtract recent observations from the counterfactual. The result reflects the difference between what is actually happening and what would have happened during normal times – in essence, the impact that COVID-19 has had.

Results

Model Training and Evaluation

The most common way to test a time series model is via holdout. We will train the model on a certain amount of data at the beginning of our dataset and test the model’s accuracy on the remaining portion (the “holdout”). Since we will be using this model to make calculations over the last few months, a 90-day holdout is appropriate. We are training the model using data from January 2016 through September 2019 and testing its accuracy on the time period from October through December 2019.

A quick visual inspection indicates that the result looks very realistic:

We will use two measures of accuracy:

1. Mean Absolute Percent Error (MAPE): MAPE is a measure of how far off our projection is from reality on a daily basis. For example, a MAPE of 10% would indicate that, on an average day, our model would either over- or under-estimate the true value by 10%.
2. Total Error (TE): this is a measure of how inaccurate our projection is throughout the entire projection period. If the quarterly forecast calls for 1,000-unit sales and only 500 units were actually sold, the TE is 100%.

TE is going to be more important for us, given the task at hand. We are trying to measure the impact of an intervention over a long period of time. If our purpose was to accurately predict the exact impact that the virus will have on any given day, MAPE would be a more appropriate error metric. MAPE is still useful for choosing the most accurate model, as TE can be misleading.

There is no single rule-of-thumb for a “good” MAPE; accuracy in time series forecasting is entirely dependent on the dataset. In a real-world application, we would estimate several models and select the one with the lowest MAPE relative to the alternative models. Furthermore, our holdout TE is 8%, which tells us that over long stretches of time, the projections are very reliable.

Projections of COVID-19 Impact

The final part of this exercise is to compare our projection with the observed values. Remember that what we are calculating here is a proxy for cruise_tickets_booked – cruise_ticket_canceled:

Unsurprisingly, we see that net bookings are way below the counterfactual from January onward. The total difference between our projection and observations is -3,120 units from January 1, 2020 through March 16, 2020. For comparison, the total unit volume from the same time period in 2019 was +3,316.

Concluding Thoughts

This post is meant to be a high-level illustration of how to calculate the impact of a market intervention on a business or product and not an exhaustive demonstration of how a real-world project would be undertaken. There are a number of improvements that can be made:

• Including “covariates,” signals that tend to correlate with a specific business or product, can greatly improve model accuracy and help control more precisely for latent factors not directly associated with the intervention in question.
• Relatively little model tuning was performed here, but a large number of techniques and rigorous tuning are generally employed to build as accurate a projection as possible.
• Using “real” data instead of Google Trends search indexes would allow for greater and more actionable insights, including actual dollar-value assessments of lost revenue.
• Adding more data to a time series generally makes forecasts more accurate. We are working with data from the beginning of 2016, but the historical data goes back further.

A potential next step in this process, which we did not cover, is to build a forecast for the severity of the impact of COVID-19 from epidemiological equations and build that into the system. One can then project the future impact that the virus will have. This is a straightforward process once the modeling framework above has been built.

As a final note, there are many applications for this process outside of the rather unfortunate one we look at here. Business impacts from campaign launches, new competitors entering the market, changes in the price of raw materials, and so on can call be estimated in this way.

From all of us at Softcrylic, please stay safe and we wish all the best for you. If you have any questions about this or any of our other work, please do not hesitate to reach out!