Are you ready for Black Friday?
It’s that time of year again when we start to see a lot of articles on the subject of Black Friday and key drivers towards readiness. This year, we have asked our development team for their top level thoughts. This isn’t a deep dive into the depths of load testing, but gentle technical reminders for good practice and preparedness. Potter Rafed, Senior Engineer, runs us through some key points to be vigilant about when preparing for a period of high-trading volume.
Monitoring & logging
Having robust monitoring and logging processes in place is something that doesn’t only apply to peak periods of high trading volume. Whilst the prevention of system failures should be our primary goal, retailers should be prepared for a time when they may occur.
Monitoring, logging and establishing observability practices, enables preparedness and can identify potential performance problems prior to events such as Black Friday. Get fully acquainted with your performance data.
Notifications
If we don’t know we have an issue we can’t really fix it, right? We need to ensure that the monitors we have in place can send alarms to flag and escalate problems should they occur. For periods of high trading volumes such as Black Friday or Christmas sales, we should tweak the alarm thresholds to match expected loads and prevent us being overwhelmed by false-positives.
We use DataDog which allows us to easily access and configure notifications. Build a regular/peak notification plan and don’t forget to include multiple points of contact.
Traceability
Usually an alert will be triggered by a particular event and having access to detailed traceability will be a great help. It is good practice to increase logging levels in key areas throughout these peak periods to ensure there is enough data for debugging potential problems.
Make sure to include a “Correlation ID” between service communications, so that a particular problem can be identified and traced back throughout the whole stack.
Composable integrations & search
Having adopted a composable commerce platform means that load testing over peak doesn’t only apply to internal systems. Integrations, such as search, also need to cope with the high trading volumes expected.
Working closely with integration partners constitutes a key part of any Black Friday plan. Providing them with the retailer’s expected load and collaborating with them whilst you run your own tests. The sooner these predictions are calculated and communicated, the more time the integration partners will have to run their tests.
Robust CI/CD
Having good monitoring and alerting allows the development team to diagnose and fix the issue, but if we can’t get that bugfix live, then the problem still persists. A robust Continuous Integration, Continuous Delivery (CI/CD) pipeline with automated checks and testing should be in place. This reduces the dependence on external resources and enables developers to quickly add a bugfix to production.
Having a good and fast CI/CD process can greatly help mitigate damage caused by an unexpected outage during a peak or business forecasting inaccuracy.
Collaboration
Still on the subject of expected loads, but with a more product based angle. Dr Sarah Gilchrist expands on consumer products goods (CPG) supply chain forecasting accuracy in the September 2022 Gartner article “Improve Forecast Accuracy by Capturing Key Demand Planning Inputs From Retailers” (06/09/22). Dr Gilchrist highlights collaborative planning and the importance of utilising information on promotional events and activity. It makes sense for goods and web traffic forecasting analysts to work closely with developers in preparing a website for peak flow.
“The collaborative planning of promotions with retailers is one of the biggest opportunities for improving a CPG company’s forecast accuracy.”
Dr Sarah Gilchrist, Director Analyst, Gartner
Damage control
The questions that must be answered in advance for any expected periods of high volume trading:
Can an issue and its impact be quickly identified?
Can the retailer respond in a timely manner?
Can a solution be formulated and deployed to resolve it?
Who needs to be made aware of the issue and at what point?
To summarise – a robust load tested system, together with a skilled and well prepared IT team are required to support a retail Black Friday event. A clear strategy in place and a solid set of forecasted, expected load data to work with are key to preparing that team.
With such a small window of opportunity we agree with Dr Gilchrist, collaboration is key across all areas of a business. Once Black Friday is over, the last big push for Christmas begins.