June 30th Weekend Payment Issues - Not our kind of weekend
As most of you know, we experienced some payment processing issues this weekend. While we will be finalising a deep analysis over the next few days, we wanted to explain what we already know.
What happened? On Saturday morning we were informed by one of our third-party payment processors that they are experiencing intermittent transaction processing issues. Internal checks showed that it only affected a small % of customers, and usually when trying again the payments went through.
Saturday 13:00 - as there was no improvement we updated our status page to ‘Investigating: Intermittent payment processing issues’.
Our payment processor communicated with us every 1-2 hours, assuring us that all their resources are focused on solving this issue as soon as possible.
These issues continued through Saturday. We increased the capacity of our customer experience weekend team so that we could answer tickets as quickly as possible.
On Sunday morning the issue continued and by mid-day seemed to be getting worse. We updated on our social channels and status page and began sending notifications to customers.
Sunday 14:20 the rate of errors improved dramatically and at 15:22 we were notified by our payment processor that the problem was intermittent latency within the authorisation processing flow, that a fix has been deployed and transaction processing is back to normal and stable.
Sunday 15:50 - we updated our status page to ‘Resolved’ and sent notifications to inform customers that payment processing is back to normal.
What was the impact on Curve customers? Some transactions were declined. Some transactions went through and were reversed causing confusion to our customers.
What have we got in place to mitigate these risks? Mastercard Stand-in Process - we have established a set-up for our debit programs by which if there is a disconnect between MasterCard and our payment processor, MasterCard will ’stand-in’ and will approve transactions based on a set of parameters. It means that in these cases we take the risk on ourselves and trust our customers that if needed we will be able to capture funds for outstanding balances at a later time. This process is in place and reduced the number of transactions declined.
Curve Stand-in Process - Similar to the above, but means that if there is a disconnect between Curve and our acquirer, we stand-in and approve the transaction, assuming the risk to ourselves. This process was not needed in this incident but has been very valuable in other cases such as the Visa downtime.
Status page - in this link: status.imaginecurve.com you can always check the status of our systems and register for email/ SMS updates.
Social media and notification updates - we do our best to keep our customers updated when something happens. There is a delicate balance when it comes to these incidents - do we inform all of our customers if only a small % are impacted? What do we say when we don’t know how long it will be?
What could we improve? We continue to work closely with our payment processor to ensure they put the right processes in place to avoid these incidents and mitigate the impact when they happen. We plan to introduce additional stand-in processes, for other parts of the process, which will further mitigate the impact of incidents.
While we communicated with our customers throughout this incident and received some positive feedback from customers we think we could have done a better job at getting the word out to the customers who needed to know so that they could carry a backup card or know that trying again sometimes worked. We will be reviewing our internal procedures so that we can handle communication for any future incident in a better way.
Will we build our own processor? Yes. At a later stage. We consider payment processing a core part of our business and would plan to build a system that serves our customers, supports our mission and provides us the independence to do so. However, there are significant risks in implementing that decision and unfortunately even when you have full control systems sometime fail (remember the Visa incident, right?).
The most substantial risk is shifting focus and resources away from our goal - a connected financial world.
At this stage in the journey we believe that our resources are better allocated to building the features and products that provide our customers with an added value, a superior customer experience and answers needs that other products do not. It’s a question we continuously discuss and hope you trust us during this journey.
We sincerely apologise for the inconvenience and confusion this has caused and will continue to work hard to meet the reliability and level of service we owe to our customers.
For any questions get in touch at firstname.lastname@example.org.