$12 Million In Duplicate Charges From Shell Oil Telco Crash
Written by Evan SchumanFebruary 2nd, 2011
On Saturday (Jan. 29), a telco outage at Shell Oil stations directly caused more than $12 million in duplicate charges for its retail customers. This is yet the latest case of a chain getting burned because the payment industry has no consistent way to deal with in-progress credit and debit charges when systems crash. Does store-and-forward need to be tweaked to be made more crash-proof?
First Data circulated a confidential memo on Monday (Jan. 31) that said it was reversing 401,120 transactions—totaling $12,135,608.19—of Shell's retail transactions from the weekend. First Data was asked to "post the reversals to the cardholder accounts impacted as soon as possible in order to limit negative impact." (Guess "negative impact" means customers screaming at call center reps.)
This Story Is Only Available For Premium Subscribers. Click Or Login In Below To Read The Rest Of This Story.
Already a Subscriber? Login Here
Pages: 1 2
5 Comments | Read $12 Million In Duplicate Charges From Shell Oil Telco Crash
Leave a Reply
Readers, specifically those who want to comment on a story:
Our Comment SPAM system is getting very aggressive these days and has been blocking legitimate comments. If you post a comment and don't see it appear within 2 hours or so, can you please send a heads-up to customer-service@storefrontbacktalk.com? Ideally, please include the time you posted the comment. That will allow us to try and hunt for it. Thanks! P.S. We're working on fixing the system, but we don't want to lose any valuable comments in the meantime.
Our Comment SPAM system is getting very aggressive these days and has been blocking legitimate comments. If you post a comment and don't see it appear within 2 hours or so, can you please send a heads-up to customer-service@storefrontbacktalk.com? Ideally, please include the time you posted the comment. That will allow us to try and hunt for it. Thanks! P.S. We're working on fixing the system, but we don't want to lose any valuable comments in the meantime.
-Christine
February 3rd, 2011 at 10:39 am
Amazing that people and other businesses never pass on an opportunity to take a shot at AT&T. This problem was caused not by the Telco outage, but rather the flawed First Data Software code. Network outages occur, it is a fact of life in a wired network there will always be outages. Apparently Shell and First Data did not contemplate that an outage might occur and what would happen if it did.
February 3rd, 2011 at 12:42 pm
The thing that is so crazy about this story is that it is so simple to prevent. Every transaction initiated requires a unique ID. Whether the ID is explicitly stated or calculated by hashing values in the transaction, the ID provides a means for screening duplicates.
But this must be an age old problem, because I can remember the VP of Tech Services reviewing transaction dumps to manually screen duplicates. We operated our own “switch” and routed the credit card transactions coming in from the stores to the providers. I never got an explanation why this couldn’t be programmed back then, but apparently it still exists.
February 3rd, 2011 at 1:48 pm
In dealing with issues like this for over 20 years and designing systems and subsystems to prevent or minimize issue like this, I don’t blame any of the parties mentioned in the article. Instead, I blame a fundamental flaw in the core authorization/clearing systems run by the card brands. The one piece missing is an end-to-end Globally Unique ID (GUID) assigned to the transaction at the point of origin. This point of origin GUID assignment should be as close to the point of entry as possible (or point of initiation in the event of recurring transaction).
Right now there is no such capability and without it, you can write all the preventative code you want but at some point a decision needs to be made — error in favor of the cardholder, or error in favor of the merchant and arguments could be made for both. Adding this GUID would (or should) guarantee no double postings no matter how many times the transaction is retried or reposted.
As to the liability argument, good luck. The only winners will be the lawyers. Even SLA’s won’t stop the finger pointing: the POS vendor should have coded for this, the merchant should have audited prior to settlement, a velocity threshold should have alerted First Data, etc., the possibilities are endless.
February 3rd, 2011 at 6:59 pm
This also happened on Friday January 28 in the afternoon as I experienced the double bill from a Shell gas transaction around 4:30pm CST 01-28-11.
March 23rd, 2011 at 11:18 pm
Absence of uniqueness has been a huge question mark & I thought that this has been a black box to me as I did not get clear answers. Thanks to Steve Sommers on throwing light. There are many ways by which this can be achieved. As we move towards chip & pin transactions, one quick suggestion could be card number combined with card transaction counter (CTC). Would ISO include this or other as an enhancement to 8583? This uniqueness will make a lot of sense to payment industry.