Two Numbers Prove EMV Is Not The Silver Bullet for Credit Card Fraud

The compromise of tens of millions of credit cards at Target and other retailers has raised the awareness of the risks of credit card fraud. In the wake of this data breach, significant attention has been given to how a new credit card technology called EMV many have suggested will significantly reduce this kind of fraud. (In fairness, this technology is new only to the United States; it’s been widely used elsewhere since 2005.) The premise of this suggestion is that EMV cards are harder to use fraudulently because they contain a chip which is much more difficult to duplicate than the magnetic strip used in U.S. credit cards today. Unfortunately this premise is wrong.

While EMV is marginally better than what we have today, it does nothing to solve the fundamental problem that enables fraud in the first place – more on that in a moment. First, though, it’s important to understand two payments industry metrics:

  1. Card-not-present transactions represented 24% of all credit card transactions in 2012, representing a 15.1% compound annual growth rate (CAGR) since 2009 and a much faster increase compared to the 4.6% CAGR increase in card-present transactions. Card-not-present transactions are a broad category of transactions that basically boil down to the use of a credit card not involving a swipe: things like online purchases, purchases over the telephone, recurring bills to a credit card, etc. These are in contrast to card-present transactions where the card is physically swiped to make a purchase.
  2. More than 50% of fraudulent credit card transactions (by value) occurred through card-not-present transactions. The rate of card-present fraudulent transactions by value was 9.16 basis points in 2012, compared to 11.38 basis points for card-not-present transactions.

Source: 2013 Federal Reserve Payments Study

These numbers show that card-not-present transactions represent a significant portion of transactions in the non-cash payments ecosystem and are particularly susceptible to fraud. The introduction of EMV does virtually nothing to solve this problem because EMV only adds new protections against fraud for card-present transactions. Fraudsters can and will focus their efforts where there is the least friction to fraud: card-not-present transactions.

This highlights that the root of the credit card fraud problem has almost nothing to do with the physical plastic card that consumers carry around. The real problem is that those plastic cards represent payment instruments that are long-lived and difficult to change. The entire credit card ecosystem has been built around passing around 16 digit numbers that only change every few years. When those numbers are compromised, it is costly to the issuer and inconvenient to the customer to change to a new number. When credit cards were first invented there was no better solution available, so this solution made sense. In the intervening 55 years, technology has come a long way. We have lots of better solutions now, but they require merchants, issuers, and the card networks to change the way the payments system works.

If we could magically erase the last 55 years of knowledge about how to operate credit card systems and start over with modern technology, the solution the banking industry would invent would look nothing like what we currently have. Payment instruments would be short-lived tokens that would be useless if intercepted and virtually impossible to guess. The system would be designed to inherently prevent the storage of widely reusable payment information by third-parties, like when an online merchant stores your credit card number. Reuse of a compromised token would be immediately detected. We would not be cobbling together a standard like EMV that solved part of the problem but was completely hamstrung by the constraints in the current payment networks that caused the fraud problem in the first place.

If we want to address the fraud problem seriously, we need to be prepared for more significant change in the ecosystem. We have the technical capability. We just need to find the will.

Why Mobile Wallets Aren’t Mainstream… And Why They Will Be Soon (Part 2)

In part 1, I laid out the history of obstacles to mobile wallet adoption: complexity of the NFC Secure Element ecosystem, the expense and risk for retailers, and a “chicken and egg” problem between merchants and consumers.

… But All Those Challenges Have Virtually Disappeared

As insurmountable as those challenges may seem, they are all going away in the next year. Recent changes by card network rules that govern the liability of fraud and simplifications to the NFC ecosystem (led by Google) are eliminating these obstacles. It’s important to understand the nature of these shifts to appreciate how significant they are in setting the direction of mobile payments this year:

Retailers have a new reason to deploy NFC-capable terminals

To understand why this is requires a quick primer in the way credit card fraud liability works. (Note: the explanation that follows is somewhat simplified; there are exceptions and nuances to the way liability in credit card transactions is assigned. This is meant to provide a broad overview, not be a comprehensive explanation of the subject.)

Today, and for most of recent history, credit card transactions are generally grouped in to two types: card-present, like when you or a cashier swipe your card at a register, and card-not-present, like when you manually enter your card information on a web site or speak it over the telephone. When a fraudulent transaction happens for a card-present transaction, the bank is liable for the fraud and the merchant gets paid as if it was a legitimate transaction. When a fraudulent transaction happens for a card-not-present transaction, the merchant is liable for the fraud (i.e. the merchant doesn’t get paid, even if they shipped the goods for the transaction). These rules are set by the major credit card networks (Visa, Mastercard, Discover and American Express).

This is changing for most merchants in October 2015. The credit card networks have mandated that for merchants to continue not being liable for fraud from card-present transactions, the merchant has to support a new credit card standard called EMV. EMV, short for Europay, Mastercard and Visa, is a global standard that makes it harder to make fraudulent card-present transactions by embedding a chip in to the credit card. Supporting EMV credit cards requires an EMV capable credit card terminal that can take advantage of the additional security in the embedded chip in EMV credit cards.

As a result of this liability rule change, merchants have a significant financial incentive to deploy new credit card terminals. These new EMV terminals also support NFC. While October 2015 is still a ways away, merchants have already begun planning and executing their migrations to EMV terminals. Voila! As a result, most merchants will have NFC-capable terminals regardless of their philosophical beliefs about the NFC standard.

It’s likely that you’ve already come across some of these new NFC-enabled terminals. They typically have a MasterCard PayPass or Google Wallet logo. Some notable merchants that have started to deploy NFC include Toys “R” Us, American Eagle, Foot Locker, Macy’s, The Container Store, McDonald’s, and Jamba Juice. The fact that major national retailers have started to deploy NFC terminals combined with the liability shift will make the choice to support NFC easier for other retailers who have been waiting to make a decision.

The NFC ecosystem was recently simplified

With Google’s most recent release of the popular Android mobile operating system, Android 4.4 (“KitKat”) now supports a standard known as Host Card Emulation (HCE). Without getting in to the technical details, HCE has eliminated the mobile network-controlled Secure Element. This means that credit cards can now be used in NFC transactions without requiring technical changes, support or explicit approval by the mobile phone networks, the phone manufacturer, or the bank issuing the credit card. There are lots of details that are still being worked out here and it’s likely that the issuing banks will still play a role in this ecosystem. Nonetheless, a major technical obstacle for creating NFC wallets has been eliminated for Android devices. As Android’s adoption continues to increase and as more NFC-capable devices are sold in the U.S., this creates a critical mass of the consumer market that can now use this technology.

While this is great news for NFC, it doesn’t mean that non-NFC wallet solutions are dead. Starbucks and Square in particular have great market positions to lead the charge for non-NFC wallets. With the major structural obstacles removed from the mobile payment ecosystem it is likely we will see more innovation and there is still plenty of opportunity for multiple winners.

What This Means for Consumers

Why does any of this matter to the average consumer? Mobile wallets aren’t just a replacement for physical plastic credit cards. If that’s all a mobile wallet did, consumers would have no reason to change their behavior. Mobile wallets make the payment process better for consumers: wallets can reward loyalty (like the Starbucks app), make paying much easier (like the Square Wallet with auto-checkin), or help you make better decisions when you pay (like real-time budgeting feedback or rewards optimization).

It also means that things are going to get more complicated, at least for a while. There will be lots of mobile payment solutions vying for consumers’ attention. Ultimately, these solutions will be less about the plumbing of the payment and more about the value-added services that can be built on top of the payment value chain. This is not a winner-take-all proposition; there will be multiple “winners”. Until that consolidation happens, consumers may face a landscape of emerging wallet options to choose from.

Why Mobile Wallets Aren’t Mainstream… And Why They Will Be Soon (Part 1)

Screenshot_2013-11-20-23-47-30 image

The last few years have seen tremendous investments being made in mobile wallets, with everyone from well-established players like Google and Verizon to early-stage startups like LevelUp and Tabbedout getting in the game. With all this investment, some people have started wondering why mobile wallets haven’t gained more traction. Some pundits in the payments community have gone so far as to say that mobile wallets – or at least some forms of them – are dead.

I appreciate this perspective, but I also believe it to be totally wrong. Mobile wallets will gain traction in 2014. But to understand why, a little background on the obstacles to date is necessary.

Obstacles to Mobile Wallet Adoption

Understanding the challenges with mobile wallet adoption requires a little bit of background knowledge in the technology that makes mobile payments possible. Very broadly speaking, there are two types of mobile wallet solutions for making purchases:

Near Field Communication (NFC) Wallets

NFC is a set of technology standards that make it possible for devices – like a smartphone and a credit card terminal – to communicate wirelessly when they are in close proximity to one another. You can think of NFC as being similar in some ways to Bluetooth: there are lots of complicated technical details that dictate how it works, but the end result for consumers is you can buy any Bluetooth headset and pair it with your cell phone without worrying about the details. NFC makes it possible to securely exchange the same – and more – information as what is contained on your credit card’s magnetic stripe wirelessly over very short distances.

NFC is also like Bluetooth in the sense that there are a variety of ways it can be implemented. The two most popular ways that NFC has been used so far is with stickers that are essentially hard-coded to a single credit card and with smartphones using something called a Secure Element. The Secure Element is like a safe in your phone. Most applications can relatively freely access the memory and storage on your phone, but the Secure Element provides a heavily restricted area to store credit card account information. The Secure Element can only be accessed through trusted parties – more on this later.

In order for a retailer to accept a payment using NFC, they must have a credit card terminal that is NFC enabled and their point-of-sale systems must support NFC transactions. The terminals look a lot like the terminals you’re accustomed to seeing where you can swipe your card yourself (for example, in department stores or grocery stores), except they have the additional hardware that knows how to “talk to” NFC stickers or radios in NFC-enabled smart phones.

Non-NFC Wallets

This is a diverse group of wallets, but non-NFC wallets that support making payments all share some basic similarities. If a mobile wallet doesn’t use NFC to process payments, it must have some other way to send your payment information to the retailer when you are checking out. Some wallets – like the Starbucks mobile app – do this by using a QR code or other unique graphic on the phone that the retailer will scan. (Some wallets turn this model around and the customer’s phone scan’s a barcode provided by the retailer.) The retailer then communicates with a centralized server that matches that unique bar code up to your payment instrument and makes the charge. Other wallets (like the Square Wallet) do this by using the GPS or other location services to determine when the phone is at a retailer’s location and processes the charge by requesting permission from the customer.

These types of wallets doesn’t require any specialized hardware in the smartphone like NFC does, but it generally requires the retailer to do something special to accept payments from the wallet. Some of these wallets are designed to only work with specific merchants. The Starbucks app is a great example of a highly specialized wallet that only works at one retailer.

The Challenges for Wallet Adoption

Despite the good intentions of everyone involved in the payments ecosystem, widespread adoption of mobile wallets has been a challenge in the United States. There are many reasons why this is the case, but there have been a few very specific barriers to mobile wallets catching on:

Accepting a new form of payment requires significant investment from retailers. In the case of NFC it requires updating every single credit card terminal. For non-NFC wallets, it requires updating the retailers entire point-of-sale system and selecting specific non-NFC wallets to support as they are not interchangeable. Any way you slice it, this is a time-consuming effort that requires substantial financial investment. Additionally, any changes to the point-of-sale or checkout process adds risk to the most critical part of a retailer’s business: how they get paid.

NFC required cooperation between the banks, the credit card networks, the phone and phone OS manufacturers, and the mobile phone network operators. While the NFC specification arguably provided the most robust, detailed roadmap for implementing mobile payments, it was also the most complex. A big part of this cooperation was necessary due to the nature of the Secure Element: only trusted parties could access the Secure Element to add your credit card information to the phone. As a result, multiple parties needed to cooperate just to add a single credit card to an NFC phone to enable payment:

  • The mobile phone network operators control access to the Secure Element, so adding a credit card required their active participation. (This was actually a significant technical challenge because of the way the Secure Element is accessed, not just a problem of “flipping a switch”.)
  • The credit card networks (e.g. Visa, Mastercard) control the standards related to how information was stored for a credit card bearing their logo. There are several reasons for this, but one in particular that is important to understand: the credit card number on a Visa credit card is actually owned by Visa – not the issuing bank nor the cardholder. This is true for all popular credit cards, and the credit card network makes the rules about how those numbers are used.
  • The banks that issue credit cards manage the issuance of the credit card numbers owned by the card networks and allocated to the bank for their use. As a result, the banks were a necessary participant to help control how the credit cards they issued were provisioned on the Secure Element.
  • The phone and mobile operating system manufacturers have their own interests in promoting certain wallets. For example, Google has an interest in promoting the Google Wallet on any device running Android. Complicating this is that some phone manufacturers have competing interests in promoting other wallets. One example of this is Samsung, who has their own wallet implementation that also runs on Android phones.

No specific mobile wallet solution had a majority market share among consumers. Given the technical challenges and financial investment necessary to support mobile wallets and in the absence of any other motivation to change, retailers were understandably hesitant to jump on the mobile wallet wagon. Picking a mobile wallet solution that ultimately didn’t achieve widespread adoption would carry a significant penalty with virtually no benefit to the retailer in trying to be an early adopter. This created a “chicken and egg” problem because without significant retailer adoption, no mobile wallet was going to achieve traction with consumers.

… But All Those Challenges Have Virtually Disappeared

As insurmountable as those challenges may seem, they are all going away in the next year. In part two of this article, I’ll explain how changes from the card networks and from Google have virtually eliminated the obstacles to mobile wallet adoption.

What the reaction to Coin tells us about the state of payments

Image of a Coin card

Coin recently launched to much fanfare. Coin combines up to 8 credit or debit cards in a single device the size of one credit card. Coin is an impressive engineering feat, but I find the reaction to the product to be more interesting.

What the reaction to Coin tells us about the state of payments in general

Much of the positive press came from people in the payments industry or from journalists who like techie gadgets. It’s not surprising that this very small part of the population are excited about Coin. What surprises me was the number of normal people – those who don’t care about payments, digital wallets, or the latest electronic contraptions – who are genuinely excited about the possibility of only carrying one physical (Coin) card and having access to all of the payment instruments normally in their wallet. This is the group of people I often test new payments ideas with and are generally disinterested in new solutions.

The fact that normal people are excited about Coin proves that carrying lots of credit cards around is a real problem. In many ways, the non-geeky part of the population is more excited about Coin than they are mobile wallets. This highlights an important consideration for any new payment solution: it’s hard to get people to change their behavior. Coin relies on the old, conditioned behavior of paying with plastic by swiping. Mobile wallets – arguably superior to Coin on purely technical grounds – require changing that conditioned behavior. I believe this is the fundamental difference in the way that Coin and mobile wallets are perceived by the general public.

The good news is that the positive reaction to Coin further confirms what most of us have suspected: consumers will adopt new payment mechanisms that make their lives easier. The lesson to be learned from Coin is that as flawed as the current payment ecosystem may be, it is what consumers are comfortable using. Any innovation that requires consumers to change their behavior – even if it is a remarkably better solution – will have to overcome the inertia of resistance to change. Payments geeks may be anxiously awaiting the widespread adoption of mobile wallets, but the general population just wants their wallet to be thinner; swiping isn’t the problem. Coin achieves that without the magic of NFC handsets, bar codes, or new POS hardware.

Will Coin work?

This is the most important question. In a purely technical sense, the answer is mostly yes. Coin’s FAQ alludes to the possibility that a Coin card will not work outside the United States; I strongly suspect the lack of EMV in Coin is the reason. In places where EMV is (or becomes) a requirement for card acceptance, Coin won’t work. While EMV issuance and acceptance will increase in the United States if the networks proceed with the liability shift, it’s unlikely this will be a major problem for Coin in the next 12-24 months.

The more important factors in determining whether Coin will work are related to acceptance and customer behavior. Acceptance could become problematic if merchants or networks perceive Coin as a risk. Merchants might refuse to accept Coin because they don’t know what it is and are concerned about the risk of fraud because Coin doesn’t carry a legitimate network logo. Or, if fraudsters find a way to exploit Coin (or begin using fraudulent plastics that look like Coin for easier acceptance), merchants might decide it’s not worth taking the chance on a transaction. Likewise, if the networks see Coin as a threat, they could explicitly direct merchants not to accept Coin as payment. It’s unlikely that an acquirer or a network can tell the difference between Coin and an authentic plastic, so the ability of the network to enforce such a directive is questionable at best.

The other question is whether consumers will find enough value in Coin to pay for it. The cost of Coin for early adopters is $50, but their marketing material suggests that price will go up to $100 when the product is generally available. The Coin card will last up to two years according to their FAQ. Will enough consumers be willing to spend $100 every two years for the convenience of Coin to make the product commercially viable on a large scale? With Coin’s current capabilities, I think the answer is no. That said, additional value-added services based on using Coin could create that stickiness. There are lots of interesting capabilities that could sit on top of Coin though, and that’s where I’d place my bets if I was Coin.

The value of packet captures: IE, HTTP 1.1 and misbehaving socket communication

Internet Explorer (specifically version 8, and possibly other versions as well) has an annoying bug that affects HTTP 1.1 persistent connections, particularly with chatty AJAX web applications.

I discovered the bug while troubleshooting a problem for a client. The problem presented itself as a web server or application error: users would sporadically see the application “freeze” for exactly five minutes, then see the results they expected. The problem was unrelated to load (it happened with only one user), occurred in all areas of the application (i.e. there was no consistency as to what the user did when it occurred), and the logs were suspiciously void of any error or indication of a problem at the time of the occurrence. Requests from other users’ browsers during this period would work correctly.

The absence of errors in the logs was particularly perplexing. The problem occurred in an environment using Apache 2.2 (worker MPM) on HP-UX with mod_jk talking to a tcServer instance. Both the web server and the app server were being load balanced by a Cisco ACE. Apache was configured with mod_jk, mod_log_forensic, and mod_dumpio all configured to log literally everything. The tcServer was configured to full debug logging. The Cisco ACE reported that the web and app server load balancing groups were fully available. And yet, every few hours of testing, the application would “freeze”. No log entries on the web server showed any requests taking more than 500ms to complete, much less the five minutes observed by the browser.

Packet captures recorded from the web server finally revealed what was happening. The web server was configured with HTTP 1.1 enabled and a five second keepalive timeout. Internet Explorer would open an HTTP 1.1 persistent connection and a series of AJAX requests could be seen between the browser and the web server. Eventually, there would be a five second break in activity. Apache, behaving appropriately, would send a FIN, ACK to the browser, attempting to close the connection. The browser would respond with an ACK. At this point, the web server has the socket in FIN_WAIT_1 waiting for Internet Explorer to complete closing the connection by sending its own FIN to the web server.


Except that never happened. In all the cases where the
“freeze” occurred, Internet Explorer would send another AJAX request across the socket within a few hundred milliseconds of sending the ACK response to the web server’s FIN, ACK. Because the web server marked the socket as FIN_WAIT_1, it technically received the traffic from the browser but did nothing with it because the client had acknowledged the connection close request.

Internet Explorer waits five minutes for an AJAX response before timing out and will retry the request once. The freezing of the application was caused by the browser waiting the full five minutes, timing out, then retrying the request. This second request, five minutes after the first one, would succeed because it was sent across a new socket connection. From the user’s perspective, it appeared as though the application froze for five minutes. In reality, the application server never got the first request because Internet Explorer completely misbehaved by sending packets across a TCP connection after it itself had acknowledged close request.

Solution: with this particular application, disabling HTTP 1.1 on the web server was an acceptable solution. I suspect that increasing the timeout period (perhaps to as long as 60 seconds or more) would significantly reduce the probability of IE misbehaving, but I haven’t tested that configuration.

A more important take away from troubleshooting this problem: packet captures are the number one tool for understanding what is happening on the network. Packet captures never lie. If you have two or more networked components that are not working correctly and you’re not sure where the problem is, do not guess. Packet captures always reveal the truth. Browsers, thick clients, application logs – anything above layer 5 of the network stack – will inevitably deceive you about what is happening underneath the covers. Wireshark is your friend. Anyone who troubleshoots applications professionally should understand the basics of the TCP protocol and how to interpret a packet capture. It completely eliminates the guesswork of the source of so many types of problems in distributed environments.

Banks (and their customers) are missing out by not API-enabling banking platforms

I’ve written before on the ways that banks could better serve their customers by improving the security of third-party data access and organizing their teams to nurture a culture of innovation. These changes are important because banks are missing out on the power of an API-enabled banking platform on top of which third-parties can build innovative new products. There are lots of new capabilities that could be created by integrating banking services that would absolutely delight customers. For today, I’ll provide just a few examples of what I mean by API-enabled banking platforms and the types of services I’d love to build with them.

Banking Platforms and APIs
Regardless of the size of the bank or the technology solutions the bank has selected, every single bank has – at a minimum – core banking systems that:

  • maintain customer data (legal name, SSN/TIN, date of birth, and so on)
  • are the system of record for accounts (account type, balance, and other account metadata) and the transactional data related to that account
  • provide some form of a product catalog, describing the attributes that are common across all accounts of a particular type, such as the name of the product, interest rates, and so forth
  • provide support for legal, regulatory and compliance needs, like performing AML and KYC functions, storing and reporting data necessary for fraud and other criminal investigations, and satisfy other product-specific like Community Reinvestment Act compliance for mortgages or Regulation CC compliance for check deposit fund availability

Those systems are the bare minimum – the table stakes for running a bank. Potentially more interesting are the systems and services that banks provide for value-added features like:

  • card-linked offers, like PNC’s Purchase Payback, Bank of America’s BankAmeriDeals, or Regions Bank’s Cashback Rewards
  • card reward systems for managing cash back, point or miles benefits
  • person-to-person transfers like Fiserv’s Popmoney or clearXchange’s person-to-person network
  • electronic bill payment and presentment systems

While the core banking systems generally tend to be older, legacy systems (often mainframe based), the systems that offer value-added services tend to be newer. Some of the vendor-provided solutions are already designed around an API-centric architecture.

So, why does this matter?

How Bank APIs Could Enable New Capabilities
In addition to APIs being used within a bank to build their own applications, imagine a bank offering – with the appropriate security, auditing, and permission management by the customer – APIs for third-parties to consume and integrate in to new applications. Third parties could:

  • Build integration between Yelp and bank-managed, card-linked offers. Imagine connecting Yelp with your bank offers and seeing your card-linked deal from PNC’s Purchase Payback program next to the reviews of a restaurant you’re considering visiting. This integration benefits everyone: the customer is happy because he or she gets the benefit of the card-linked offer and has a positive experience with PNC, the restaurant gets business they may have missed out on without the connection between Yelp and PNC, the bank drives customer engagement with their products, and Yelp likely would increase their customer engagement.
  • Embed relevant transactional services into their applications. For example, imagine the ability to connect a mobile app that does budgeting with your bank’s bill payment system. Rather than looking at your budgeting tool then having to switch apps to login to mobile banking, find the merchant you’re trying to pay, and pay the bill, you could pay the bill from within the budgeting app. Again, everyone in the value chain is delighted: the budgeting app provides a more engaging experience for the customer, the bank retains the stickiness associated with bill pay customers, and the merchant got paid the way they always did.
  • Compete with non-traditional services that have historically disintermediated banks. Consider why PayPal has been so successful: the payment experience can happen in the context of what the customer is doing, like sending money to a friend to pay a split restaurant bill. This integration can happen in web or mobile applications and reduce the friction of the payment process. Banks want their customers to use their products and services, but they haven’t created the capabilities to integrate with the bank in non-bank channels. Is it any wonder the banks have been disintermediated?

What It Would Take
While many of the systems of that power the capabilities above tend to be modern, service-oriented applications, they typically have not been designed for consumption outside of banks’ internal networks. As a result, some effort is required to preparing these services for integration with or consumption by third-parties:

  • Security and auditing: Any capabilities that a bank exposes must be secured and have the ability to be audited. This is necessary of any well-designed system, but banks are subject to rigorous regulatory and supervisory requirements that typically far exceed those of a non-bank technology company. Because most of the systems that banks would want to expose were not designed with the notion of external, third-party access, additional access controls are necessary.
  • Fraud and privacy protection: Even if banks only exposed non-transactional services (i.e. services that don’t directly move money around), there are fraud and privacy concerns. Information that is exposed or leaks from these services could be used by fraudsters as part of an attack that ultimately achieves unauthorized access to a customer’s account, like using information to answer security challenge questions with a call center. Also, a customer may opt in to these services to only later fully appreciate how information is shared with third-parties and result in privacy concerns. While these are considerations for any company, financial institutions are held to a much higher standard for protecting their customers, both by their own customers and by regulators.
  • Federated identity management: In order for many of these capabilities to work in non-bank channels the notion of federated identity management must be accepted by banks. Conceptually this means accepting that an identity presented by a third-party (say, Yelp) is equivalent to the identity of a customer of the bank for certain information. There are well-established technical standards for accomplishing this (e.g. OAuth2) that are already widely used by companies like Facebook and Twitter. Thousands of applications use this type of integration today, and if you’ve ever “logged in” with your Facebook or Twitter account to another site, you’ve seen it in action. For banks, though, this style of integration can be tricky: they need to decide exactly how much to trust these other forms of identification.
  • Operational support: Once the technical challenges are addressed, banks need to create the processes and procedures to operate the capabilities. This includes supporting their customers (who will inevitably have questions or concerns) and the third-parties that consume these new services (an entirely new type of “customer” for banks). This is as much about how these services are supported as it is who supports them.
  • Legal, regulatory and compliance implications: With technical and operational challenges behind them, a bank would still face questions about the legal, regulatory and compliance implications of providing these types of services. Smaller institutions would likely face less scrutiny than the large, national banks. Nonetheless, there is very limited precedent or regulatory guidance for these types of services. The first bank that attempts this (and their regulators) will be operating without a playbook.

These are not insurmountable obstacles but they do explain why banks aren’t tripping over each other to be the first to market to offer these services. The banks that challenge the status quo and begin offering these services will be at the forefront of a new era of banking. That era will open up new ways for banks to drive revenue and engage their customers.

Mobile payments are alive and well

The mobile payments space is alive and well despite some speculation to the contrary, such as the recent suggestion that American Express has simply given up on it. From that article:

“It appears as if American Express is tired of waiting for the mobile
payment craze to kick off and is taking the matter into it’s own
cards, and in this case awards points. In fact, one of Amex’s points
is that the program is a seamless integration that does nothing to
change the way you currently pay for a cab.”

The author suggests that American Express’ latest offering – the ability to automatically use points when paying for a cab ride – be read as abandoning mobile payments. I disagree. I believe this program was designed to provide additional creative ways for cardholders to redeem points. American Express customers may like this, but American Express also has an incentive: outstanding reward point balances create a liability on the balance sheet. The more ways customers have to redeem points and relieve the liability pressure on the balance sheet, the less likely it is for large point balances to build up. What American Express has done with cab payments is no different than the integration with Amazon to shop with points.

However, the broader sentiment expressed by the author is important. In particular, he points out that:

“There is nothing about paying with a mobile phone that is better than
paying with a credit card. So the perceived risk of security is
certainly not worth it in the eyes of most consumers. The fact that
the risk of fraud is extremely low for those making payments (rational
information) makes no difference either.”

This is the biggest problem with (most) mobile payment solutions as they have been implemented to date. Consumers are willing to use mobile payments when they are better. The mobile payment solutions that have offered loyalty benefits (like the Starbucks mobile app, the most successful mobile payment implementation in the US to date) or a truly better experience (like Square Wallet with the ability to auto check-in and pay) have had success with adoption. Expecting consumers to change their behavior when they get nothing in return is foolish; consumers need to get something in return for changing how they behave.

The real innovation in mobile payments isn’t in the plumbing of moving money around (card networks vs stored value vs ACH). All of those details are interesting only to payment geeks. The real innovation in mobile payments is in the new, value-added capabilities consumers will get that is built on top of that plumbing. Lots of people are working on mobile payments innovation and American Express’ cab payments with points has nothing to do with it.

Troubleshooting Heroku Postgres out of memory errors

If you see a Postgres out of memory error, it can be very alarming, particularly because this type of error tends to appear at the worst possible time: when you’re trying to scale your application. In a Rails environment, this type of error manifests itself as an ActiveRecord error:

ActiveRecord::StatementInvalid: PG::Error: ERROR: out of memory

Experiencing this error on Heroku Postgres can be even more unsettling because your immediate response may be “add more memory”, except doing so requires provisioning a new, larger database as a follower, waiting for it’s commit log to catch up, then going in to maintenance mode and switching it over. Resist the urge to do this for just a few minutes, remain calm, and do a little investigation!

If you’ve never seen the error in your application before and it appears under heavy load, there is a chance that you have a poorly performing query and throwing more memory at Postgres may not resolve the problem. It’s worth taking a few minutes to understand what’s going on before you spend time taking action on a problem with an unknown cause. Luckily, Heroku makes this easy.

First, install the Heroku pg-extras plugin if you don’t already have it installed:

heroku plugins:install git://

Second, look for running queries against the database:

heroku pg:ps

In the output, look for queries that are idle or several queries with the same structure even if they have different where clauses. If you see this, there is a chance that these queries are what is causing the Postgres out of memory error. Look at the query plan for those queries by executing an EXPLAIN on the query in question. If the query is expensive (lots of rows, sequential scans, multiple hash joins, etc.) you can further confirm they are the source of the problem by killing the query, again using the Heroku Postgres extras:

heroku pg:kill procpid

where procpid is the PID shown in the output from the heroku pg:ps command. If this temporarily resolves the problem, it’s a good sign that the queries you killed are causing the out of memory problem.

At this point, a permanent resolution to the problem probably requires improving the performance of the query. If the number of simultaneous problematic queries is somehow constrained, adding memory to Postgres may mask the problem. However, if the number of simultaneous problematic queries is a function of load on your application, adding memory will only mask the problem until you hit the next tipping point; eventually the problem must be fixed. Refactoring the query to limit the number of rows or eliminate sequential scans is the best place to start.

Speed up database-intensive model/controller activity on Heroku

While Rails logging is usually a very small component of overall response time of an application, there are cases where the overhead of logging can significantly degrade performance. In one application hosted on Heroku with significant database-intensive operations for several controller methods, disabling ActiveRecord logging resulted in a 34% improvement in response time. (To give you a rough idea of what I mean by database-intensive, the average response time for these requests were ~7 seconds with >90% of that time taken up by ActiveRecord activity. The log output from ActiveRecord-related log entries in this application exceeded 100gb a month.)

Disabling logging is easily done by adding this block in your environment configuration:

  config.after_initialize do
    ActiveRecord::Base.logger = Rails.logger.clone
    ActiveRecord::Base.logger.level = Logger::INFO

Different hosting environments will likely see different performance benefits; I suspect hardware plays a large role. The performance improvement was negligible when I benchmarked the same change in my local development environment that wrote logs to SSD storage, for example.

Bottom line: your mileage may vary, but disabling AR logging may be an easy way to improve performance of database-intensive operations.

Minimizing Heroku slug size when you have a lot of image assets using CloudFront, S3 and the asset pipeline

The Rails asset pipeline was a major improvement in organizing static assets in a Rails application. Rails also offers a great mechanism to configure an asset host for rendering static assets including images. Amazon’s AWS CloudFront offering is an easy to configure CDN that can be used in combination with the Rails asset host configuration, but if you have a lot of images and host your app on Heroku, this can lead to problems from having a very large slug.

Sometimes you may have a large number of files (for example, if you have a large photo gallery) that you could host on an external static hosting service like AWS S3, but other assets (like those that are used in actually rendering your site) that would be more easily maintained and served using the Rails asset host with a CDN in front of them like CloudFront. This would reduce the Heroku slug size, but the problem with this approach is that you only have one asset host to configure and CloudFront only retrieves files from one location (your Rails app, or an S3 site, but not both for the same CloudFront distribution).

A typical CloudFront configuration with Rails looks like this:


When CloudFront gets a request for an asset that isn’t in its cache, it makes the request to your Rails application for the asset. The Rails asset pipeline will serve the image to CloudFront, CloudFront caches the asset for future requests and responds to the client’s request.

Unfortunately, if you only have one asset host and one corresponding CloudFront distribution, CloudFront will only point to one location to retrieve uncached assets. This means all your assets need to be served by your Rails app, requiring the assets themselves to be part of the app and increase your slug size.

Alternatively, you could point CloudFront to a web-enabled S3 bucket for assets, but then all your assets need to be in the S3 bucket. This is a problem if you want to manage some images for your site locally but keep others on S3. Luckily, there is a solution that allows you to do this:


To make this work, add a line to routes.rb for the types of assets you want to serve from S3:

Then create the assets controller with the appropriate methods as defined in your routes:

This makes the controller proxy the request to S3 (or any other static asset host) for the paths you’ve specified but all other requests are handled normally. In practice the controller methods would probably be made a little more flexibly by using a configuration parameter for the path to the S3 host and adding more intelligence for setting the image type. Also you could add some error checking so that an HTTP error getting the real asset would be passed along to CloudFront directly.

Most applications won’t require this hybrid approach. However, if you have an application that has a bunch of images you’d like to host elsewhere and some you’d like to host in the asset pipeline, this can significantly reduce your slug size.