eCommerce Analytics

Collaborative filters, the heart of the recommendation engines used by companies such as Amazon and Netflix, are quite good at predicting items that might be of interest to you. Essentially, these filters work by trying to group you with people that have expressed similar preferences — whether it’s by CDs you rated, the movies you chose, or the items you bought — and then finding the items that the other people in the group like that you have not yet seen.

When the dimension of time is introduced into the environment, however, collaborative filters can quickly lose their predictive power. This is particularly true when filters are used in retail for regularly purchased goods. What may interest a customer at checkout time is not likely to be what interests them six months down the road.

For example, my cat has a chronic eye condition that requires semi-regular treatment. On average, we purchase eye drops about once a year though it can vary anywhere from three to 18 months. Due to the quirks of cat physiology, after application the medication drips into the cat’s mouth and, if her reaction is any indication, tastes horribly. To mitigate the yuckiness we usually give her some treats. I imagine we’re not the only ones who do this.

A good collaborative filter might find that cat treats are a good cross-sell for eye drops. I would, in fact, be likely to add cat treats to my shopping cart at check out if they were offered. A retailer using a recommendation engine would get extra business from me. A win for everybody.

Three months later, however, that same retailer is now sending me promotional emails for cat treats because their collaborative filter has no concept of time. It should be sending me offers for eye drops — that’s something I would probably be interested in buying again. Instead, after a few weeks of receiving irrelevant offers, I find myself not opening these emails or, even worse, consider opting out of receiving them. Not only has retailer lost an opportunity to make an additional sale, but they’ve come close to losing a valuable way to communicate with me.

The key is to use recommendation engines that are specifically designed to handle the time varying nature of email. In work we’ve been doing with customers, we’ve seen than a recommendation engine that considers when a person is most likely to purchase coupled with the purchasing sequence of groups with similar preferences yields results that are three times better than traditional collaborative filters. I have little doubt that we are the only group pioneering this concept and suspect that we’ll see a lot more email specific recommendation engines in the future.

In the months and years to come, smart retailers will look past off-the-shelf recommendation engines that are optimized for cross-sell at check out. They will see that the opportunities are tremendous for those using recommendation engines that are specifically designed to understand time.

Istobe develops predictive models that recommend which products to market to customers via email and which are the best times to market those products. But how does Istobe measure the actual ROI returned by these models? The Istobe team burns many cycles discussing measurement techniques for the lift that we are delivering to our clients. And we’re constantly updating the formulae that we use to evaluate how our predictive models actually perform in production. Ultimately, the measured lift that we generate is the result of another model where we tie in the relevant factors according to different weights. What are the relevant factors? Read on.

Our model vs. current practice or our model vs. the naive approach
This actually isn’t a debate among us but it’s the most important part of understanding what kind of monetary benefit we’re actually delivering to the customer. Oftentimes, a model’s output will simply deliver lift in contrast with the naive approach. That is, the model will assume that our client is, at worst, merely flipping a coin in terms of the next-best product for their customer. Or, at best, the model assumes that the client’s customers will likely want the most popular product. So our models self-reflexively examine their benefit against these two benchmarks. However, when it comes time to actually measure how much better our model is, we always measure against our clients’ current practices. The assumption is that our clients already have a smart strategy for targeting their customers. So we get their rules for targeting their customers and then figure out how much better our models are at generating the right type of product offering.

Our model’s email timing vs. typical email timing
Email timing is starting to get a lot of traction at Istobe these days. After all, if the email is never opened then it doesn’t matter if the product that our clients are offering is a better fit for a set of customers or not. And there are better and worse times to send emails if you want them to be opened. So we take into account the timing that we suggest vs. the normal send times of these emails. Basically, timing is just another part of our models’ output. The models take into account the whole path for purchasing a product and getting an email to the right person at the right time is the first step in that process. When we track the Istobe improvement, we build email open rate into our evaluation and track how much lift we give our clients by understanding how many more opens and click-throughs our models were responsible for.

That’s about enough for today but I’ll talk about two other evaluation factors on Thursday that are a little more arcane: Email influence zone and opt-out rate.

In the age of Amazon, up-selling a customer has become tantamount to suggesting more products during checkout time. In fact, Amazon’s recommendation engine is a relentless upseller and its success has spawned a breed of companies dedicated to building recommendation engines that they license to online storefronts. The truth, however, is that good upselling seeks to harness a larger percentage of the customer wallet over a longer period of time.

While Amazon and other storefronts are undoubtedly suggesting some of the best items to customers when those customers are at the point of purchase, this type of selling also has the feel of the chewing-gum rack at the grocery store checkout counter, attempting to capitalize on customer impulse. Checkout-time recommendations are great if you’re a grocery store and selling a product that is quickly consumed. But what if those checkout-time items have a shelf-life? More likely than not, you’ve saturated your customers for the foreseeable future. If you’re not selling a consumable, upselling with shopping-time recommendations is tantamount to cannibalizing future revenue from that customer.

Think about it. If someone adds a third t-shirt to their purchase because they react impulsively to an upsell recommendation, are they likely to buy a t-shirt from you two months later? The risk of checkout-time up-selling, of course, is that while you do sell more in that instance, adding one more needless box to an already-tipping armload, you are not delighting your customers. Worse yet, you may give them a sense of buyer’s remorse, an association that your company most definitely does not want.

To truly delight your customers, isn’t it better to give them what they want when they likely want it, not when you feel they are in a captive position? In other words, truly satisfying your customer doesn’t mean compelling them to engage in impulse buying. Instead, your company should be seen as a trusted advisor that brings the right information at the right time. Oftentimes, this is not when they are on your website. Let’s take the case of a customer who purchases yarn. Would you rather sell them so much yarn - in colors that they may not ultimately want - at one time that they feel buyer’s remorse? Or would you like to be there with an email right around the time that our theoretical knitter runs out of yarn? Which approach do you think will turn that customer into a loyal customer who buys at your store again and again?

This is the customer service side of marketing that seems lost with checkout-time recommendation engines. I believe that the data mining behind recommendation engines is unquestionably the future of marketing. In fact, as a co-founder of Istobe, I bank on it since our specialty is helping to understand customers - and their buying patterns - based on their purchasing history and their demographic information.

Up-sell today is not necessarily what up-sell will be tomorrow. As customers become more savvy in their online purchasing, bridging the gaps between site visits will be increasingly necessary to establish a relationship with your customers and to signal to the customer that your company is a trusted advisor not a relentless upseller.