eCommerce Analytics

Just finished getting through the backlog of my Don Dodge RSS feed today and I’m happy to report that venture capitalists seems to think that businesses like Istobe are about to break out. Let me qualify that. Venture capitalists seem to think that using data to improve e-commerce is an industry that clearly needs some maturing and that maturing time is nigh. Istobe represents that maturing, combining hundreds of models and data integration routines into a package that lets you target the right customer with the right product at the right time.

Investors believe that the maturation in this industry will occur in the next five years. Well, so does Istobe. We believe that it’s time to put your data to use. If you don’t use it, data is no more than the new shelfware: that software you just had to have before you realized you lacked the in-house talent to unlock its value. Istobe is your outsourced in-house data analysis talent that lets you ask simple questions and get answers without analysis, questions like: I need to sell this overstock of shirts, to whom should I market them? In reply, Istobe gives you a list of your customers that are most likely to buy your shirts and the probability that they will buy. This is the new paradigm in predictive modeling that Gartner calls the data mining packaged application. Let’s just take a quick look at the moment in time at which we are poised.

Data collection methods are clearly refined

Nowadays, everyone is sitting on a pile of customer data that they don’t know what to do with. As Rob Hayes, partner at First Round Capital, says in the Stefanie Olsen article from which Dodge draws his inspiration, “Everyone talks about all the data that’s being created and how valuable it is, but the way you make it available is by doing something actionable with it.” The glut of data is due, in part, to years of CRM implementations and the current CRM zeitgeist. You can’t turn around without being inundated by a flood of marketing for “next generation” CRM systems. In fact, I used to be one of those marketers at a not-too-small company that builds Dynamics CRM.

Of course, online everything has made data more prevalent as well. Returning some kind - any kind - of functionality in exchange for your profile is no longer new hat. Every widget and social network known to man requires you to divulge information before you start using it. And then it collects your clickstream as you use the app. Heck, I’ve got at least 50 different login/pass pairs that I need to remember now.

Purchases, as well, fall into this category. With online shopping increasing at a terrifying clip, all of your purchases are more seamlessly collected and tied to your profile and your clickstream, meaning that now, more than ever; pre-purchase behavior and purchaser characteristics - on a large scale - are at a marketer’s fingertips.

Data analysis tools have matured but haven’t turned the corner

So there are various types of data out there right now that need tying together and, ultimately, analysis. But have the tools to merge and make sense of that data improved? Not appreciably. Really, when you get right down to it, the tools used to perform data analysis are still catch-all tools that can build any model you want or merge any type of data you want. But they can’t help you with specific business problems. In other words, the tools exist for database experts (data integration) and PhD statisticians (statistical modeling tools).

For years, these tools have gotten easier for experts to use but haven’t gotten any easier for business users. This means that your $300K worth of in-house data integrators and PhD statisticians have become slightly more productive over time but translating their language into the language of business is as difficult as ever. And turning the data they spit out into a meaningful business strategy is just as tough.

Recently, I’ve noticed that the vast majority of articles and blogs out there on marketing and customer analytics primarily focus on how to use data-mining to increase revenue and cut costs in B2C oriented companies. Today, I wanted to take 5 minutes to outline some ways B2B companies should be using analytics to increase their top line while minimizing costs.

Knowing Your Leads

Many sales people will tell you that they already know which companies and decision makers they should be selling into. The marketing folks will tell you they already know who matches their ideal lead profile. Funny part is that in many cases these two ideals don’t match up.  A recent marketing research report concluded that in nearly 50% of companies out there, marketing and sales do not have a direct view into what the other side is doing. Data analytics can clearly help bridge this gap by following a customer through - from lead to purchase to maintenance - showing which leads result in the most profitable customers - not just in terms of initial sales, but also in terms of potential lifetime sales. It’s the best way to make sure both departments stay on the same page and the entire company increases its overall hit rate.

Better Cross Sell and Up Sell

It shouldn’t surprise anyone that better targeted email results in better cross-sell and up-sell rates. Personalization is a hot topic in the B2C space, but there is no reason why B2B companies shouldn’t have the ability to tailor their emails to match the company profiles of their customers.  A CTO shouldn’t be receiving the same sales pitch as the marketing folks and someone in the energy business shouldn’t get an email that is designed for online retail. These seem like easy rules to follow, but the trick comes in knowing how to segment these customers by industry, contact, and previous purchase history to minimize the work needed to pitch the right offer at the right time.  No one has time to draft up 100 emails to each type of customer - which is why predictive analytics is critical for optimizing the segmentation process to forecast which customers are going to want similar products in the future.

The True Cost of Support

Many times support personnel get relegated to the back office and we tend to forget about them when factoring in the true costs of sustaining a relationship with a customer. If a customer spends $1,000 on one of my products, but then uses up 100 hrs of one of my support staff, are they still a profitable customer? Factoring in and then forecasting support costs can be an extremely valuable way to judge the lifetime value of a customer and help management decide whether they should renew contracts or continue maintenance agreements.  Working this back into lead generation can also help marketing make sure they focus on customers that will not only buy product, but also won’t cost the company an arm and a leg once that customer comes onboard.

These are really just a few ways B2B customers can make use of data analytics. There are many more, and next time I’ll try and focus on some leading edge cases out there that show how B2B companies are using analytics to increase revenue and cut costs.

I noticed that the RRW Consulting blog alluded to an article on Friday that I have been promoting to my peers: a research report by the Aberdeen Group (abstract here) that discusses the importance of email personalization. The one-to-one marketing emphasis in the article is precisely the kind of email targeting that we espouse here at Istobe. Today, I want to expand on one aspect of the Aberdeen report that we spend extra time on at Istobe: the importance of the buying cycle in determining what kind of email message to send your customers.

In the Aberdeen article, Ian Michiels mentions that web analytics provide great clues to assessing where customers are in the buying cycle. For example, if a customer invests a vast amount of time clicking about a product group, that customer is likely doing research and is in the market to buy a product in that area. A discount offer, Michiels says, would likely get this customer - who is now highly qualified and advanced in the buying cycle - to act on their desire and make a purchase.

I totally agree with this sentiment. But as Chris mentioned in detailing his experience with GPS systems at Amazon, there is another way to do this. Customers can clue you into what they want via their clickstream. But even if you don’t have clickstream data, transaction histories, once supercrunched, can give you a leg up on finding customers who will likely buy next. In other words, this supercrunching can help you locate the customers that will likely buy before they locate you.

How does this work? Well, other customers have come before them and laid out patterns that aren’t perceptible to you and I but are very perceptible to Istobe’s predictive models. Istobe’s models throw out those customers that are not likely to buy again and then work with those who are. From there, Istobe’s models assign the products that are likely to be purchased by these likely buyers.

I won’t argue that this method is more statistically powerful than clickstream data, which is a solid indicator of future behavior. But I will argue that clickstream data takes vast amounts of resources to capture and use, a difficult proposition for online retailers who are just dipping their toes into analytics. And using transactional data to predict who will buy next is a more proactive approach. So what do you get from that proactivity? Probably a two- to three-month head start on your competition. You can focus on targeting your “most likely” customers with act-now offers while your competition waits for these customers to visit their web site.

Something that we constantly talk about around Istobe is why more small companies aren’t interested in implementing customer analytics. Especially given that these companies are pressed now more than ever to keep up with big companies that run customer analytics algorithms as part of their hourly routine. And in an era when a tome like Supercrunchers proclaims a new dawn for predictive models in business. Even with a value proposition that demonstrates a solid return per thousand customers, we often have problems convincing customers that our mojo is good and that data mining is more than just voodoo. Well, today I prepared a really simple sample revenue calculation to quantify the value of Istobe’s predictive models. Take a look at how much more an Istobe-based email campaign makes on a yearly basis.

The rule of any business venture is it must either make money or cut costs for clients. If MIT Sloan didn’t drill this into us, we got a refresher into this the other day when Doug, Chris, and I met with Chris Merrill, founder of Thrive Networks, The Orchid List, and Owner of Ass Industries. He broke that down pretty plainly for us when talking about our need to educate our customers. Well, here is as plain a case as I can make for the revenue increase that Istobe brings to the table when we mine customer data to determine what product they want to buy next.

The Revenue Increase

Assuming a company that emails 500,000 customers weekly, 50 emails a year, and $1.90 per email open, the Istobe predictive models allow a company to make $412,300 more than just basic email blasts and nearly $80,000 more than a baseline targeting approach. There is a full calculation below but let’s talk about what some of the numbers are before you examine them.

The Setup

First, let’s take a very conservative number of 4% as the increase in predictive power of our models over a general baseline. In this case, we assume the baseline to be the company offering the same product that a customer already bought. As creatures of habit, consumers are typically likely to buy the same thing from a company that they just bought. Let’s take dog food at a grocery store for example. Whoever buys dog food probably has a dog and there is a good chance that they’ll keep buying dog food. It’s a safe bet. The same goes for internet retail. If I buy a shirt from an internet retailer, it’s likely I’ll go back there to at least look for another shirt - assuming I didn’t hate the first shirt and the customer service was adequate. Now, the pitfalls of offering the same product is something I won’t go into in detail here; suffice it to say that offering the same thing that a customer is likely to buy again and again from your company is not the best business move. Really, your company should be looking to expand the share of your customers’ wallets. And this means offering them new products that they currently buy elsewhere. Like I said, a topic for another day. So Istobe does about 4% better than offering the dumb alternative, which is offering the same product again. 4% doesn’t sound like a lot but let’s play this through.

Now, let’s assume that a client uses our predictions to send out targeted email. It’s been assumed that normal behavioral targeting enhances your open rate by 40%. This means that just offering your customers the product that they bought last time increases your open rates by up to 40%. Well, I’m not so sure that I trust those studies so let’s just play it conservative and say that targeting spurs a 20% open rate increase. What that means ultimately is that Istobe predictions improve the targeting - the 20% increase - by 4%.

As a base open rate, we’ll use 3.5%. That’s a pretty conventional rate when sampled from our customers. When you apply the Istobe predictive power (4% increase) and the lift from general 1:1 targeting (20%), Istobe’s open rate is 4.4% and the baseline targeting rate is 4.2%. The normal open rate, remember, is 3.5%. This means that, if we take 1000 customers, Istobe will get 44 opens, baseline targeting will get 42, and the traditional email blast will get 35.

Here, we’ll use $1.90 per open as the amount of money that our fictitious company earns per open. Ultimately, not all opens are sales and this figure essentially backs out the complications associated with the website, ordering mechanisms, sizing problems, etc. In other words, it’s a way to understand how much each open is worth on average.

When you take into account that each email open is worth $1.90, and that Istobe has 44 opens per 1000 customers, Istobe’s predictions make our fictitious company $82.99 per 1000 customers per week. Baseline targeting makes $79.80 and regular email blasts make $66.50. Calculated for 500,000 emails per week for the course of a year, we get the figures that I began with at the top.

The Calculations

When we talk with customers about improving the success rate of cross sell and up sell opportunities on their existing products, many reference Amazon.com as the retailer they believe is doing the best job in the industry.  While I agree that Amazon does a fine job with following up on self-generated leads, I don’t agree that they do an especially good job of anticipating or prompting additional purchases.

Let me give you an example. I bought my mother a cookbook three weeks ago for her birthday. I wrote out a gift card, opted for the additional gift wrap, and shipped it to my parents’ home. Additionally, because I’m particularly anal when it comes to data organization, I tagged it within Amazon as a present that was bought for my mother. Now, knowing all of that info, any guesses as to the next product Amazon decided to pitch me on? Cookbooks. I can maybe understand the tendency for someone to purchase the same product again after purchasing it once…maybe…but in this case, really? After doing everything I could to signal to their system that the purchase wasn’t for me (different ship to name, different address, tagged as belonging to a different person), I still get an offer for cookbooks?

Now, don’t get me wrong, Amazon does do many things right. They are one of the best on following up on customer clickstream data. A recent visit to the site to research GPS systems for an upcoming trip resulted in two emails offering deals on GPS systems I actually might be interested in.  My problem with Amazon, and more specifically, the perception of their analytics is that their email marketing model is based on being reactive - the customer is responsible for coming to the site and narrowing down the type of product they want to buy - and Amazon reacts to this behavior with an appropriate offer.  But for those retailers that aren’t Amazon, who can afford to sit and wait for customers to tell you what they want?

The key to predictive analytics is being just that… predictive.  Most retailers don’t have Amazon’s budget and need to have better information about who is going to buy next month or next quarter, and more importantly, what they’re most likely to buy. And for most, the surprising fact is it’s not much of a leap to get there since the majority of companies are already sitting on all the information they need to better target email blasts by matching offers with customers who are willing to buy.

I praise Amazon for their ability to follow up with customers that express interest in certain products, but as far as predictive analytics….there are better solutions out there.

We talk a lot about the nuts and bolts of predictive analytics on this blog - how we build customer analytics models, how we interpret them, and how we establish whether or not they’re working in production. However, one item we have yet to tackle - and a very important item at that - is how to integrate data mining and/or predictive models into your organization and your company’s routines. Indeed, the black box in the middle should be the necessary data integration (getting the data into the right format for the models) and running a bevy of models against the data to see which one projects as the most effective. But it’s the before and after this black box that really make or break any attempt to incorporate predictive models into a company’s processes: data collection and incorporating data mining results into your processes.

Do you need to collect more data than you already have?
The main problem that companies run into is the need to collect more data in order to build the necessary predictive models. For example, if a company wants a model that tells them which of their products a specific customer might buy next, does it already collect the data necessary to support a model that does that? Ultimately, the question is: Is it worth the time and cost needed to invest in collecting more data that may not yield any better results when supercrunched? New collection methods take months to set up and sour clients on predictive models before the fun has even begun.

We believe that it’s important to try to work first with the data that our clients have and add new sources of data over time. This is a handy tip for business users who really want data mining to work in their organization: start small by using the data you have and get more sophisticated as you go. I know it may be tempting to put in that clickstream collection database right now so that you can use online behavior to segment and market to your customers. But trust me, get the buy-in from your organization first by proving that predictive customer analytics work. Then ask for the stars.

Are you ready to completely change the way that you do direct marketing?
Some companies would have you subscribe to a whole new way of doing business in order to use their models. Customer-centricity is my favorite new business philosophy. Though I firmly agree with the need for firms to be customer-centric, is it realistic to expect companies that use offers, coupons, and holidays to draw new and existing customers to their websites - and have for years - to change their direct marketing approach to accommodate a new set of tools? Not really. Company culture and routines are slow-developing and even slower-changing.

So the question at the end is really: What are you going to do with all this newfound predictive power? How are you going to fit in customer-centric model results - We’re 99% sure that Brad Pitt will buy two Baby Bjorns with lumbar support - to your next email blast that has a summer theme and features hats and sunscreen? And what about the extra creative necessary to relay that customer-centric message? You’re going to have to make a new email blast featuring the Baby Bjorn with lumbar support, in addition to the summer-themed email.

Well, I’m pretty sure that the summer themed email blast was probably going to draw some business but I’m also positive that luring customers with what they want when they want it is a lucrative way to market. The best practice is to initially skim the cream off of the predictions, to take those that have the highest probabilities of succeeding, and run with them. Collect a group of customers that has the highest probability of buying the Baby Bjord - Mr. Pitt among them - and send an intermittent email to just that group. Now watch your open rates and clickthroughs soar.

On Monday, I began a discussion about how Istobe evaluates the ROI from email marketing campaigns based on our predictive models. At the end of my post, I promised a discussion about other factors that we take into account when evaluating the lift. And…voila. Today we unveil those factors: the email influence zone and opt-outs, and we discuss how Istobe accounts for them in our lift calculations.

Email influence zone
Sometimes referred to as decay rate in the catalog industry, the email influence zone (EIZ) - not unlike the catalog influence zone (CIZ) - is essentially the time period after an email is sent. And we assume that each succeeding day after the email is received has less effect than the day before. Thus, the moniker decay rate. Catalogers have believed for years that their catalogs have a carry-over influence: the catalog accounts for many web purchases. In fact, this is very reason that catalogers are loathe to cut the number of catalogs that they ship. Even to those customers who have never purchased from the catalog itself. We believe this is also true of email marketing.

Basically, the idea behind the EIZ is that an email offer has an effect on online purchases that have no other obvious origin and which relate to the product that we predicted. For example, if our models predict that shoes are the likely next product for a particular customer and that customer purchases shoes online five days after receiving an email that advertises shoes, then we can assume that the email - and our product recommendation - influenced the customer’s purchase. Our model gets credit for a small percentage of this purchase even though the purchase didn’t come directly from an email click-through. The EIZ period that we calculate differs per client depending on the frequency with which our clients send emails.

Opt-outs on the Istobe watch
If we’re going to give ourselves some of the credit for purchases that occur in non-email channels, we also have to take a hit for bad events that occur during our watch. The bad event that Istobe tracks carefully is email opt-out. We track whether the opt-out rate goes up during our watch. If it does, we have to assume that next-best offer has somehow turned customers off. If the opt-out rate does go up, we deduct a portion of our lift because we believe that we were responsible for that incline in opt-out rate. We’re responsible for that small piece of customer attrition.

Taken together with the variables I spoke about last time, these are just four factors that we constantly adjust in determining how successful we are on behalf of clients. And we’re always looking for new ways to perceive actual lift. If you have new ideas for evaluating predictive-model efficacy, please email me. I’d love to talk about them.

I want to take a minute and build off of Matt’s post from yesterday. While lift, confidence, and support may sound like terms that are more applicable to therapy sessions, they are actually the metrics that we use to rely the trust we have in our models. Many people we talk with are familiar with these terms on a certain level, but when pressed, their understanding boils down to the following: higher is usually good and lower is usually bad. I wanted to use this post to define these metrics a little more thoroughly and talk about how they’re calculated.  Hopefully, readers will come away with a better understanding of what they mean and exactly how they’re used.

Support

In order to talk about more complex terms like lift and gain, we need to first start with the basics: support.  Support, sometimes referred to as the cover, is the number of data points (customers, transactions, etc.) that meet a set of rules and/or assumptions.  If I do a market basket analysis and find that customers who buy milk also buy cereal, the support would be the number of customers in the sample set where this holds true. Obviously, you can only estimate the value of the support number when given the size of the total sample population which is why we have our next metric: confidence.

Confidence

Since a rule with a support of 900 looks good when the sample size is 1,000 and not so good when the sample size is 1,000,000, we need a way to easily figure out whether or not our support is significant. Confidence is a ratio that takes the support number and divides it by the number of instances where the rule may hold true (or to be more exact - where the antecedent of our rule holds true).  For instance, in our milk/cereal example above, confidence would be the total number of customers who bought milk and cereal divided by the total number of customers that bought milk.  While it’s true that the higher the confidence the more reliable the rule, it is important to note that knowing the total sample size and the support value as well as the confidence is necessary to get an accurate picture of the rules significance in regards to the total population.

Benchmark

I define benchmark here because it makes it easier to explain both gain and lift.  Benchmark is the total number of items (customers, transactions, etc.) that meet an outcome divided by the total number of items in the database. Let’s go back to the milk/cereal example. Since cereal is the outcome that we are trying to predict, the benchmark would be the total number of transactions where cereal was purchased over the total number of transactions in the database. In layman’s terms, if we were randomly picking 100 items out of the database, it is the percentage of those items where the outcome would hold true. Benchmark is valuable because it puts a lower bound on the value of a model. If a model can do better than the benchmark value, then it provides real value to the customer.

Lift

The most common term that is used in statistics and especially analytics is lift.  Lift is a way to measure how much better a model is over benchmark. It is defined as the confidence divided by the benchmark and any value that is greater that one suggest that there is some usefulness to the rule. Many applications show lift in a chart. In these instances, the total population is divided into deciles - ten even groups - into which members are placed based on their predicted probability of response. The highest responders are put into decile 1, etc.  Lift is then calculated for each of these deciles and plotted on a line chart.

Hopefully, this provided a little more insight into how we calculate the value of a model. Next time, I’ll run through a complete example to show how these are calculated in practice.

Istobe develops predictive models that recommend which products to market to customers via email and which are the best times to market those products. But how does Istobe measure the actual ROI returned by these models? The Istobe team burns many cycles discussing measurement techniques for the lift that we are delivering to our clients. And we’re constantly updating the formulae that we use to evaluate how our predictive models actually perform in production. Ultimately, the measured lift that we generate is the result of another model where we tie in the relevant factors according to different weights. What are the relevant factors? Read on.

Our model vs. current practice or our model vs. the naive approach
This actually isn’t a debate among us but it’s the most important part of understanding what kind of monetary benefit we’re actually delivering to the customer. Oftentimes, a model’s output will simply deliver lift in contrast with the naive approach. That is, the model will assume that our client is, at worst, merely flipping a coin in terms of the next-best product for their customer. Or, at best, the model assumes that the client’s customers will likely want the most popular product. So our models self-reflexively examine their benefit against these two benchmarks. However, when it comes time to actually measure how much better our model is, we always measure against our clients’ current practices. The assumption is that our clients already have a smart strategy for targeting their customers. So we get their rules for targeting their customers and then figure out how much better our models are at generating the right type of product offering.

Our model’s email timing vs. typical email timing
Email timing is starting to get a lot of traction at Istobe these days. After all, if the email is never opened then it doesn’t matter if the product that our clients are offering is a better fit for a set of customers or not. And there are better and worse times to send emails if you want them to be opened. So we take into account the timing that we suggest vs. the normal send times of these emails. Basically, timing is just another part of our models’ output. The models take into account the whole path for purchasing a product and getting an email to the right person at the right time is the first step in that process. When we track the Istobe improvement, we build email open rate into our evaluation and track how much lift we give our clients by understanding how many more opens and click-throughs our models were responsible for.

That’s about enough for today but I’ll talk about two other evaluation factors on Thursday that are a little more arcane: Email influence zone and opt-out rate.

Maybe. But I can guarantee your revenue per customer does. And not in the way that you might believe.  There is strong evidence that reducing email in an intelligent way actually increases your revenue per customer.

Just yesterday one of my colleagues asked me whether, in addition to the weekly timing of an email send, the quantity of emails sent to one person mattered. In other words, is there a limit to the email offers that a marketer should send? The intuitive answer is: of course. If we look at catalogs alone, consumer dissatisfaction with this method of direct marketing is at an all-time high. After all, no less than six websites have sprung up that allow consumers to opt out of catalogs. You’d have to have a powerful argument for me to believe that overzealous emailers are perceived any differently than overzealous catalogers.

My partner Doug Bright has already spent some time fleshing out this hidden cost of excessive email. So I’ll just add some more beef to his already meaty argument. In March, 2006, noted marketing researcher Dr. V Kumar, along with Rajkumar Venkatesan and Werner Reinartz came out with an article entitled “Knowing What to Sell, When, and to Whom.” You can see the abstract here at the Harvard Business Review. The article is utterly fantastic; you should get a hold of it.

What does this have to do with overemailing? Well, at the end of the article, the authors reveal an interesting, yet tangential, finding about email in their research. They found that purchase increases were tied to marketing communication in a strange way. It was not linear. In other words, more communication did not continually yield more purchasing. Instead, the authors found that above a certain threshold of communication, customers were put off. To quote the authors, “Clearly, many companies may be actively damaging their customer revenues in attempts to make sure that no opportunity for a sale is missed.”

The upshot is that they found that a data-driven approach to reducing marketing communication leads to “not only lower costs but to a revenue increase per customer.” When then tested this hypothesis using data-driven models and A/B testing at two client sites, the reduced communication strategy outperformed the traditional “blast ‘em” approach on both occasions. How much did it outperform the “blast ‘em” approach? I’m glad you asked, because these are the truly staggering numbers. For the B2B firm they worked with, the potential profit based on $1600 of additional revenue per customer, came to $320 million in additional profit. Now the cynical might say that this was mostly a reduction in cost. And I would have to admit that’s true. However, what the authors found was that the revenues for all product groups still increase, meaning that customers were spending, on average, $365 more with the reduced communication schedule. Similarly, at the financial services firm they worked with, the authors found an increase of $400 per customer using this data-based communication schedule.

To me, these results are unequivocal: sending too many emails not only is a waste of time and labor, it also hampers your sales. We all know it’s tempting to equate activity with results. But it may be better to turn your attention toward an intelligent use of your data to figure out who you really need to email and how many times you should email them.