Search vs Choice – A new type of engine for a new kind of age!

Published June 2, 2014   |   
Suresh Shankar

Around 15 years ago, the defining problem of the Internet age was how I find the information I need? This gave birth to search engines and Google’s great vision of “organizing the world’s information”.

A decade later, as Google continues to improve its search with innovations such as Knowledge Graph with the explosion of information, we have reached an inflection point – the defining problem of today is no longer search or even vertical search.

If we look at our lives today, it is the proliferation of choice that is, counter-intuitively, the biggest headache faced by most people. We live in the Age of More, an age characterized by more sources of influence, more devices, a more real-time tsunami of data, more transparency, more channels of communication….

As Barry Schwarz puts it, more is rapidly become less, and we are faced with the paradox of choice. Too many choices has led to decision paralysis, postponed moments of reckoning, increased expectations, decreased satisfaction in life. In other words, choice becomes misery.

The defining problem of today is ‘how do I find the choice that’s right for me’? This will give rise to the next era of disruptive technologies and companies. Over the next few years, you will hear more about choice engines and “organizing the world’s choices”.

How exactly do you organize the world’s choices?

Choice is inherently complex. As a consumer, you are guided in general by four major factors in any decision you make.

First, your taste drives your choices. As you begin your journey through the various stages of your purchase decision, especially when your interest is triggered, or as you are searching for information or considering what you want from your purchase, you consider a large pool of choices – largely based on your tastes and preferences. Examples of these are – I like Asian food, I like fine dining and not fast food, I prefer places that serve good wine… In general, it is highly likely that your actual choice will be driven and constrained by these tastes.

Second, your past behavior drives choice. Typically your behavior, when you have actually bought something, is a strong indicator that you like something enough to have chosen it and paid money for it. Repetitive past behavior is a stronger indicator as is frequency of occurrence. Examples include, I eat every Friday at XYZ sushi bar, I buy ABC wine regularly from my local wine merchant….

Third, influence drives choice. Our tastes and our behaviours in many cases are driven by others, whether they be friends, family or experts we trust. And influence can be general, as in I trust the New York Times on all matters of public policy, or specific, as in I trust Sonia’s choice in movies, I like Maya’s advice on food….

A final point: one’s current context drives choice. The time of day, the weather, the day, the location are all filters that are instrumental in the final decision we make. Examples include ‘I prefer eating lunch close to office on weekdays’, ‘It’s a hot Saturday, let me get a beer’ etc.

Every decision you make is generally a blend of these different influences. The importance you attach to each of these factors varies from decision to decision. And the information you have to help arrive at the decision is generally very different.

Consider a typical decision – you have just arrived in a new city after a 6 hour flight, it’s 6.30 PM and you want to get a decent vegetarian meal. What do you do?

You fire up Google, search for good restaurants in that city, and filter them by type, price, location etc. You search the review sites and blogs, both international and local for reviews and ratings on what might be a decent choice. You ask the hotel concierge or ask some friends on Facebook, email or SMS for ideas. You whip out your mobile phone and look for stuff that is ‘nearby’.

A good 30-60 minutes later, tired and weighed down by information overload, you do what most would in the coffee shop of the hotel you are staying at, or even order room service. The good meal you planned will have to wait till tomorrow.

Sounds familiar I’m sure.

The challenge you face is not lack of information. The challenge is to filter the right information, for you, the decision-maker. To do so, you need to spend huge amount of effort and time to navigate the choices.

But the one thing we do not have enough of today is time. So what does a Choice Engine do? And how does it solve this problem?

With the use of cutting-edge technology, a Choice Engine first tries to recognize the underlying factors behind how choices are actually made.

Choices are determined by a person’s taste, influence, context and behaviour. To form meaningful conclusions about one’s choice, the engine needs to combine these factors. It is a complex task. Because we cannot easily get access to the data relating to one’s taste, influence, context and behaviour. Even if we do, they may not be in a structured form for the machine to understand.

However, technology allows us today to arrive at a certain level of precision in understanding one’s tastes by analyzing one’s opinions and views that he/she scatters daily on the web in the form of reviews, likes, ratings, shares, comments etc about food, movies, books, restaurants, Pinterest, Facebook, Twitter etc. These are the testimonies of our ‘tastes’. Although there is a tendency among some experts to classify them as superficial or skin-deep, sheer volume of such a trail can be undoubtedly considered as a strong indication of our tastes.

Influence is someone/something that shapes our tastes or behaviour. This type of data can be gathered from what we read and whom we follow on the net and social media. However, influence can be inferred by the frequency and level of two-way communication between two parties. If I tend to share something on football with David and he responds or shares it, it is likely we have some influence on each other’s views on sports. (Nevertheless, it must be said that it is harder to infer who influences whom without substantial data).

Coming to behaviour, it is the best and most researched part of choice. In general, a behaviour in the past is considered to be the best predictor of a future behaviour, because if a behaviour in the past involves conscious decision making, it is likely to repeat in the future as well. Generally, enterprises like banks, retailers, telcos, airlines, hotels, consumer good companies etc, own large amounts of behaviour data of the customers.

Technology makes it easy today to crawl the web and collect tremendous amount of data about tastes and influences. Those data can also be collected by establishing data partnerships with aggregators and from individuals with use of explicit permissions.

Although data acquisition is relatively easy, combining data is a bigger challenge in terms of both scale and complexity. Perhaps, the best way to actually handle these data sets is to construct a Taste Graph that allows us to understand the linear and non-linear relationships between all the entities or products in these data sets, and the relative strength or affinity between each of them. Once built, it is a data monster. Just imagine a massive graph where you can see the relationships between any restaurant, any movie, book or any type of shirt, anywhere in a country.

This isn’t over yet. Adding Influence or the level of Trust is even harder. Essentially, we need to compute the likelihood that if person A likes something, B will like the same. Simple correlations based on exhibited common behaviours is one basic way. But experts like Dr Jennifer Goldbeck with her Tidal Trust algorithm have gone beyond and established ways in which we can measure the level of trust across many levels of separation.

Combining taste and Influence data with behaviour presents a lesser technical challenge, but a critical one, to make real the promise of Choice Engines. For years, the company you bank with or shop at has a deep view of your behaviour inside the organization, but a very limited view of you as a person.

Now, adding enterprise behaviour data to the Taste Graph will allow a more holistic view of a customer’s likely behaviour, tastes and influence, something so far available to very few companies like Amazon.

This resultant Choice Graph can be used to predict much smarter choices for each individual consumer. This graph will do two things better than anything available today.

1. It will enable the discovery of a wider set of tastes and influences for each consumer than are known today (based on the external Taste Graph).

2. It will allow for a deeper and more predictive personalisation for each such consumer, based on the deep enterprise behaviour data available to a company.

In essence, such a Choice Graph will allow a bank to not only know that an individual like Zara likes certain types of movies, food , nd books, it also allows them to conclude that she will most likely shop at a certain brand of fashion (though she may never have bought any of these before using a bank product).

Finally, let’s add an important layer of context. Context data comes from the sensors of every device we carry, such as phone, tablet, computer and wearable sensors. They can tell us about the location, time, date, weather and activity we indulge in. They can track the digital trail we leave behind including every site we visit, every click we make. Increasingly, with the proliferation of cameras, they can even capture every gesture or face we make.

By launching apps, placing cookies or using FB or Twitter pages, the enterprise can now use the context data that is automatically collected in such engines, to narrow down the choices that are relevant for Zara and present them to her as a simpler set of choices.

So, how does all these benefit Zara? Going back to my previous post… Zara has just arrived in a new city. She is hungry and looking for a place to eat. She fires up her Bank’s SimplerChoices app. It knows she is vegetarian, that she doesn’t like spicy food. The sensors tell it where she is. She looks at the screen and there are 4 choices of restaurants that serve vegetarian food – one recommended by a friend of hers who has been there recently, one reviewed well in her favourite newspaper, another that has rave reviews from other vegetarians, and a fourth that is really close by and affordable. She clicks on two of them to get to know more, and after 5 minutes, she’s made her choice.

Easy. Simple. Quick. And really useful.

For Zara, yes. For the bank, yes, because they have delivered a great experience to their customer.

Is such a Choice Engine within our reach today?

It is. It is tough to build. The challenges involved, though non-trivial, are mathematical and computational, but are very solvable. It has to do all of the math and computing, starting from ingesting massive data sets, to combining them intelligently to form Taste and Influence Graphs, to mixing them with enterprise data, to computing all possible and relevant choices for each customer and presenting a simpler set of prioritized choices based on the current context of the customer.

It is precisely because of this power that Choice Engines have a tremendous amount of promise. But a good choice engine needs to go beyond these to strike a balance between privacy and personalization, and allow enterprises to ensure their internal data is kept confidential. These are major issues worthy of a separate post.

I am sure you have many questions. Are Choice Engines the same as Recommendation Engines? How will you ensure my enterprise data is kept confidential? How will I benefit as a CMO or business leader? Keep watching this space for some answers.