Last week while driving to a meeting, I had an epiphany. Those familiar with the state of highways in the Bay Area in the morning rush hour can relate to this. I had to be in Burlingame on the water’s side of 101 at 9:30am. However, it was 8:30am before I could leave from Sunnyvale near 280 south of 85. At that time, no matter which highway I took (280, 85, 92, 101) one hour to get to Burlingame was an impossible task. Yet with real time traffic updates on the GPS and timely decision making about getting off and back on the highways at the right exits, I was able to make it to the cafe in Burlingame at 9:20am. Phew! Well let me assure you that it was no small feat. Now to the epiphany.
Which companies are involved in enabling this efficiency to ordinary commuters like us? It used to be not too long ago, that the radio stations had their helicopters roaming the skies to give traffic updates to their listeners. And once we got inside our cars, we had no way of knowing if just a few miles further down the road there was an accident and a traffic jam.
Let’s understand the value chain at work here. The following categories of companies are involved in making this magic in the traffic and maps ecosystem. (Ref: How GPS Real-Time Traffic Works)
Traffic/Navigation Ecosystem Value Chain
What is enabling these companies? In the last few years, ability to collect all types of data across the Internet and use it for analytics has exploded. With the Internet of Things wave, this is only going to grow exponentially. We have long ago left behind the world of megabytes, gigabytes and terabytes. Petabytes and exabytes are the new currencies of data.
Technology Platform Shifts
Two technology platform shifts have fueled this transition – Cloud Computing and Big Data. The Cloud is essentially a metaphor for the Internet, but broadly speaking, it relates to a set of networked services, served up by virtual machines, so that these services can be scaled up or down transparently to the application developer. Big data, in turn, relates to the collection of data sets, both structured and unstructured, so large, that traditional mechanisms of relational database systems and warehousing were incapable of handling. Newer data management technologies had to evolve to be able to handle such data sets. In come, Hadoop, Hive, Map/Reduce, NoSQL, in-memory databases, etc.
Big Data Infrastructure Value Chain
There is another value chain at play at the infrastructure layer. Now, over time, layers in a value chain consolidate or collapse or get vertically integrated for the sake of efficiencies. Companies like IBM and EMC try to go across layers of the stack from physical storage to analytics to applications. This is a natural evolution of the stack, driven by the ambitions of the players at each level to better control their own destiny. (There are probably more companies in each of these categories, and they may already be present in multiple layers of the stack, so I am happy to receive feedback, corrections, etc.)
Each of the application ecosystems above is already deep into figuring out how to take advantage of the avalanche of data that is available every single moment. Every GPS coordinate, every mouse click, every location check-in, every credit card swipe, every login, and every non-user-generated event like a cell phone connecting to the tower is in some database somewhere, waiting to be correlated with history of past behavior and current adjacencies to derive some meaningful insight that can predict something of value for somebody.
So if you are a real-estate application ecosystem, big data in this context means trying to understanding who buys and sells, from and to whom, what, how, when, why, where. This has implications for the ecosystem value chain, from realtors, real estate agents, home buyers and sellers, etc. Real estate companies are trying to predict which houses are likely to come on to the market in the next three months and can manage their advertising efforts accordingly. (Ref: What big data means for the real-estate industry)
Big Data Product Management
The Last Mile Problem