Built in the 1950s, California’s aqueduct is an engineering marvel that transports water from Northern California mountain ranges into thirsty coastal communities. But faced with a potentially lasting drought, California’s aqueduct is running below capacity as there’s not enough water coming from sources. In terms of big data, just the opposite is likely happening in your organization—too much big data, overflowing the river banks and causing havoc. And it’s only going from bad to worse.
The California aqueduct is a thing of beauty. As described in an Atlantic magazine article;
“A network of rivers, tributaries, and canals deliver runoff from the Sierra Mountain Range’s snowpack to massive pumps at the southern end of the San Joaquin Delta.” From there, these hydraulic pumps push water to California cities via a forty four mile aqueduct that traverses the state and dumps into various local reservoirs.
You likely have something analogous to a big data aqueduct in your organization. For example, source systems kick off data in various formats, which probably go through some refining process and end up in relational format. Excess digital exhaust is conceivably kept in compressed storage onsite or a remote location. It’s a continual process whereby data are continually ingested, stored, moved, processed, monitored and analyzed throughout your organization.
But with big data, there’s simply too much of it coming your way. Author James Gleick describes it this way; “The information produced and consumed by humankind used to vanish—that was the norm, the default. The sights, the sounds, the songs, the spoken word just melted away. Now expectations have inverted. Everything may be recorded and preserved, at least potentially: every musical performance; every crime in a shop, elevator, or city street; every volcano or tsunami on the remotest shore.” In short, everything that can be recorded is fair game, and likely sits on a server somewhere in the world.
So what got us here in terms of IT architecture isn’t going to be able to handle the immense data flood coming our way without a serious upgrade in terms of capability and alignment.
IT architecture can essentially be thought of as a view from above, or a blueprint of various structures and components and how they function together. In this context, we’re concerned with what an overall blueprint of business, information, applications and systems looks like today and what it needs to look like to meet future business needs.
We need a rethink of our architectural approaches for big data. To be sure, some companies—maybe 10%–will never need to harness multi-structured data types. They may never need to dabble with or implement open source technologies. To recommend some sort of “big data” architecture for these types of companies is counter-productive.
However, the other 90% of companies are waking up and realizing that today’s IT architecture and infrastructure won’t be able to meet their future needs. These companies desperately need to assess their current situation and future business needs, and then design an architecture that will deliver insights from all data types, not just those that fit neatly into relational rows and/or columns.
The big data onslaught will continue for the foreseeable future, and is only going to grow more intense from exponential data growth. But here’s the challenge: the human mind tends to think linearly—we simply don’t know how to plan for, much less capitalize on this exponential data growth. As such, the business, information, application and systems infrastructures—at most companies—aren’t equipped to cope with, much less harness the coming big data flood.
Want to be prepared? It’s important to take a fresh look at your existing IT architecture—and make sure that your data management, data processing, development tools, integration and analytic systems are up to snuff. And whatever your future plans are, consider doubling down on them.
Until convincing proof shows otherwise, it’s simply too risky not to have a well thought out plan to cope with stormy days ahead of too much big data.