Category Archives: Challenges

Putting the puzzle of Big Data together
Jun 7 2012

Big data’s big requirements

There is no shortage of information on how to use parts of the most common big data solutions, like Hadoop. But what about the other pieces of the puzzle necessary to get real business value from this technology? For starters, there is a need to make decisions around:

  • Mobile strategy and its support
  • Web delivery
  • User interactivity/experience
  • Data support and operations
  • Security
  • Storage (for both big data and traditional SQL)
  • Scalability
  • Revenue Generation models
  • Filtering knowledge and noise
  • Integration into existing applications and processes

For these areas, there is less information available and just as important a need.

Fragmented solutions

In reality, there isn’t a single application development platform that covers a full solution. There are instead many choices, each having tradeoffs in usability and scalability.

Survivability

Also, there are solutions that have already come and gone in the short time big data has been in vogue. The question arises, “How does one now what will be around and still supported in two years’ time?”  Predicting the future popularity and support for the many available tools is a significant challenge.

Maturity

Open Source is an excellent way to ramp quickly and cheaply, but the solutions aren’t necessarily as mature as market requirements. As things stand today, it would be easy to get a few months into development of a solution before a particular tool’s shortcomings become apparent.  A great example would be that basic features like multi-language support are missing from some of the common solutions. Some lack authentication capabilities.

The UI

Lastly, user interfaces are no longer a common part of the equation. Less investment has been made in UI technologies in the haste to bring back-end capabilities to market. Avoiding these problems involves having enough knowledge of the space to make sound choices.

Big Data means broad solutions to complex problems. There are enormous opportunities ahead for those who consider the ecosystem beyond the big names, like Hadoop. 

May 26 2012

When the answer is Small Data

Big Data is advertised as the secret to unlocking actionable intelligence. Collecting and sifting through vast amounts of data finds the patterns that change everything. But is elusive ‘data in combination’ the answer that we should expect from analytics? Not necessarily.

More and more often, crunching large amounts of data gets to the opposite result: The answer to many questions is found in far less data than expected. Looking at what’s being answered by large-scale analytics today, the patterns that are emerging often show surprising results like:

  • A clothing retailer discovers that fit matters more than color, or vice versa
  • A wine recommendation engine proves that color matters more than most other attributes, but only when a customer is an occasional wine drinker
  • Only the three most recent transactions show a customer’s preferences and not their composite shopping history

Does that mean that Big Data itself is an overreaching goal for organizations? No. To understand that few factors matter, large data sets need to be created and analyzed. A Small Data answer still requires validation through data that often has velocity, volume and variety. Knowing for sure that Small Data is the answer is just as tricky, and maybe more so.

If we’re not careful, assuming complexity can blind us to the fact that simplicity is the real answer.

The key to today’s Big Data capabilities is to have an open mind and be ready for the answer that you don’t expect.