Like many industry buzzwords, Big Data means many things to many people. In short the concept refers to problems related to ‘information overload’. But let’s explore this concept in detail.

Big Data refers to collections of information that have become unwieldy to store, too massive to sort, and too complex to easily summarize while sometimes growing at exponential rates.  This combination of factors makes it impossible for humans to deal with in a timely manner.

An entire industry has sprung up around these problems - this is the Big Data space we hear so much about these days.

Some ‘big data’ companies focus on the back-end infrastructure needed to help solve these data problems (companies like Vertica, 10Gen, or Cloudera). Other companies focus on collecting data, adding structure to it, and then serving it to users who don’t have the time to do all that (companies like Factual). Some companies focus on analytics and making such data sets actionable by humans (companies like DataSift or Palantir).

Small Data, then, is the last mile. If you picture a pyramid with the infrastructure guys on the bottom (meaning they provide the foundation for the rest) with the data warehouse guys a layer above, and the analytics guys above that, Small Data companies would represent a subsection of the industry that sits at the tip of the pyramid. These companies focus on end user experience with apps and visualization products that appear to be simpler than they actually are.

Thus the term ‘small data’ is a bit of a misnomer. Small data companies often do quite a bit of computational heavy lifting, whether they extend all the way down to the base of the pyramid, or stop somewhere in between.

At MetaLayer, we’re very much focused on the end user’s experience, but we’ve also made it easy for professionals to take our simple tools and sit them on top of big data infrastructure in Enterprise environments.  Big data made small.