IT departments are being overrun with big data – gigabytes, terabytes, petabytes of data – that is racing through their servers, often untamed and uncontrolled, with more data crowding onto the IT network every day, every hour.

While businesses can choose to keep all this data penned up, they are missing an opportunity to mine valuable business intelligence from their big data unless they harness it and analyze the information they have corralled, using that insight to gain a competitive edge. While a portion of this big data is well-organized information on products and finances, much of it is unstructured, from social media posts and photos to disparate research findings and customer feedback.

Microsoft has developed a range of solutions to manage big data and to help companies both understand and share valuable insights across their organizations. While Microsoft could offer its own big datasheet on these solutions, their value can be best appreciated in the success stories of Microsoft partners. We asked a handful of Microsoft partners to provide advice and insights on the ways big data can change a business in a big way, transforming a stampede into an organized flow of critical information.  Check out what Microsoft partners Infusion, BlueGranite and Neudesic have to say about big data.

Q: What does the term big data mean to you?  What do your customers think when they hear big data?

Infusion: Big data means the exploitation of very large data stores and the underlying technologies to manipulate those stores.  For our customers, big data can mean a variety of things, usually the utilization of large data sets for specific business purposes. For some clients, large equals several gigabytes; for others, it means mean terabytes.  For this reason, the definition, even within our firm, is fluid and inexact.

BlueGranite: Big data is an evolution of traditional data warehousing and business intelligence solutions to accommodate greater data volumes and increased data variety.  The goal remains the same: extracting insights from data to make better decisions and improve organizational effectiveness.

Customers are interested in incorporating a greater variety of data for analysis, including unstructured or semi-structured data sources, such as call records, survey responses, or social media content. 

Neudesic: Big data means we are coming into an age that requires us to understand more about how we can leverage extremely large amounts of data that has been created over the recent past and couple it with ongoing data creation and the ability to leverage it in the business to gain a true competitive advantage.

Most of our customers understand they have a lot of data that has been stored over the years, and they have been waiting for ways to dive into this data to make it into tribal knowledge. 

Q: What is the most common frustration customers experience when it comes to the amount of data and information their organization acquires?

BlueGranite: In some cases, customers are already collecting a great deal of the information they need to monitor and analyze their organization.  Unfortunately, often the data isn't integrated properly, or data quality is an issue, hindering their ability to utilize the data effectively. 

Another issue we see is the challenge IT teams face in keeping pace with changing business requirements.  The formal, corporate data warehouse and business intelligence solutions might not evolve quickly enough to address the specific information needs present in the various departments and teams across the business.  That's leading to a drive toward more ad-hoc and self-service solutions developed by power users and data analysts in tools like Excel and PowerPivot.  Most organizations should expect and plan for a mix of corporate-governed formal BI and user-driven ad-hoc and self- service BI solutions.

Q: How do you assess the right solution for each customer?

Neudesic: We apply an approach we have used time and time again to understand the business through a series of assessments, discussions and interviews with both the business and technical staff.  This helps us understand the applications that can be built with little technical difficulty and offer a high value to the business – leading to the creation and isolating of quick wins. 

Q: Typically how long does it take for a customer to realize the benefits of these solutions?

Neudesic: Once we have gone through our strategic roadmap assessment they will start seeing an immediate impact within the first few sprints of work using our agile process; this means weeks, not months. 

From the day we start to the day your end users are employing their first parts of the software, it can be as quick as eight weeks. This would be with a roadmap already laid out to follow for the next few years of their big data development.

Q: What’s the next trend or solution when it comes to big data?

Infusion: The current trend points to the integration of big data with the cloud. Gradually, companies are acclimatizing to the idea of storing sensitive data in the cloud, as providers such as Microsoft are addressing concerns around security and privacy in real time – removing the barriers that have caused companies to pause in the past.  Given the heavy hardware requirements around big data, and the simultaneous push from users wanting to access it more and more from a myriad of devices, it will be interesting to see how cloud technology might benefit this space, i.e., “Data on Demand.”

Q: Are certain industries more open than others to implementing solutions that allow them to work with big data (retail, financial services, manufacturing, etc.)?

Infusion: Data-intensive and mature industries, such as financial services and manufacturing, are most open to implementations around big data.  Such verticals, which depend on information for their survival, are more open to cutting-edge technologies to store and exploit it.  In particular, Infusion has done considerable work with Wall Street and Bay Street clients with end-results that are smart and intuitive.

BlueGranite: Certainly, the data-intensive industries, such as financial services, healthcare and retail, are adopting big data solutions.  Common scenarios in these industries include predicting customer churn, detecting fraud, and spotting patterns in top selling or lagging product sales.  We also see interesting solutions in manufacturing, such as analyzing a production process to predict and avoid failures.   

Neudesic: A number of industries that require quick actions, which means quick decisions like what you see in financial brokerage, manufacturing and healthcare, are also quickest to harness big data.  This will blend the business intelligence systems we see today, with real-time structured and unstructured data feeds, with a completely natural-speech-driven interface.