Problems and Opportunities with Big Data: a Project Management Perspective
Introduction
Being able to make wise investments in “big data” capabilities may require more collaborative approaches to project management and decisionmaking than are currently being practiced. Two recent articles illustrate this:
The first article states an often heard complaint that, despite much marketing hype, real-world examples of big data value are few:
“It does not help when so-called experts suggest that an information revolution is changing—and will continue to change—the nature of the workplace itself. Big data projects can have a transformative effect on business operations and processes. Yet evidence of a knowledge lead revolt in the offices of global blue chips is thin on the ground. Instead large organizations seem to be drowning under a sea of information.”
The second article focuses on the opportunity side of the equation. It addresses how technology management needs to evolve in the face of growing technical data handling capabilities:
“From the top of the stack, more users want access to more data and more combinations. And from the bottom of the stack, more data is available than ever before—some aggregated, much of it not. The only way for data professionals to deal with pressure of heterogeneity from both the top and bottom of the stack is to embrace a new approach to managing data that blends operations and collaboration to organize and deliver data from many sources to many users reliably with the provenance required to support reproducible data flows.”
One of my own interests — making organizational data more open and accessible to both internal and external user groups — dovetails with both these perspectives.
Both views show how people are grappling with the challenge of using modern tools to make sense of the increasing volumes of data in digital form. If there is no clear understanding of what benefits will flow from big data investments, investment justification will be a challenge. Anyone who has ever sold a new tool or technique understands this. This is why practical demonstrations and understandable testimonials are so important.
It is clear that tools and technology are improving in ways that can make sophisticated data usage more accessible and immediate to those with the necessary knowledge. For example, the turnaround time for processing large amounts of data is dropping as new platforms and tools are introduced. Processing speed and the ability to combine analyses of different data types can be documented objectively and technically.
One question is, can we demonstrate the real world benefits of such techniques in meaningful ways without requiring management to possess data science degrees? Just talking about the benefits of speed and volume in technical terms is not enough. Showing how these benefits translate into useful insights and support for useful planning and decisionmaking is required.
Healthy skepticism
The first quotation above, in my view, reflects a healthy skepticism brought on by much of the hype surrounding big data. Such skepticism can be a good thing if it forces proponents to articulate the “whys” and “wherefores” instead of just the “hows” of big data. This need for proof is why supportive “case studies” are in such demand at meetings and conferences. Consultants and vendors understand this. This is why so many stories about big data “successes” are industry sponsored.
The second article’s emphasis on infrastructure is typical of tech-oriented solutions that focus more on the technology side of the equation. Yet there is more here than just a veneer of newness. Combining “big data” with “DevOps” makes great sense. I especially like the emphasis on the need for communication and collaboration which are critical to figuring out how best to grapple with taking advantage of powerful new tools.
Déjà vu
I admit to having a sense of déjà vu when reading articles like these. “Tech hype” has always been with us. Mature tech managers and their bosses long ago mastered the art of healthy skepticism when hearing vendors expound on new technologies. A “show me” attitude is a healthy one to promote here especially when price tags are high, a lot of changes are required, and business value may be difficult to pin down.
There’s always a lot of faith and trust required at this stage of technology adoption. That tech vendors are investing heavily in tools, advertising, marketing, and hype need not divert us from asking the tough questions. Inevitably open discussions about how best to take advantage of new tools will lead to serious discussions about strategy, governance, quality, and costs.
Pushback
Pushback can come in many forms ranging from good old-fashioned resistance to change (how many times have we heard, “You’ve got to change the culture!”) to major implosions of high-profile projects or vendors. What may be different this time around with big data may be the speed with which new tools are introduced and the seeming ease with which tools can be used to analyze and visualize vast amounts of data quickly.
Range of changes
New tools and technologies may also stimulate the need for changes to current management processes which, more often than not, generates pushback from those being changed. Adopting big data tools and process changes may be associated with a range of organizational changes including:
- The move to cloud services as a replacement for current infrastructure.
- The need to learn new tools and techniques.
- Resistance from business process owners who don’t want to change.
- Overemphasis on tools and technologies while shortchanging business and strategy.
- The need to align data services with core business needs, not just with “easy to do” and “low hanging fruit” initiatives.
While it’s certainly important for IT folks to understand the new data tools, employees and managers in all functional areas are impacted and need to articulate what they want from the data and the new tools.
This is where the need for collaboration and communication emerge along with the need for basic project management techniques and support. Stakeholders need to be identified and brought on board. Resources need to allocated and managed. Progress needs to be tracked and reported in some fashion. New products and services need to be aligned with goals and objectives that are important to the organization. Inevitable changes need to be addressed as learning takes place. Most important, project goals and objectives need to be clear and understandable to all.
No best practices
With that in mind I would have to say that, regarding the first article cited above, skepticism is healthy but this skepticism needs to be couched in terms that are specific not general. Searching for “best practices” and examples of whether or not other organizations are able to take advantage of big data applications may have little relevance to one’s own organization. Understanding one’s own needs and requirements must be the starting place, otherwise we run the risk of doing the same old things but with new tools.
One criticism of the second article is related to this. That is, while I do believe that we may need to re-think how we manage infrastructure and IT services to take advantage of new tools, services, and platforms, we also need to make sure that the needs of the organization are reflected in how priorities are set and initiatives managed. Again, that’s basic project management but it points out the need for communication and collaboration in how priorities are set and how solutions are implemented.
Whether we call this “DevOps” or “DataOps” is not the issue. The issue is whether or not we can effectively manage projects and programs that take advantage of big data while involving all necessary stakeholders throughout every point in the data value chain, not just IT staff and data analysts.
Data literacy and strategic alignment
If it is inevitable in today’s fast paced world that many fingers are going to be stuck in the “data management pie,” we need to make sure that the heads operating those fingers have a basic common understanding of how data are generated, managed, and used; let’s call this “data literacy.” Without this participants will talk past each other with the end result being inefficiency, blind alleys, and disappointment.
The connection between better data services needs to be aligned clearly with the organization’s goals and objectives. Again, this is a basic requirement for effective project management. Management, IT, and data analytics experts need to communicate and collaborate right from the start so that everyone is on the same page.
Acknowledgement: The author is grateful to Julia Glidden of 21c Consultancy Ltd for her helpful comments on an earlier draft of this article.
Copyright (c) 2015 by Dennis D. McDonald. For articles like this scroll down. For information about my consulting go here.
