The Tip of the Spear: Connecting Big Data Project Management with Enterprise Data Strategy
Prepared for the ATARC Federal Big Data Summit, Dec. 8, 2015, Marriott Metro Center, Washington, DC. Click or tap here to download a .pdf of this article.
Introduction
“If data analysis is Big Data’s “tip of the spear” when it comes to delivering data-dependent value to customers or clients, we also must address how that spear is shaped, sharpened, aimed, and thrown – and, of course, whether or not it hits its intended target. We also want the processes associated with throwing that spear to be both effective and efficient.”
In Meeting the Mission of Transportation Safety, Richard McKinney, U.S. Department of Transportation’s CIO, describes four components for what can be called an “enterprise data strategy”:
- Data governance
- Data sharing
- Data standards
- Data analysis
He also mentions additional factors relevant to DOT’s data strategy:
- The volume of data is increasing and we need to be ready for it.
- Managing data is not the same as analyzing it.
- We need to be thinking now about what type of analysis we need to be doing and what resources will be needed to do the analysis.
Based on the 20+ personal, telephone, and email interviews I’ve conducted so far as part of my big data project management research I would add a fourth item to McKinney’s list:
4. We need to devote at least as much time to planning and managing the people and business processes that make data analysis possible as we devote to the analysis process itself and the technologies that support it.
This Paper’s Target Audience
This paper is for managers and executives who are looking for ways to add “big data” benefits to their organization’s operations.
Perhaps the organization is experiencing a significant increase in the type and volume of data it is generating and wants to make sure it points its evolving analytical capabilities at addressing its most important goals, objectives, or problems. Or, perhaps the organization just wants to do a better job of making use of the data it already has.
Tip of the Spear
If data analysis is Big Data’s “tip of the spear” when it comes to delivering data-driven value to customers or clients, we also must address how that spear is shaped, sharpened, aimed, and thrown – and, of course, whether or not it hits its intended target.
Management needs to involve both business and technical resources in its planning process. While a variety of technical skills and capabilities may be required to develop and govern an effective data analytics program, the focus of this and the other reports in this seriesis not on technology or analysis tools per se but on how to make sure that data analysis and data governance processes are driven by business requirements and the delivery of practical benefits to the organization.
Challenges
Making the data analysis process – the tip of the Big Data spear — effective and efficient is where good project planning and management come in. Challenges to doing this in connection with data intensive projects are many and include:
- Siloes. Data are often generated and managed in system- or mission-specific siloes. As a result, creating and implementing an effective enterprise-level data strategy that rises above and encompasses multiple programs, systems, and/or missions requires not just data analysis skills but a mix of technical, organizational, and political skills – not just good “project management.”
- Sharing. Making data accessible and useful often means that data need to be shared with systems and processes outside the control of those who “own” the data to be analyzed. Key steps in sharing data are that (a) data need to be identified and inventoried, and (b) technical and business ownership of the inventories data must be determined. In many organizations this inventorying is easier said than done and may require both manual and automated approaches to creating the necessary inventories.
- Standards. Efficient and sustainable analysis of data and metadata may require development or implementation of data standards. Existence and use of such standards differs by industry, data type, and system. The costs for developing and adopting standards to facilitate data sharing and analysis will also vary and may have cost and schedule implications at the project, program, enterprise, and industry or community levels.
- Delivering value. Modern data analysis tools and techniques provide mechanisms to identify patterns and trends from the increasing volumes of data generated by a steadily widening variety of data capture mechanisms. Challenges in predicting what will be found when data are analyzed places a premium on making sure we are asking the right questions. This in turn impacts our ability to justify project expenditures in advance.
Portfolio Management
Responding to the above challenges requires not only project management skills but also a project planning process that takes into consideration the organization’s goals and objectives.
As one of my interviewees suggested, the challenge faced in complex “big data” projects has just as much if not more to do with overall strategy and “portfolio management” as with how individual projects are planned and managed. Effectively designing and governing a portfolio of projects and processes requires not only an understanding of how the portfolio supports (relates to, is aligned with, interacts with) the organization’s objectives. It should also incorporate a rational process for defining project requirements and then governing how the organization’s resources are managed and applied over time.
Given how pervasive and fundamental data are to an organization’s operation, skill in data science and analytics is a necessary element but will not be in many cases be a sufficient contributor to success. Technical and analytical skills must be accompanied by effective planning, oversight, and management in order to ensure that the data analysis “spear” is being thrown in the right direction. While it is not unusual to assign responsibility for big data projects to the IT department, both business and functional leaders from outside will also need to be involved.
Delivering Value Quickly
Ideally a defined portfolio of projects will support an organization’s strategic plan and the goals or missions the organization is charged with pursuing. In the real world, though, we can’t spend all our time planning, we may also need to “get tactical” by delivering value to the customer or client as quickly as possible.
In organizations that are not historically “data centric” or where management and staff have a low level of data literacy, an early demonstration of value from a targeted data analysis initiative will be important.
Balancing Tactics and Strategy
Unfortunately, challenges such as those identified above in many cases cannot always be addressed effectively in tactically focused short term projects. For example, convincing “data silo” owners to cooperate may take time given how the organization has been traditionally structured and managed.
Attention to enterprise-level data strategy while delivering useful results in the short term has implications beyond what is being attempted in an individual project’s scope. Treating data as an enterprise resource may even require changes to how the enterprise itself is managed.
As we all know, it’s not unusual for change to be resisted!
It’s not unusual for a tactically-focused project that’s delivering a practical data-based deliverable to uncover the need for a more global (or strategic) approach to managing data, metadata, data security, privacy, or data quality. In such instances it makes sense for the project manager, when communicating with project stakeholders to clearly identify strategic concerns along with reporting on current work. Experienced project managers will already be doing this.
An effective enterprise level data strategy will be one that balances the management of a portfolio of individual data intensive “agile” projects with parallel development of an upgraded enterprise data strategy and governance process. Doing one without the other could have negative consequences, for example:
- Focusing only on a narrowly defined data intensive analytics project by itself may generate immediate value through frequent useful deliverables but may not address underlying technical process issues that impact long-term efficiency and sustainability.
- Focusing only on an enterprise data strategy without delivering tactical benefits reduces the possibility that that less data-savvy managers understand the “big picture” down the road.
As experienced project managers know, concentrating on “quick and dirty” or “low hanging fruit” when under the gun to deliver value to a client in the short term can generate short term benefits.
They also know this approach can actually increase costs over time if strategic data management issues related to data standards or quality are repeatedly kicked “down the road.”
Also, delivering a “strategy” without also engaging users in development of real-world analytical deliverables might also mean that strategically important recommendations ends up gathering dust on the shelf somewhere.
Communication Strategy
As experienced project managers understand, one of the most important ingredients in successful project management is communication among project staff, communication with the client, and communication with stakeholders. Even when focusing on delivering incremental value quickly, we need communications about project activities, especially among key stakeholders, to focus both on tactical as well as strategic objectives. This may require accommodating a variety of communication styles as well as different levels of data and analytical literacy, especially when both business-focused and technology- or analytics-focused staff are involved.
The project manager must think carefully about what it will take not just to deliver a useful result now but also what it will take to make sure that participants and stakeholders understand the meaning and significance of what is being delivered. This may be straightforward in organizations that are already engaged in heavily data-centric activities. In other types of organizations more explanation and learning will be required.
Where to Start
A planning process that takes into account how the unique needs of the organization interact with an improved data management strategy is what is needed. Working without a plan that links improved data management with both tactical and strategic benefits is a recipe for, at best, wasted time or money. At worst, working without such a plan can lead to financial or organizational disaster.
In the case of big data, a revolution is occurring both in the management of steadily increasing volumes of data and in how data are organized, stored, and analyzed. Even managers of very traditional data collection and publishing operations are seeking ways to improve how they manage and use data based on what they hear about the potential for “data science” and “data engineering.”
What I’ve been finding, though, is not that project management practices need to change. Instead, what organizations want and need to accomplish through improved data access and management needs to be better defined — and ideally, quantified — so that investments in time, technology, and other data-related resources are planned and made wisely.
This means that the “front end” of the project and program planning process needs to address fundamentals such as business requirements, business strategy, use cases, return on investment, and the related issues of security, privacy, standardization, competition, and innovation.
Whatever specifics are being addressed in the short term, the strategic implications of improved data management need to be addressed as well. We may find, for example, that what is needed initially is not a move to an entirely new data management architecture but the creation of business processes that make better use of the data resources that currently exist.
What’s Next
Among other things I’ll be turning my attention next posts in this series to address this planning process more specifically, possibly beginning with the relationship between communication and requirements definition.
If you or a client think you need help in this area, please let me know; my contact information is below. I will be happy to explore with you confidentially how I can help you to rapidly put these ideas to work in your own planning process.
Acknowledgements:
Thanks are due the following for sharing their thoughts with me about big data project management: Aldo Bello, Kirk Borne, Clive Boulton, Doug Brockway, Ana Ferreras, Keith Gates, Douglas Glenn, Jennifer Goodwin, Jason Hare, Christina Ho, Randy Howard, Catherine Ives, Ian Kalin, Michael Kaplan, Jim Lola, David McClure, Jim McLennan, Trevor Monroe, Brian Pagels, John Parkinson, Dan Ruggles, Nelson Searles, Sankar Subramanian, and Tom Suder.
Copyright 2015 by Dennis D. McDonald, Ph.D.
