Big Data Project Management Research Report #1: Setting the Stage

September 4, 2015 Dennis D. McDonald

Introduction

I’ve been doing some preliminary research on what makes “big data” projects unique and what such differences might mean for successfully planning and managing projects where the goal is to make large volumes data accessible and useful.

I’ve started by asking three basic questions in a series of informal interviews:

Do you think that too much attention is being paid to big data tools and not enough attention to big data project management?
Do you think that big data projects pose unique or special challenges to traditional approaches to project management?
What advice would you give to a project manager tasked with starting up a big data program in an organization that operates with a mix of legacy and newer cloud-based systems?

Some of what I’ve learned so far does seem to be unique to data intensive projects. At the same time, many of the challenges posed by “big data” projects will be recognizable to project managers based on previous experience with other large or complex projects.

Scope

My focus is not on data science tools or on data analytics but on the work that needs to be done – perhaps “behind the scenes” — to make tools and analytics usable and manageable. This work might involve changes to an organization’s IT infrastructure or even a shift to using cloud based resources. Addition of staff with data-specific skills may also be required.

At minimum, projects – groups of related tasks that support accomplishment of a defined set of goals and objectives – will have to be planned, ramped up, and implemented.

Also, my initial focus has not been on changing the “culture” of a sponsoring organization by making it more “analytics-friendly” or “data-driven.” While those may be real and important goals, my initial focus has been more tactical: figuring out what project managers need to do to deliver value by making an organization’s data better organized, analyzed, managed, and used.

What follows are some initial impressions from my interviews with other project managers as well as some hypotheses.

Preliminary Findings

Below are some initial observations about planning and managing successful data intensive projects. Also included are some hypotheses to address in future research:

A. Where will the value generated by the project be located? Is the focus of the project on (a) internal IT systems and processes, (b) on the business units served by the IT department, or (c) on the provision of direct service to the organization’s customers or users? Each of these focal points will impact how data and data related systems and processes interact with the project and how associated work will be managed and communicated.

Hypothesis: Projects that focus on delivering value outside systems and processes controlled primarily by ITmay require more complex planning and governance methods than those more externally focused. Business justification of more externally focused projects may have to extend beyond cost-saving (as is the case with some infrastructure focused projects) to metrics related more to organizational performance such as profitability or financial ROI.

B. Is the project “self contained” or part of an ongoing operation or program? A “traditional” type of systems-focused project generates a deliverable that combines data and software in support of a changed process of some sort. Sometimes the deliverable is an upgrade to an existing system or process. At other times the deliverable is a replacement involving significant changes to an ongoing operation or process. Some “big data” projects might make use of data in new or innovative ways but in the end might end up supporting fairly traditional users and decisionmaking processes. Other projects might require, in order to move out of the prototype or “sandbox” stage, a major change to how the organization manages or makes use of data.

Hypothesis: The more that existing systems and processes will be changed by a big data project the harder the project will be to (a) justify and (b) implement.

C. Where are we in the project’s lifecycle? Are we in the planning stages of starting a big data project or program from the ground up? Have we gotten the go-ahead and are in the process of ramping up the project (e.g., gathering resources, communicating with stakeholders, identifying potential risks, developing a communication plan, etc.) Or, are we coming in “from the outside” as internal or external consultants and are being challenged with delivering value viaa project or program that is already underway?

Hypothesis: Project planning and justification will vary significantly depending on whether the project is being started from the ground up or is being “added on” to ongoing projects or programs where the value of an added “big data” component is being considered.

D. What is the level of the organization’s “project management maturity”? Does the organization already support a formalized approach to project management? Examples of such formalism may include having (a) a formal process for project justification, (b) an established process for project review and reporting, and/or (c) a project management organization (PMO) that helps project managers to administer their projects. Or, does the organization treat project management as something technical experts are assumed to be able to do as part of their existing jobs?

Hypothesis: Organizations that already have (and actually use) established project management processes will be able to accelerate the adoption of new “big data” systems and processes. Whether they are able to accelerate development of new systems and processes that focus on big data analytics is another question.

E. What is the level of the organization’s “data management maturity”? Does the organization already support a formalized approach to metadata and data standards development and administration? Are governance processes in place that address related data management issues regardless of departmental boundaries? Is a formal “data stewardship” program in place whereby individuals in different departments coordinate how changes to data definitions and standards are managed and adopted? Has the organization established a “chief data officer” position with real clout? Does top management view data as a strategic resource – and knows what that means?

Hypothesis: The lower the organization’s data management maturity is, the more the project will have to do in terms of basic data management in order to get the job done. This can be viewed as a problem as well as an opportunity.

Discussion

As suggested earlier I’m not focusing – yet — on changes to the organization’s “culture” or to its existing view of data as a resource fundamental to how it does its work. There will be a wide variation in these depending on how important data already are to the organization’s mission. A single project, even if data-intensive, is not going to change that in the short term.

Organizations that already gather and manipulate data as a core service (e.g., social networks, e-commerce firms, government agencies that concentrate on managing environmental or financial data, etc.) may be better positioned to manipulate their core data to support analysis, modeling, and predictive applications than more traditional organizations that focus on manufacturing or supporting physical products.

Organizations that have a wide range of internal and external data resources to draw on may have to spend a lot on making data available and accessible for big data analytics. Therein lies one of the challenges of focusing on data as a deliverable: not knowing exactly what will come out of the analysis, especially if the project is pitched as requiring a fundamental change in how data are managed or manipulated. (Such uncertainty may justify incorporating a risk analysis in the project’s initial planning.)

So far my research has not focused on tools but clearly how tools are managed and used will make a big difference, especially tools that either (a) make it easier and less expensive to ingest or prepare data for analysis or (b) tools that allow for us to “make sense” from unstructured data. (I admit that past experience managing projects involving a lot of conversion and standardization of structured and unstructured data makes me wary of solutions that promise significant reduction in manual processing but I am open to being convinced otherwise!)

www.ddmcd.com

Big Data Project Management Research Report #1: Setting the Stage

Introduction

Scope

Preliminary Findings

Discussion

Related reading