Planning for Big Data: Lessons Learned from Large Energy Utility Projects

by Dennis McDonald (1) and Tom Sipp (2)

Introduction

What can we learn from working with large energy utilities and their data that will help others plan and manage “big data” projects?

Let’s first consider how “big data” projects can differ from the large database systems we’re already accustomed to. They do have similarities. Implementing big data projects, especially those that are designed to connect with or support existing products or services, still requires careful planning, collaboration with stakeholders, capture and understanding of requirements, thorough testing before rollout, training, and support. Nothing new there.

But there are differences. Data volumes – potentially in the petabytes – may be such that traditional storage, networking, and data structuring approaches are insufficient. This requires consideration of how to supplement or replace traditional data management architectures and infrastructure which in turn raises the issue of possible duplication and additional costs.

We also have to consider skill sets and staffing requirements. Managing a new technology stack that integrates in different ways with legacy IT systems may require skills not currently on staff. The new data may need to be analyzed with modeling and visualization tools that differ from existing analytical and reporting tools. Add to this the management issues associated with servicing the business with these new tools: are we focusing on financial modeling? Customer demographics? Remote sensor data? Exploration for new markets? Uncovering opportunities with existing customers?

Are we counting on discovering and building on little-known patterns and trends that might — or might not — help us generate new products or new services? Are the potential users of these new “big data” services willing to put the time, energy, and money needed into developing and taking advantage of hoped-for new insights? Or will they demand immediate payback in the short term?

The major concern here is not just managing new technology but the degree to which the new technology has to be integrated with existing systems and processes. Data projects with a primarily analytical or exploratory focus will have different integration requirements from those that focus on supplementing or replacing existing systems and processes; it is the latter category that we are most concerned with in this paper.

Key Insights

In the energy utility business when we’re considering “smart meters” that report on and interact with household energy consumption data, we’re talking about potentially huge increases in data tied to energy consumption at the household level. This involves potentially significant changes to existing systems and business processes throughout the sponsoring utility, major customer support and training requirements — and significant startup costs.

Data growth challenges are significant and have to be viewed along at least three dimensions, as suggested by the Gartner Group:

• Volume - potentially in the petabytes. Where and how will the data be stored, in what format, and for how long?
• Velocity - the speed of data being collected and processed. What are the data sources and how are they managed? What channels will data travel over and what bandwidth will be required?
• Variety - the range of data types and varied sources. What changes or supplements to existing data dictionaries and metadata will be required?

Even if we do solve the data management and data handling issues, we still need to deal with system integration and process changes and costs. These will be driven as much by the need for new technology and skills as by the unique mix of systems and business processes we need to change or integrate with.

Example: Smart Meter Program for a Major Public Utility

As an early adopter of “smart grid” approaches to managing energy distribution, the management of one major public utility set the following four primary business objectives for its “SmartMeter” project:

1. Increase the accuracy of consumer billings.
2. Improve outage management through real-time notification of system problems.
3. Enable individual consumers to improve how they manage energy costs.
4. Reduce overall costs by automating the collection of energy usage information.

In the light of the massive amounts of data required to accomplish these goals, several data-related requirements were established:

• Accessibility – there must be a way to easily retrieve and utilize the information in a meaningful way.
• Privacy – Customer records must be protected. Also, an individual consumer must not be able to view or edit anyone else’s information.
• Protection – Both the operational (shorter-term “live” data) and archived files must be kept secure from both external and internal nefarious influences.

One of the major challenges pertained to the architecture, particularly the IT infrastructure. The amount of data that was anticipated was enormous – it was estimated that multiple petabytes (thousands of terabytes or millions of gigabytes) would need to be collected. Because of the volume of data, mass storage had to be planned and procured.

Collection of data from individual homes was also a challenge. Some homes had internet access so the data could be transmitted over the web. Others required a direct line using radio transmissions from the meter to the nearest device on a utility pole.

Lessons Learned

Experience gained in executing projects such as the above has relevance to many organizations. Here are some examples:

• Business Ownership. Responsibility for each new data element must be established early on. Defining and managing how data are defined, managed, and used cannot be left to chance as data volume, speed, and variety have significant cost and performance implications.
• Stakeholder Involvement. In the case of smart meter projects where data collection devices are placed in customer homes, project “stakeholders” will include both company employees as well as the customers themselves. Multiple business functions will be involved in defining requirements, in system design, in installation, and in operation. How customer involvement is managed will in turn drive public perceptions, technology adoption, and ultimate success.
• Funding. Project funding for massive data projects is always an issue. If customer rates will be impacted by the project and rates are determined by a governmental commission, and if significant investment must be made before going “live,” a major effort requiring involvement of multiple business functions will be necessary to obtain rate change approval and funding.
• Marketing. Individual consumers can be suspicious of new technologies, particularly when a device attached to their homes is involved. Communities might object on the grounds of accuracy, requiring extensive testing. Government officials might express concern that the privacy of their constituents will be compromised. Ameliorating such concerns in a society growing more aware of digital privacy issues may require significant time and resources.
• Business Process Change. Even if the technical side is planned and executed relatively well, the necessary changes to business processes will often encounter challenges. It is highly recommended to map out impacts to the various business functions before implementing a project of any magnitude. Remember: stakeholder involvement is a key element in any project’s success — especially in large “big data” projects.
• Risk and Change Management. Murphy’s Law – what can go wrong will go wrong – must be considered. Risk Management and Change Management must be important components of any project and are especially critical to the success of large, complex projects involving so many “moving parts.”

Conclusions

The smart meter project referenced above touched on many business processes, required engagement with many different stakeholders, and involved a significant front-end effort to secure funding.

Not all projects will be so complicated, such as those that leverage existing data or those which don’t touch on as many business units and business processes that require changes.

Initially it was the complications of the IT infrastructure changes that defined much of the complexity and scale of the smart meter project. New routes to collecting, distributing, and interacting with data had to be created. It wasn’t possible to just “piggyback” on existing data collection, management, and distribution methods. New ones had to be developed. Yet, it was also the coordination of many business process changes with multiple departments and functions that was required to be successful.

Implementing the smart meter project also required evolution of a changed relationship with the customer. This profoundly influenced the company’s operations and business processes. The ultimate success of the project was a testament to the efforts of many dedicated people working together — as will be the case with other “big data” projects that involve significant system and process changes.

An expanded version of this paper is available here.

(1) Dennis D. McDonald, Ph.D. is an independent consultant based in the Washington DC area. He has worked throughout the U.S. and in Europe, Egypt, and China. In addition to consulting company ownership and management his experience includes database publishing and data transformation, integration of large systems, corporate technology strategy, social media adoption, statistical research, and IT cost analysis. His web site is located at www.ddmcd.com and his email address is ddmcd@yahoo.com. 

(2) Tom Sipp is a Managing Partner with FSG Partners LLC, an IT Advisory Services comprised of former CIO’s and other IT Executives, all of whom have unparalleled experience within myriad industries, including manufacturing, financial services, and public utilities.  In addition to a rich history of IT leadership, Tom has directed and implemented many multi-dimensional projects containing complex scope, budget, data, and scheduling requirements. His email address is tsipp@fsg-partners.com.