Orchestrating AI Large Language Model Tools to Enhance Project Management Document Creation
By Dennis D. McDonald, Ph.D.* & Michael Kaplan, PMP**
Overview
We are investigating the use of rapidly evolving large language model tools such as ChatGPT and GPT-4 to help automate creation of project planning documents. We’re encouraged that use of such tools can (a) speed up as well as reduce the cost of developing detailed project documentation, and (b) incorporate intelligence and experience potentially available from a variety of reliable sources. We also recognize the need to incorporate human review points throughout the documentation process to ensure both accuracy and adequate management and technical oversight.
Background
Many tech workers find the creation of project documentation to be both onerous and boring. Nevertheless, situations exist where the lack of detailed project documentation creates risk or delay, such as demonstrating compliance with government-imposed safety or security requirements, onboarding new or replacement staff, training customer support staff when systems or operations are introduced or changed, or ramping up a new project based on a recently awarded contract. Management and project reporting can also suffer if baseline processes aren’t planned and documented correctly. Speeding up and reducing the cost of creating useful project documentation seems a practical and down to earth use of these recent technologies, especially if they can help when inevitable changes occur after project start that require modification of the original plans.
What do we mean by “Document”?
A “document” is a package of information in human-readable form that can take physical form (e.g., printed on paper) or be presented on a screen (e.g., as a slide deck containing a mix of information in a variety of formats). The contents of the document may include text, images, tables, spreadsheets, links to external data, or numeric data.
Challenge: Customization
One challenge is ensuring that such documentation is not boringly (and uselessly) generic but instead is meaningfully customized to the unique requirements of the project or program being planned and documented. Fortunately, there are ways to interact both manually and programmatically with tools such as ChatGPT and GPT-4 to incorporate up-to-date details that reflect both current as well as application specific requirements. Tools and apps are already coming to market that allow professional users to interact with, say, the GPT-4 API in a coding-free and straightforward manner. We expect the availability of such toolsets to evolve rapidly. This will make the production of situation-specific documentation simpler and more straightforward, but it will NOT remove the need for human planning and decision-making.
Challenge: Stakeholder Involvement
Another challenge is determining how to optimize the balancing of human and automated involvement. Ideally, the new tools will simplify meaningful human stakeholder involvement. Project planning documentation should make explicit from both management and implementor perspectives what is expected of a project, regardless of whether, say, agile or traditional waterfall methods are being employed.
One assumption we make is that, even when management agrees to use AI based tools to generate documentation, management must feel confident that they are “in control” by making sure that appropriate approval points are inserted throughout the process.
To illustrate a possible process for maintaining such control, see Figure 1 below, “Using AI-Based Text Synthesis to Accelerate Creation of Project Documentation.”
Roles
Figure 1 splits roles between “stakeholders” and “machine.”
Human stakeholders initiate and manage the process with different projects potentially requiring different mixes of stakeholders, e.g., business sponsor, technical sponsor, project manager, user representative, PMO manager, etc. One goal here is to maintain human accountability and responsibility via formal signoffs throughout the process.
The machine role is to compose drafts of documents of varying kinds starting with a draft of the project’s Business Case based on human-supplied input data. As shown in Figure 1 stakeholders are always designated as responsible for “review and edit” before any documents are completed with a final machine edit for grammar and clarity.
Figure 1. Using AI-Based Text Synthesis to Accelerate Creation of Project Documentation
Copyright (c) 2023 by Dennis McDonald & Michael Kaplan
Note that stakeholder roles in Figure 1 for document composition (as opposed to review and edit) are tagged “As necessary.” Our experiments using AI systems to generate a variety of project documents are encouraging to the point where we believe that substantial time and resources can be saved be engaging AI systems to generate the first and final drafts of a wide range of project planning and management documents.
Document Class
1.Business Case
The starting point is the business case. The business case form, format and data input fields may remain the same for all project types. The data in the business case will change for every project. This typically includes descriptions of the following:
Project Identification (e.g., budget type, business case ID, business case type, impacted areas, impacted portfolio name, project name, regulatory compliance requirements, short project description)
Project Details (e.g., alternatives considered, estimated completion date, estimated in-service dates, hardware purchase requirements, infrastructure change requirements, preferred start date, prerequisite interdependencies, product name, project background, project deliverables, service purchase requirements, software purchase requirements)
Project Scope (e.g., what is in, what is out)
Project Sponsors (e.g., business sponsor, sponsor organization, technical sponsor)
Objectives & Benefits (e.g., business benefits, business objectives, cost avoidance, success criteria)
Resource Requirements (e.g., total cost estimate, change management labor, contingency cost, external service cost, hardware, software, internal labor, training labor, external labor.
Delivery Organization (e.g., delivery method, delivery organization, project manager, technical lead)
Risk Mitigation (e.g., risk assessment, risk mitigation, cybersecurity implications, privacy implications)
Organizations will differ in the level of business case detail already in organized or structured form that can be input to the AI support system. Some organizations as a matter of course may include 50 or more variables in their standard business case templates. Also, the above list’s examples can be further detailed as necessary. Others may want to employ AI to help identify and structure the data needed for a business case.
Some original data collection may be required (A in Figure 1). It is important to remember that the more accurate and detailed the input data for the business case, the better the AI system will be in drafting and formally documenting it (B in Figure 1).
We have found in experimenting with either ChatGPT or GPT-4 that asking increasingly specific questions generates increasingly detailed responses. As an experiment, ask GPT-4 a question like “What are the main categories of data needed to construct a formal Business Case for a corporate project?” and then ask the question several different ways. Each time the response will vary somewhat with different detail being added when the project type is varied.
2.Project Charter
Before generating a Project Charter, management and stakeholders must review and edit the draft business case (C in Figure 1) with AI being used to do a final edit and formatting to sponsor specifications of the business case (D in Figure 1).
The Project Charter, generated from the Business Case (E in Figure 1) is a short document that management agrees expresses the purpose and objectives of the project along with summaries of how it will be implemented. It summarizes the “who, what, where, why, and how” aspects of the project. Essential Project Charter components may include:
Project objectives, scope, and constraints. This should include a clear and concise description of why the business needs the project.
Identification of stakeholders defines roles and responsibilities of sponsor, project manager, PMO, team member, end-users, testers, etc.
Milestones and timelines, i.e., what needs to be accomplished and when.
Budget and resource requirements identify estimated direct and indirect costs for people, systems, equipment, training, and other resources.
A description of Risks and assumptions will help in proactively mitigating issues that could impact the success of the project.
With both the Business Case and the Project Charter the intent is to provide a generic framework for project-specific data that describe the unique nature or context in which the project will be implemented.
Depending on the level of detail provided in the Business Case, based on our experiments we believe it is possible to use tools such as ChatGPT and GPT-4 to generate the first draft of the Project Charter (E in Figure 1). This is reviewed and edited by stakeholders (F in Figure 1) before the final automated completion of the Project Charter (G in Figure 1).
3. Project Definition
At this point we have documented the Business Case and prepared a shareable Project Charter, each representing the view of the business. Now from them we develop the Project Definition based on these two documents (H in Figure 1). In summary, the purpose of the Project Definition document is to:
Formalize understanding of the Project Charter by the organization tasked with its delivery. In other words, input from the Business Case and Project Charter are interpreted from the vantage point of the delivery organization, i.e., the organization tasked with responsibility for carrying out the work.
Generate detailed planning elements to control the details of project definition.
Provide an initial description of the project “shape” (schedule, task dependencies, scope, resource availability) that can then be used as a framework for planning.
Drive the drafting of Baseline Project Documents including the detailed Project Management Plan.
Document the impact on the organization of NOT doing the project.
This Project Definition document is the “meat and potatoes” document we will use as input to drafting detailed Baseline Project Documents (see below) which will include the Project Management Plan and other documents. These documents are prepared from the viewpoint of the organization tasked with actually delivering or managing the project to success.
A Project Definition document, stated from the perspective of the delivery organization, can include the following:
Project objectives.
Project background (e.g., business needs and environment, project general scope and plan constraints, solution background, key requirements)
Target solution and overall approach (e.g., general description of the target solution, major project components, high-level work breakdown structure, levels of uncertainty, technical environment, key options, overall approach)
Project scope e.g., position with respect to related projects, scope, key points that must be addressed, major deliverables, acceptance criteria, customer satisfaction)
Planning framework (e.g., high level project schedule, brief description of each work unit or sub-project, key assumptions, dates, and dependencies, overall risks and risk response)
Organization (e.g., sponsor organization, delivery organization, administrative oversight (e.g., PMO), contractor, performing organization)
Financial Estimates by category. (e.g., human resources, hardware, software, training, contractors)
We have found in our experiments that systems such as ChatGPT and GPT-4 can produce documents describing such high level Project Definition categories for a variety of project types. Examples we have tested include “Project Office 365 Implementation” and “Corporate Zero Trust Strategy Implementation.” We have been impressed with the level of detail provided by the systems which draw not only on general web-accessible knowledge but also in response to (a) the manner in which questions are submitted including how programmed prompts are engineered and (b) the degree to which the system “learns” from previously generated and finalized documents.
4.Baseline Project Documents
Once the Project Definition is finished and accepted it is then used as input to the AI-based drafting of the Baseline Project Documents (K in Figure 1). These planning documents are then reviewed by management (L in Figure 1) before being finalized as the basis for project management and reporting (M in Figure 1) by the delivery organization. Depending on the size and complexity of the project being planned, we expect that these processes may be iterative as project stakeholders and those responsible for project delivery review and fine tune the various documents:
The Project Management Plan describes in detail how the project will be managed, executed, and controlled. In the case of government contract proposals, for example, it is not unusual for the contracting agency to require that a draft of the Project Management Plan be submitted as part of the proposal. Traditionally, the ability to construct project management plans for such proposals has been heavily dependent on one’s “inside knowledge” of the contracting agency’s current state and conditions. Given various language models’ handling of contextual information about the customer (e.g., about the government agency issuing the contract) it might be possible to improve the specificity of project management plans included in proposals.
Requirements Documentation describes what the project’s deliverables are supposed to do including both operational characteristics (how the deliverables work) along with a description of impacted business processes.
Work Breakdown Structure and Schedule. These operate together to show the relationships among tasks, subtasks, and—importantly—responsibilities and schedules.
Resource Management. This plan identifies human and system resources required to perform the project and provides the foundation for management reporting. (We have been impressed with some of the resource estimates generated by the tools in our experiments but anticipate that further refinement will be needed).
Policy and procedure. Access to contextual information during the document drafting process will be critical so that the baseline documents reflect the language, resources, and policies of the sponsoring organization. If standard policies and procedures already exist these can be considered as inputs to the drafting carried out by the tools, for example:
Risk and Quality Management Plans describe how risks and quality will be managed and measured. In the case of risks, processes for monitoring and mitigating risk are described and how these efforts relate to other elements in the project management plan.
Communication Management describes how management, team members, and stakeholders will communicate, collaborate, and share information.
Change Management describes how necessary changes will be identified, prioritized, approved, and implemented.
Risks & Lessons Learned
General templates for a variety of project types abound on the Internet. Such templates can serve as the basis for manual production of documents such as tose described above above. What we have found in our experiments with ChatGPT and GPT-4 is that documents such as the above can, given appropriate business information input, be produced much more quickly than manual processes in a form (a) customized to the specific project situation and (b) at a lower cost than is possible with traditional manual template-based document creation approaches.
The approach outlined above does involve potential risks that will require additional experimentation and testing, including the following:
Accuracy and Quality: While ChatGPT and GPT-4 can generate text, their accuracy and quality may vary depending on the training data, context, and specific language used in the project documentation. They may also struggle with technical jargon and industry-specific terminology, leading to errors or inaccuracies in the documentation. Mitigation: Train core systems with domain-specific language and up-to-date contextual information (e.g., provide as input to the document drafting process a baseline schema describing current system architectures and infrastructure details).
Bias and Ethics: Machine learning models can also be influenced by biases in the training data and as a result may generate biased or discriminatory language. It's important to carefully review and edit any documentation generated by this process to ensure it aligns with ethical and inclusive practices. Mitigation: This is one of the reasons why we specify Review and Edit steps in Figure 1 since it is still possible for AI language model based engines to generate inaccurate information (Sometimes called “hallucination”).
Lack of Contextual Understanding: Tools like ChatGPT and GPT-4 may not always understand the full context of a project, or the specific needs of the stakeholders involved. This may be true no matter how detailed and concrete the Business Case is. Mitigation: This is another argument for human involvement throughout the documentation process -- and holding stakeholders responsible for the quality and accuracy of output.
Security and Confidentiality: Depending on the nature of the project, using an external AI model to generate documentation about a project dealing with sensitive information could pose risks to security and confidentiality. An example of this might be how tools such as ChatGPT and GPT-4 would be used in planning implementation of the US Defense Department’s Zero Trust Strategy framework. Mitigation: It's important to ensure that any sensitive information or data are not shared or accessed by third-party services. If security is required it may be necessary to host the AI language model on a secure network especially if it is to be trained with details of infrastructure and system architectures, or data about individual users (as might be the case with medical AI applications).
Questions and Future Research
Language model-based tools are evolving rapidly as are methods and software that can be combined with them. Examples of possible future research include:
Document type. Our focus here has been on project management related documentation. The examples of document types provided above are well understood in the environment of project management. What other classes or types of business or organizational documents could also be the subject of structured experimentation? For example, how might such tools assist in defining cybersecurity assessment maturity levels such as those incorporated in CMMC Certification?
Formality and certification. While the document types described above are standard as are the management processes surrounding them, we have not yet addressed strict formalisms such as adherence to a standard methodology like PMI’s PMBOK.
How much context? Adding contextual detail as input to the document creation process does increase relevance and specificity of the output. But how much detail is enough and what is the most appropriate way to provide that detail? For example, if the documentation is being developed to guide IT related projects, what is the best and most efficient way to provide a baseline of corporate information about existing resources, system architectures, and infrastructure? Also, it may be useful to incorporate corporate policies and baseline requirements as something for the AI software to consider.
At what cost? We are encouraged that the development of documentation can be accelerated and the cost of its development reduced using these tools, but we have not yet carried out any quantitative assessments. Also, the costs of using the base language model tools and any additional integrated or linked applications (e.g., to augment any necessary mathematical calculation requirements) may need to be factored into consideration.
Error checking. To what extent can the tools themselves perform error checking, and how is this related to human assessments at various checkpoints?
Newer tools and language models. Not discussed above is that, when using the tools in these experiments, it was possible to alternate language models which can generate different results. For example, OpenAI provides multiple Large Language Models (LLM) with newer products like GPT-4 being more feature rich and more accurate while providing API features to improve prompting, memory, tokenization, context, and cost. The newer models should, we believe, address some of the issues identified above.
Workflow Management. The organization of human and machine processes suggested in Figure 1 is a simple application of basic workflow management concepts. This raises the question of how best to manage the overall process and at what points to inject automated workflow management tools.
***
Copyright (c) 2023 by Dennis D. McDonald and Michael Kaplan
About the Authors
*Dennis D. McDonald, Ph.D.
Dennis’ writing, editing, and research work combines analytical and communication skills with extensive project management experience. His industry work includes support for civilian and military contractors, higher ed data governance strategy, corporate database and software consolidation and retirement, corporate adoption of collaborative technologies for knowledge sharing, and open data system implementation. His writing and content development work includes proposals (business and technical), project planning, research reports and white papers, and market research. He is a member of Alexandria Virginia’s Public Records Advisory Commission and a volunteer for the Alexandria Film Festival. He has served as a private school Board of Trustees member and has been a part owner of an IT-focused consulting company. He is a AAAS member and past CMMC Registered Practitioner (RP). His web site is located at www.ddmcd.com and his email address is ddmcd@ddmcd.com.
**Michael Kaplan, PMP
Michael Kaplan is a highly successful Senior Program Manager with over two decades of experience delivering impressive results for enterprises. His expertise in program management, project management, and digital transformation has enabled him to protect the national power grid and secure the critical systems of six global banks against cyberattacks. He has also played a pivotal role in enabling 30 organizations to transition into digital enterprises, implementing VMware software and leveraging Agile, Waterfall, and hybrid methodologies. Michael has excelled in team management, leadership, communication, and collaboration. He is certified as a Project Management Professional, Certified Scrum Master, and ITIL, and proficient in the use of various tools. Michael has worked with a broad range of clients in the financial and insurance sectors, healthcare, educational institutions, and government agencies. His natural talent for connecting the dots across people, process, and technology domains, his commitment to delivering excellence, and his professionalism make him a highly respected Program Project Management Consultant. He can be reached via email at kaplan.usa@gmail.com.
