LLM Tools: Force Multipliers and/or Sanity Checkers?
Shortly after Michael Kaplan and I completed Using ChatGPT to Accelerate Creation of Business Case and Project Definition Documents I saw the following article in the July 13, 2023 issue of Science from the AAAS: Experimental evidence on the productivity effects of generative artificial intelligence.
As a consultant I do a lot of writing for clients so I was intrigued by the article’s summary:
Automation has historically displaced human workers in factories (e.g., automotive manufacturing) or in performing routine computational tasks. Will generative artificial intelligence (AI) tools such as ChatGPT disrupt the labor market by making educated professionals obsolete, or will these tools complement their skills and enhance productivity? Noy and Zhang examined this issue in an experiment that recruited college-educated professionals to complete incentivized writing tasks. Participants assigned to use ChatGPT were more productive, efficient, and enjoyed the tasks more. Participants with weaker skills benefited the most from ChatGPT, which carries policy implications for efforts to reduce productivity inequality through AI. —EEU
In my own consulting (e.g., in helping clients develop project plans and technical proposals for government IT- and data-related contracts) I have used tools such as ChatGPT as research tools, for example, in creating outlines for sections of technical proposals. The actual writing and editing I have done myself, largely because (a) I have a lot of personal consulting and project management experience I can bring to bear on the writing, and (b) because the process of writing scripts to prompt tools such ChatGPT to output specifically constructed documents is still evolving, as Michael and I have discovered.
The findings of the article are telling and seem to suggest that less experienced writers will benefit more than experienced writers from using LLM tools. As Michael and I have discovered in our experiments, however, there’s more to editing a ChatGPT-generated document than just doing a well-informed sanity check given the amount of control one can exert over the way that ChatGPT processes and structures output. Such tools are more likely to be valuable “force multipliers” for knowledge workers but we still need more research into costs and benefits in order to understand how best to employ and govern LLM based writing tools.
Copyright (c) 2023 by Dennis D. McDonald
