« Live on WDEV - Remembering Tim Hayward | Main | Our Daughter and Family Doing What's Right »

April 11, 2024

Human-in-the-Loop Artificial Intelligence

Teams of humans and AI agents will be the model for the organization of the future.

There are many tasks which Artificial Intelligence (AI) can’t perform well without help and supervision; it has an unfortunate tendency to hallucinate (make up an answer). For that matter, there are many tasks humans can’t perform well without help or supervision and humans are expensive and in short supply, at least in the work force. Two techniques have been developed in the last year of experimentation with AI and Large Language Models (LLMs) like ChatGPT: multi-agent collaboration and human-in-the-loop processes. Both help mitigate hallucination and produce content faster and more cheaply than humans can alone.

Multi-Agent Collaboration

An agent is nothing more than a packaged use of an LLM. The simplest agents accept a human prompt, ask AI to respond, and return the result. More complex agents retrieve information the LLM has not been trained on from a database or from a web search and send that information along with the request to the LLM. Agents can ask the LLM to write code in order to generate a response to the human prompt. For example, if the human wants a graph of the Dow Jones Average (DJA) for the next month compared to the S&P, several agents might be involved. The LLM will not have been trained on data as recent as last month, and so one agent retrieves the daily closing prices of both the DJA and the S&P from the web using the general knowledge of the LLM to formulate the web requests. A second agent writes computer code necessary to produce a graph of the two indices on the same set of axes. A third agent executes the code and the resulting graph is displayed to the user.

Using multiple agents is also a good way to reduce hallucination. Agent One (the writer) generates a draft answer to a question; Agent Two (the fact-checker) does web searches to validate assertions in the draft; Agent Three is a reflection agent which critiques the draft both using the fact-checking results and looking for bad style or bad usage. The answer then goes back to Agent One for revision and back though the gauntlet of agents Two and Three until all are satisfied. The final answer is sent to the human who asked the question. Interestingly (for technical reasons I won’t explain here) even if all agents use the same LLM, they can disagree constructively. However, using different LLMs for the different agents makes the whole process even more robust and reliable.

Human-in-the-Loop

Suppose the writer, fact-checker, and reflection agents above never reach agreement. Drafts could go round and round forever chewing up expensive LLM cycles. One solution is an arbitrator agent. Another solution is to add a human into the loop. For example, after the first draft has been written, fact-checked, and critiqued, a human is given an opportunity to redraft herself, to modify or accept the critique, and to arbitrate the fact-checking. Depending on what the human decides, the article could go another round and then come back to the human again. The process ends when the human accepts a draft, perhaps after making some more modifications. In more complex processes, humans can be involved in any step the application designer feels is helpful.

A Demonstration

I programmed a demonstration application to write a news story using either the minutes or a transcript of a meeting as a source. With the shrinkage of local news outlets, there are never enough reporters to attend all the public meetings which affect our lives and should influence our votes. AI and LLMs alone can’t create a well-written news story accurately enough. But human editors working with AI can turn out stories which are well-written and accurate… and cheap enough for a local newsroom to afford.

First the human editor specifies how many words the article should be and whether the source is on the internet or the editor’s computer using the screen below.

HIL1

In the next screen (not shown) the editor gives a URL or selects a local file as the source. An input agent obtains the source document and extracts text from it. The writer agent uses the transcript or minutes to draft an article. The reflection agent does double duty: fact-checking against the source document and checking style. The human editor can then edit either the article or the critique or accept the article as written using the dialog pictured below. If there is a human or agent critique, the cycle repeats with a rewrite.

HIL2

The source document I used for this article was the draft minutes of a Stowe Select Board meeting. The reflection agent criticized the first draft for not starting out with the most important facts (“burying the lede” in news jargon), for not clearly explaining the disagreement among the Board Members, and from being sloppy with some facts. The second draft was better in these respects and was accepted by the reflection agent. However, I (playing editor) felt it was still not clear in one respect and wrote my own critique. The third draft passed muster. The whole dialog is down below but it’s long.

You can play with this demo app at https://meeting-reporter.streamlit.app/. I don’t charge anything for using it, but you do need to provide your own OpenAI API key to get past the first screen because the LLM being used is ChatGPT4. OpenAI will only charge you a few cents per use, but you do need an account and a paid API Key. If you don’t know what an API key is, you are like most of the world and you don’t want or need one, at least not yet.

Note to nerds: if you want to see or play with the source code, it is open source and available at https://github.com/tevslin/meeting-reporter. It uses Langgraph and Streamlit libraries with a tkinter version available and is hosted in the Streamlit cloud. The LLM is ChatGPT4-Turbo.

See also:

An AI Debate for another example of multi-agent collaboration

https://blog.tomevslin.com/ai/ for more on AI in general

The Transcript of the Session (tl;dr)

The original draft

 In a special meeting held on Wednesday, March 29, 2024, the Stowe Selectboard, led by Chair Billy Adams, convened to address the urgent matter of the Stowe Short-Term Rental Registry Ordinance. The meeting saw attendance from both Selectboard members and a significant number of the public, with 26 individuals participating via Zoom. The session was marked by discussions on the appropriateness of the meeting's timing and the methods for deciding on the rescindment of the recently adopted ordinance.

The Selectboard approved the agenda unanimously after a brief discussion on whether the meeting was necessary ahead of the regular April 10 meeting. The urgency was attributed to a petition received on March 28, necessitating a special meeting to adhere to statutory requirements for public notice. Despite some members expressing concerns over the rushed nature of the decision-making process, the board agreed to proceed with planning for a special meeting of the voters.

Set for Wednesday, May 1, 2024, at the Stowe High School Auditorium, the special meeting aims to vote on whether to rescind the town’s Short-Term Rental Registry Ordinance, adopted on February 29, 2024. The decision came after confirming the venue’s availability and the necessity of holding the meeting within a 60-day timeframe as required by law. Concerns were raised about ensuring adequate participation, with suggestions for a later start time to accommodate working residents and considerations for childcare to aid attending parents.

Amidst the technical discussions, a deeper undercurrent of division within the Selectboard was evident, with members voicing the need for a more collaborative approach moving forward. The meeting concluded with a call for unity and a more deliberate approach to decision-making, highlighting the tensions surrounding the short-term rental ordinance debate.

The Selectboard ultimately decided against holding an informational meeting but agreed on the creation of a fact sheet to be made available online and in the newspaper, aimed at clarifying the ordinance details and the upcoming special meeting process.

Critique from the reflection agent

The article is well-written and adheres to the facts of the source material. However, there are a few areas where clarification would improve comprehension. Firstly, the ordinance the Selectboard is considering rescinding is introduced too late in the article. Mention it earlier to provide necessary context. Secondly, the "deeper undercurrent of division" within the Selectboard needs more explanation. What are the differing views? Lastly, the final paragraph states the Selectboard decided against an informational meeting. This seems contradictory to the fact sheet creation. Clarify this decision-making process.

Second draft

In a special meeting convened on Wednesday, March 29, 2024, the Stowe Selectboard, under the leadership of Chair Billy Adams, gathered to deliberate on the pressing issue of possibly rescinding the Stowe Short-Term Rental Registry Ordinance, adopted on February 29, 2024. The meeting, attended by Selectboard members and the public, with 26 individuals joining via Zoom, was characterized by significant discussions focusing on the timing of the meeting and the procedural aspects of revoking the recently instituted ordinance.

From the outset, the urgency of the meeting was made evident, with the Selectboard unanimously approving the agenda. This urgency was driven by a petition received on March 28, compelling the need for a special meeting to comply with the legal requirements for public notification. Despite reservations voiced by some members regarding the expedited nature of this process, there was consensus on moving forward with the planning of a special voter meeting.

Scheduled for Wednesday, May 1, 2024, at the Stowe High School Auditorium, this special meeting is aimed at deciding whether to overturn the town's Short-Term Rental Registry Ordinance. This decision was reached after affirming the venue's availability and recognizing the legal obligation to conduct the meeting within a specific 60-day window. The discussion also touched upon ensuring broad participation, with proposals for a later starting time to accommodate residents working out of town and the provision of childcare services to support attending parents.

Amid these logistical and procedural deliberations, a palpable division within the Selectboard came to the fore, underscoring differing perspectives on the approach to the short-term rental ordinance debate. This internal discord prompted calls from members for a more unified and thoughtful decision-making process going forward.

Concluding the meeting, the Selectboard opted against hosting an informational meeting, instead agreeing on disseminating a fact sheet through online platforms and local newspapers. This approach aims to demystify the specifics of the ordinance and the procedural details of the forthcoming special meeting, addressing the public's need for clear and accessible information.

My Critique

The minutes do not say that they "agreed on disseminating a fact sheet through online platforms and local newspapers. They simply say that Mr. Adams asked Mr. Safford to have the staff prepare a fact sheet. Please be precise.

Final Article

Stowe Selectboard Schedules Special Meeting Amidst Division

11/04/2024

In a special meeting convened on Wednesday, March 29, 2024, the Stowe Selectboard, under the leadership of Chair Billy Adams, gathered to deliberate on the pressing issue of possibly rescinding the Stowe Short-Term Rental Registry Ordinance, adopted on February 29, 2024. The meeting, attended by Selectboard members and the public, with 26 individuals joining via Zoom, was characterized by significant discussions focusing on the timing of the meeting and the procedural aspects of revoking the recently instituted ordinance.

From the outset, the urgency of the meeting was made evident, with the Selectboard unanimously approving the agenda. This urgency was driven by a petition received on March 28, compelling the need for a special meeting to comply with the legal requirements for public notification. Despite reservations voiced by some members regarding the expedited nature of this process, there was consensus on moving forward with the planning of a special voter meeting.

Scheduled for Wednesday, May 1, 2024, at the Stowe High School Auditorium, this special meeting is aimed at deciding whether to overturn the town's Short-Term Rental Registry Ordinance. This decision was reached after affirming the venue's availability and recognizing the legal obligation to conduct the meeting within a specific 60-day window. The discussion also touched upon ensuring broad participation, with proposals for a later starting time to accommodate residents working out of town and the provision of childcare services to support attending parents.

Amid these logistical and procedural deliberations, a palpable division within the Selectboard came to the fore, underscoring differing perspectives on the approach to the short-term rental ordinance debate. This internal discord prompted calls from members for a more unified and thoughtful decision-making process going forward.

Concluding the meeting, Chair Adams specifically requested Mr. Safford to have the staff prepare a fact sheet aimed at clarifying the specifics of the ordinance and the procedural details of the forthcoming special meeting. This focused effort reflects the Selectboard's intention to address the public's need for clear and accessible information without definitively concluding on the method of dissemination.

| Comments (View)

Recent Posts

The Magical Mythical Equalized Pupil

Our Daughter and Family Doing What's Right

Human-in-the-Loop Artificial Intelligence

Live on WDEV - Remembering Tim Hayward

In Memoriam: Timothy Y. Hayward

Comments

blog comments powered by Disqus
Blog powered by TypePad
Member since 01/2005