An article in the Guardian describes how Co-op Legal Services are using a system produced by OpenAI to implement a quantified panopticon for their staff.

In a system described as “oppressive and dystopian” by a whistleblower, Co-op Legal Services is using AI models to record, analyse and award a percentage score for interactions between agents and customers seeking advice about probate, wills and estates.

A model from OpenAI has been trained to assess more than 50 discrete aspects of each phone call and provide pass and fail scores to managers who use the data to analyse employee performance. One worker said the system can mean their work is monitored by AI for several hours a day as they handle inquiries often from recently bereaved customers.

I am a member of the Co-op and I don’t like the sound of this at all, so I wrote to them at their complaints email address. Here’s my email.

I am a Co-op member (number [REDACTED]) but after having read about the AI panopticon that you are placing your staff under (“‘Dystopian’: Co-op becomes latest firm to put staff under AI surveillance” in The Guardian), I won’t be choosing Co-op Legal Services for any legal services I require in the future.

Human beings are not simply fungible units of work resource, and placing them under constant automated monitoring and rating is degrading, dehumanising, and stress-inducing. Large language models (LLMs) are not objective; on the contrary, they encode the underlying biases of their training material – material that includes as much internet content as the companies making them can scrape, including content that is sexist, racist, or otherwise bigoted – but in a way that makes it impossible to evaluate why they might have produced their conclusions.

Using LLMs to perform this monitoring adds significant environmental cost, and helps to further entrench the power of American megacorporations – OpenAI in this case – who do not have the best interests of members or even of humanity at heart. Sam Altman, chief executive of OpenAI, might believe that we must “[accept] the fact that some bad things are going to happen”, but we don’t have to actively contribute to them.

I believe that this use of AI surveillance is contrary to several of the Co-operative principles, specifically:

  • Self responsibility: because staff who are constantly observed are not able to enjoy their own responsibility.
  • Solidarity: with workers, with those whose artistic work has been ingested and plagiarised by LLMs, with those whose homes are blighted by hyperscale data centres, and with everyone else whom the AI megacorporations consider to be acceptable collateral damage.
  • Social responsibility: because the environmental cost of AI data centres is destroying habitats and increasing power consumption at a point when our planet can least afford it.

Indeed, it is with the principle of solidarity for employees of Co-op Legal Services that I write to you today. I urge you to take a step back, instead of rushing towards a future that will be bad for everyone, but especially bad for the employees who find themselves at the sharp end of this technology.

In retrospect, I missed the obvious point that as a customer I wouldn’t want the contents of my conversations being fed into Altman and Brockman’s plagiarism machine, either, but perhaps you’d like to add that to your own email to them.