(Image: Gorkie/Private Media)

South Australia Health aged care staff were so overwhelmed by thousands of false alerts by an artificial intelligence CCTV trial that they didn’t respond when technology correctly identified a resident’s fall. 

South Australian Health Minister Chris Picton released a report from PwC Australia on a 12-month trial that used artificial intelligence and CCTV in the common areas and rooms of two facilities, Mount Pleasant Aged Care and Northgate House, to detect falls, calls for help, assistance needs and screams. An independent, off-site monitoring centre would contact staff at the site by telephone to tell them of the detection.

The technology, provided by an unnamed South Australian-based company, was supposed to assist staff by alerting them to incidents to improve residents’ health and wellbeing and stop abuse, harm and neglect.

But that’s not what happened. The report found “there was no evidence that the AI-based CCTV surveillance used in the pilot had influenced either positively or negatively the quality and safety of the care provided at the sites”.

Presenting the report, Picton slammed the trial at a press conference: “The report shows a completely unacceptable level of 12,000 false report alerts.” 

Throughout the trial, the artificial intelligence technology repeatedly raised the alarm for events that a high percentage of “were sit-fall events which involved staff performing a bend to knee (crouching) motion which resembled a program movement and caused an alert”, the report said.

The technology also was unable to distinguish between “programmed events and similar movements or sounds that are reasonably expected in residential care”. Real falls or calls for help were accurately detected, however.

The trial operators had expected up to 10 false alerts a day but ended up sending so many that staff reported alert fatigue and were overwhelmed by the work it took to respond to false alarms.

“In these final months of the pilot, staff were no longer able to respond to every alert. There was at least one instance where staff did not respond to an alert that turned out to be a ‘true’ resident fall event.” 

Staff “paused” the alerts 18 times in response to excessive alerts. Seventy per cent of staff said that the trial did not improve their ability to respond to safety events, but the report noted that the artificial intelligence’s false alarms did improve towards the end of the trial.

Surveys with staff and family members of residents found that there was not a significant concern about privacy or security of the footage. However, only a third of staff said they felt a bit or very ready for the start of the pilot.

With more companies selling artificial intelligence “solutions” to industries for tasks as crucial and niche as recognising suicide attempts in prisons, there are questions about the proven reliability and accuracy of these products. 

Businesses may take artificial intelligence promoters at their word about the product’s effectiveness. There’s no industry standard or regulation for the technology. But the individual environment where an artificial intelligence technology — such as the CCTV trial — is implemented will affect how well it works.

“We can expect to see the increased digitisation of aged care and it is important to learn from trials like the pilot to determine what role technology has to play and how and when it will be appropriate to support quality and safety in aged care,” the report concludes. 

Does artificial intelligence have a place in workplaces like aged care? Let us know by writing to letters@crikey.com.au. Please include your full name to be considered for publication. We reserve the right to edit for length and clarity.