How good is an artificial intelligence?

CCTV Handbook 2021 Surveillance

AI-based video analysis holds the promise of a technological quantum leap, with significant customer benefits. But only if the competent (i.e. informed) user can appraise the technology correctly. This article is intended to set forth a few basic principles which will enable users to correctly evaluate its functionality, usability and the advantages for their own specific application.

Systems have learning difficulties too

Routines based on artificial intelligence (AI) have been growing increasingly widespread in video security technology for a long time. A steadily growing number of new applications and products rely on the algorithms to offer new analyses or make existing analyses significantly more reliable. The objective is to provide appreciable added value for users and the results speak for themselves: Not too long ago, a great deal of work was required before classic image processing was able to recognise a tree moving in the wind as a false alarm, to give just one example. Today AI does that effortlessly.


Dr Maximilian Sand, team leader artificial intelligence, Dallmeier.

The essential point of distinction between image or video analyses with classic image processing and those which use AI is that algorithms are no longer ‘just’ programmed, they can also be ‘taught’ with the aid of large volumes of data. On the basis of this data, the system learns to detect patterns and accordingly recognise the difference between a tree and an intruder, for example. But the concept of machine learning also throws up new problems and challenges.

One well-known example of this is the difference in the quality of recognition of different ethnic groups, an issue which has even made headlines in the news. Yet the background is relatively simple: An AI system can only learn substantially if it is supplied with enough and sufficiently diverse and evenly distributed data.

The quality of the AI system

All of this leads to the question of the performance capability of a system that uses artificial intelligence. What metrics allow a comparison between two routines, two different systems or two manufacturers, for example? What does it mean when a product brochure promises, for example, ‘95% detection accuracy’ or ‘reliable recognition’? How good is accuracy of 95%? And what is ‘reliable recognition’?

First of all, it is most important to understand how AI routines can be evaluated. The first step is the application- and customer-specific definition of what ‘incorrect’ and ‘correct’ mean, especially in borderline cases: For example, in a system set up to recognise persons, is a detection to be defined as correct if the image or video does not even show a real person, but instead just an advertisement depicting a person?

This and other parameters must be defined. As soon as this definition has been established, a dataset is needed in which the results that are expected to be correct are known. This dataset will then be analysed with AI to deduce the percentages of correct and incorrect detections. In this process, mathematics provides the user with an exceptionally wide variety of metrics, such as sensitivity (percentage of expected detections which were actually detected) or hit accuracy (percentage of detections that are actually correct). Ultimately, therefore, the quality of AI is always a statistical statement about the evaluation dataset used.

Summer or winter?

How usable this statement really is for the user or potential purchaser of a system depends on the distribution of the dataset. Accordingly, an evaluation may attest to good recognition performance. But if the dataset was founded solely on image material from summer months, this evaluation has no validity regarding the quality of the AI in winter, since light and weather conditions may be very different.

In general, it follows that statements about the quality of an AI analysis – particularly those quoting specific figures, such as ‘99,9%’ – are to be treated with caution if not all parameters are known. If the dataset used, the metrics used and the other parameters are unknown, in fact it is no longer possible to make a definitive statement about how representative the result is.

Exact specifications do not exist

Every system has its limits and of course this is also true of AI systems. Therefore, knowing these limits is the fundamental prerequisite for making sound decisions. But here too, statistics and reality collide, as is illustrated in the following example: Logically, the smaller an object is in the image/video, the less well an AI system is able to recognise it. So, the first question the user asks himself before buying a system relates to the maximum distance at which objects can be detected, since this has implications for the number of cameras needed and thus also for the costs of the total system.

However, it is quite impossible to specify an exact distance. There is simply no value up to which the analysis delivers 100% correct results, nor another value above which recognition is entirely impossible. In this case, an evaluation is only able to return statistics. For example, detection accuracy as a function of object size.

Best compare directly

Regarding the system limits, it is conventional practice, for example, in product datasheets to describe the limits of the system using specific minimum and maximum values to the extent possible. These include the minimum distance or a minimum resolution. This is also expedient, because customers or installers need points of reference to enable them to rate the system.

Even so, there are still many unknowns, for example, whether the manufacturer was inclined conservatively or more optimistically when specifying these limit values. So, the user is well advised always to bear in mind that there are no well-defined, clear limits in video analysis.

For all systems, errors will inevitably occur even within certain parameters and at the same time useful results can be returned outside of the limits under favourable conditions.

If one wishes to find out the true quality of an AI-based analysis as a user, this is really only possible by carrying out a direct comparison – the figures and parameters quoted by the various manufacturers differ too widely. Furthermore, of course, the boundary conditions and the input must be identical for all systems.

The optimum option for this is a live test, with demo products, rented equipment or the like. Then, the performance capability of the system in the exact usage case required is also revealed. Incidentally, this also describes the yardstick used to evaluate the performance capabilities of AI systems generally: It all depends on the specific usage case. This should be specified as precisely as possible. Only then is it also possible to generate true added value for the customer based on the right solution.


Credit(s)




Share this article:
Share via emailShare via LinkedInPrint this page



Further reading:

Hikvision launches AcuSeek NVR
Surveillance Products & Solutions AI & Data Analytics
By integrating natural language interaction, Hikvision’s AcuSeek NVR enables precise video and image retrieval within seconds, marking a transformative milestone for the security industry's advance into intelligent and efficient applications.

Read more...
2025 video surveillance market set for improved fortunes
News & Events Surveillance
Novaira Insights has unveiled its latest report, World Market for Video Surveillance Hardware and Software – 2025 Edition, forecasting a healthy growth rate of 8,1% until 2029, excluding China.

Read more...
Dallmeier and Hexagon form a technology partnership
Surveillance
Dallmeier and Hexagon have agreed to form a technology partnership, which will see the two companies combine their expertise to develop integrated security solutions for various application areas, particularly critical infrastructures.

Read more...
Analyse, automate, and optimise logistics processes
neaMetrics Surveillance Transport (Industry) Products & Solutions Logistics (Industry)
In today’s rapidly evolving logistics sector, the pressure to improve process efficiency, optimise resource usage, and ensure seamless security is more intense than ever. Smart, proactive surveillance is no longer a luxury — it is a critical operational necessity.

Read more...
Safer spaces through smart surveillance
NEC XON Surveillance
Advances in facial recognition technology are transforming surveillance from a mere recording tool into an intelligent, integrated system that enhances real-time safety, moving beyond the traditional expansion of CCTV efforts.

Read more...
Next generation of AI-powered video telematics
IoT & Automation Surveillance Transport (Industry)
Webfleet, Bridgestone’s fleet management solution in South Africa, has launched Webfleet Video 2.0, an AI-powered solution designed to enhance fleet safety, security, compliance with local regulations and operational efficiency through real-time video insights.

Read more...
Key design considerations for a control room
Leaderware Editor's Choice Surveillance Training & Education
If you are designing or upgrading a control room, or even reviewing or auditing an existing control room, there are a number of design factors that one would need to consider.

Read more...
Smart cities and the role of video security
Surveillance Integrated Solutions
As cities around the world continue to embrace smart technology, including IoT that not only connects to people, but also the surrounding activity, the integration of advanced video security systems is crucial to ensure safety and efficiency in environments.

Read more...
How intrusion protection helps secure O&G operations
Surveillance Perimeter Security, Alarms & Intruder Detection Industrial (Industry)
For O&G operators in Africa, physical security remains one of the biggest considerations, particularly when it comes to perimeter protection and the ability to mitigate intruder-related incidents.

Read more...
Axis secures the Waterfront
Surveillance Entertainment and Hospitality (Industry) Retail (Industry)
Axis Communications shares insight into its longstanding partnership with the V&A Waterfront, one of Africa’s premier retail and mixed-use precincts, through its latest, updated customer success story.

Read more...










While every effort has been made to ensure the accuracy of the information contained herein, the publisher and its agents cannot be held responsible for any errors contained, or any loss incurred as a result. Articles published do not necessarily reflect the views of the publishers. The editor reserves the right to alter or cut copy. Articles submitted are deemed to have been cleared for publication. Advertisements and company contact details are published as provided by the advertiser. Technews Publishing (Pty) Ltd cannot be held responsible for the accuracy or veracity of supplied material.




© Technews Publishing (Pty) Ltd. | All Rights Reserved.