Big data operational and management requirements

November 2019 Editor's Choice

Big data offers huge advantages in enhancing or ‘augmenting’ data, giving it added meaning and context, creating a broader perspective in which to view things, and adding to the value of data and enhancing the potential depth of evaluation. We are increasingly seeing the implementation of big data integration and synthesis to assist in decision making in a range of control room environments, and this is just the start of a process that is likely to be much more sophisticated in the future. Processes such as face recognition, number plate recognition, access and personal information, movement tracking, and various dimensions of video processing can all be combined to assist us to make decisions in the security area.

The foundation of big data is, by nature, shared and linked data. In many cases this may be live data from sources such as weather, traffic, number plate recognition, positioning or status from electronic tags, etc. In others, you may access short- to long-term stored data where delays of data transfer and processing can lead to a few seconds of latency in the process of the transmission and integration.

For example, scams using delays measured in seconds in live feeds for betting abuse have been found in a number of crime syndicate related activities. However, users are also likely to access systems that contain days, weeks or even years of recorded data. In all of these, there is a quality factor that should be evaluated very carefully. Risk implications include having inaccurate information, out of date information, faked information, or delayed updates that can have a major impact on the integrity of judgement or conclusions that arise from the data.

Data quality

Data is only as good as the capture process and ongoing management of the information in the database. One would consider the data inflow process, how consistently is data treated across the capture process, and the considerations regarding accuracy and possible deviations or inconsistencies. In addition, how timeous is the data or does it reflect conditions that may have already changed or become obsolete, how regularly is it updated and is there a way to establish the most recent update, and what happens with redundant data – is it cleaned, does it clog up the systems, when does it become redundant, and who has responsibility for this?

Obviously the design of the big data system interaction and integration would consider some of these factors, but the danger is in taking these kinds of conditions for granted and ending up with a domino effect where small errors can lead to major implications for valid decision making. Where parties responsible for data management of some of the systems are not efficient in their capture, management and maintenance it has implications for the validity of the data.

Before you start using big data sources, you need to have your own evaluations of the risk implications of using the various sources. This should include trustworthiness, confidence levels, and measures that are built in for verification. Further, it is not just the data itself. There may be various algorithms that have been set up to process the data which may have their own implications. The issue of algorithm bias and discrimination that may result from perspectives of those setting up the system came up as a prominent issue in a behaviour analysis conference in Minneapolis recently, with comments from an ACLU representative commenting about the potential of this.

Protecting yourself

Ultimately the use of big data is a social process and not just an IT one. Where data can get captured, criminals may be working actively to combat identification or labelling by systems such as number plate recognition, or face recognition. This may go to active strategies such as cloning number plates and even vehicles, not just hiding identifiers. Hacking and manipulation of such systems, or paying to have data compromised or changed need to be taking into consideration.

Users should conduct a thorough risk analysis of the systems they may be drawing from, and have procedures in place to ensure verification, legal accountability, and be in a position where they can evaluate the cost of working decisions or outcomes on the business or organisation. Use of powerful tools needs to be balanced with appropriate caution.

Craig Donald.

Relying on AI for decision making should also be part of any risk evaluation. Thinking and verifying before one acts on data should ultimately be a human decision which incorporates verification and situational awareness. I continually am surprised by the amount of social and psychological consideration that needs to go into security systems that are supposedly ‘intelligent’ and can think and act on their own. How much do you trust the people behind the data?

Craig Donald

Dr Craig Donald is a human factors specialist in security and CCTV. He is a director of Leaderware which provides instruments for the selection of CCTV operators, X-ray screeners and other security personnel in major operations around the world. He also runs CCTV Surveillance Skills and Body Language, and Advanced Surveillance Body Language courses for CCTV operators, supervisors and managers internationally, and consults on CCTV management. He can be contacted on +27 11 787 7811 or


Share this article:
Share via emailShare via LinkedInPrint this page

Further reading:

2020 Residential Estate Security Conference in KZN
November 2019, Technews Publishing , Editor's Choice
Meeting the residential security challenges in 2020 and beyond: Hi-Tech Security Solutions will host the Residential Estate Security Conference 2020 in Durban on 12 March 2020.

Sentian secures funding from Imfezeko
November 2019, Sentian , Editor's Choice
Cape Town based smart security technology developer Sentian has recently secured a round of funding from Imfezeko Investment Holdings.

A brighter future
November 2019, NEC XON, Technews Publishing , Editor's Choice
NEC XON hosted its annual summit in October 2019 at Sun City under the banner of ‘Orchestrating a Brighter World’.

What are the cybersecurity issues in video surveillance?
November 2019, Axis Communications SA , Editor's Choice, CCTV, Surveillance & Remote Monitoring, Cyber Security
he importance of the data captured by surveillance cameras – and what can be done with it – has led to a new breed of cybercriminals, looking for insights to steal and sell.

GUI for fire detection systems
November 2019, FDIA (Fire Detection Installers Association) , Editor's Choice
A graphical user interface for fire detection is a graphics program that makes it easier for the operator to access all the functionalities of the fire detection system.

Determining and mitigating risk: Where to begin?
November 2019 , Editor's Choice
Managing risk means proactively reducing the severity or seriousness of that which has not ‘yet’ occurred, says Lesley-Anne Kleyn.

A platform approach to innovation and value
CCTV Handbook 2019, Technews Publishing , Editor's Choice, CCTV, Surveillance & Remote Monitoring, Integrated Solutions, IT infrastructure
Moving to the platform model of doing business holds tremendous advantages for end users and smaller developers, but also for the whole technology supply chain.

Open does not always mean easy integration
CCTV Handbook 2019, Veracitech, Technews Publishing , Editor's Choice, CCTV, Surveillance & Remote Monitoring, Integrated Solutions
Customers who opt for best-of-breed solutions will have to rely on their integrators to develop customised integrated solutions for them.

The impact of AI on the surveillance industry
CCTV Handbook 2019, G4S South Africa, Hikvision South Africa, Myertal Tactical Security, Technews Publishing , Editor's Choice, CCTV, Surveillance & Remote Monitoring
What the impact of AI will be on companies, the services and solutions they supply, as well as on the jobs people do.

Video analytics and AI
CCTV Handbook 2019, Axis Communications SA, Dallmeier Electronic Southern Africa , Hikvision South Africa, Technews Publishing, Dahua Technology South Africa , Editor's Choice, CCTV, Surveillance & Remote Monitoring, Integrated Solutions
Artificial intelligence has the potential to deliver real benefits in the world of video analytics and many companies are already delivering customer benefits.