Big data operational and management requirements

November 2019 Editor's Choice

Big data offers huge advantages in enhancing or ‘augmenting’ data, giving it added meaning and context, creating a broader perspective in which to view things, and adding to the value of data and enhancing the potential depth of evaluation. We are increasingly seeing the implementation of big data integration and synthesis to assist in decision making in a range of control room environments, and this is just the start of a process that is likely to be much more sophisticated in the future. Processes such as face recognition, number plate recognition, access and personal information, movement tracking, and various dimensions of video processing can all be combined to assist us to make decisions in the security area.

The foundation of big data is, by nature, shared and linked data. In many cases this may be live data from sources such as weather, traffic, number plate recognition, positioning or status from electronic tags, etc. In others, you may access short- to long-term stored data where delays of data transfer and processing can lead to a few seconds of latency in the process of the transmission and integration.

For example, scams using delays measured in seconds in live feeds for betting abuse have been found in a number of crime syndicate related activities. However, users are also likely to access systems that contain days, weeks or even years of recorded data. In all of these, there is a quality factor that should be evaluated very carefully. Risk implications include having inaccurate information, out of date information, faked information, or delayed updates that can have a major impact on the integrity of judgement or conclusions that arise from the data.

Data quality

Data is only as good as the capture process and ongoing management of the information in the database. One would consider the data inflow process, how consistently is data treated across the capture process, and the considerations regarding accuracy and possible deviations or inconsistencies. In addition, how timeous is the data or does it reflect conditions that may have already changed or become obsolete, how regularly is it updated and is there a way to establish the most recent update, and what happens with redundant data – is it cleaned, does it clog up the systems, when does it become redundant, and who has responsibility for this?

Obviously the design of the big data system interaction and integration would consider some of these factors, but the danger is in taking these kinds of conditions for granted and ending up with a domino effect where small errors can lead to major implications for valid decision making. Where parties responsible for data management of some of the systems are not efficient in their capture, management and maintenance it has implications for the validity of the data.

Before you start using big data sources, you need to have your own evaluations of the risk implications of using the various sources. This should include trustworthiness, confidence levels, and measures that are built in for verification. Further, it is not just the data itself. There may be various algorithms that have been set up to process the data which may have their own implications. The issue of algorithm bias and discrimination that may result from perspectives of those setting up the system came up as a prominent issue in a behaviour analysis conference in Minneapolis recently, with comments from an ACLU representative commenting about the potential of this.

Protecting yourself

Ultimately the use of big data is a social process and not just an IT one. Where data can get captured, criminals may be working actively to combat identification or labelling by systems such as number plate recognition, or face recognition. This may go to active strategies such as cloning number plates and even vehicles, not just hiding identifiers. Hacking and manipulation of such systems, or paying to have data compromised or changed need to be taking into consideration.

Users should conduct a thorough risk analysis of the systems they may be drawing from, and have procedures in place to ensure verification, legal accountability, and be in a position where they can evaluate the cost of working decisions or outcomes on the business or organisation. Use of powerful tools needs to be balanced with appropriate caution.


Craig Donald.

Relying on AI for decision making should also be part of any risk evaluation. Thinking and verifying before one acts on data should ultimately be a human decision which incorporates verification and situational awareness. I continually am surprised by the amount of social and psychological consideration that needs to go into security systems that are supposedly ‘intelligent’ and can think and act on their own. How much do you trust the people behind the data?

Craig Donald

Dr Craig Donald is a human factors specialist in security and CCTV. He is a director of Leaderware which provides instruments for the selection of CCTV operators, X-ray screeners and other security personnel in major operations around the world. He also runs CCTV Surveillance Skills and Body Language, and Advanced Surveillance Body Language courses for CCTV operators, supervisors and managers internationally, and consults on CCTV management. He can be contacted on +27 11 787 7811 or [email protected]


Credit(s)




Share this article:
Share via emailShare via LinkedInPrint this page



Further reading:

A closed security concept for test halls and perimeter
Dallmeier Electronic Southern Africa Editor's Choice
At its factory facilities in Vilsbiburg, Germany, Flottweg SE relies on tailored video security technology from Dallmeier for perimeter security and workplace safety.

Read more...
Advanced server performance and energy efficient design
Editor's Choice IT infrastructure Products
Dell PowerEdge server portfolio expansion offers more performance, including up to 2.9x greater AI inferencing while Dell Smart Flow design and Dell Power Manager software advancements deliver greater energy efficiency.

Read more...
Free-to-use solar score for South African homes
Technews Publishing Editor's Choice
The LookSee Solar Score is one of the first of its kind to provide insight into the potential of solar power for South Africa’s residential properties.

Read more...
Fast, reliable and secure cloud services
Technews Publishing Editor's Choice Cyber Security IT infrastructure
Security and speed are critical components of today’s cloud-based services infrastructure. Cloudflare offers a range of services supporting these goals beyond what most people think it does.

Read more...
Fire-fighting force at Vergelegen
Editor's Choice Fire & Safety Residential Estate (Industry)
Vergelegen wine estate in Somerset West, and its neighbours, are set to enjoy greater peace of mind this summer, thanks to the delivery of a brand new fire truck .

Read more...
Sasol ensures Zero Trust for SAP financials with bioLock
Technews Publishing Editor's Choice Cyber Security Security Services & Risk Management
Multi-factor authentication, including biometrics, for SAP Financials from realtime North America prevents financial compliance avoidance for Sasol.

Read more...
KPMG 2022 CEO Outlook, South African edition
Editor's Choice News
Mid-November saw the release of the latest KPMG 2022 CEO Outlook, South African edition, aptly sub-titled ‘Potential Growth in Uncertain Times’.

Read more...
Do you know where your data is?
Technews Publishing Editor's Choice
Flow Security focuses on making sure companies manage their data security in real time through automated Data Security Posture Management (DSPM).

Read more...
Two cases of cyber resilience
Technews Publishing Editor's Choice
Infinidat consolidates backups and cyber resilience for a cloud service provider in the healthcare environment, as well as an energy utility based in EMEA.

Read more...
Are you below the security poverty line?
Technews Publishing Editor's Choice
While management may think their company is pulling its weight in terms of cybersecurity, the security team knows if it is operating below the security poverty line.

Read more...