Paradigm shift in the storage environment

May 2019 IT infrastructure

Tapes, removable disks, hard disks and solid-state drives (SSDs), also known as flash storage - there have been many technological advancements in the storage market and this is continuing to evolve. As artificial intelligence (AI) develops, the way we store data also promises to become smarter - and more efficient, flexible, and cost-effective for businesses.

Eran Brown
Eran Brown

Companies in South Africa and across the world that want to reliably store their ever-increasing amount of data (in the growing petabyte range) now have a large - and sometimes confusing - choice. So how can AI disrupt the current storage environment?

In principle, IT managers want data to be stored on high performing data carriers in order to ensure rapid access to the information at all times. But how sensible is it to store all data on the most powerful media at all times? For example, Flash is much faster than Near-Line Serial Attached SCSI (SAS) drives, however, it is also much more expensive. It makes little sense to store all your data on Flash, since most data is not used often.

In addition, there is data that must be stored for an extended period of time due to compliance rules, yet there is other data that may need to be accessed on a more regular basis such as for the preparation of long-term analyses. Even backup files do not have to be stored on Flash as they only come into play when restoring a data set. IT managers must therefore keep a constant eye on their data strategy in order to determine the optimal storage media for each application.

Time consuming manual approaches

Up to now, predefined policies have been used to determine exactly what data is stored where. Policies are established at the outset when the corresponding structures are created. They then remain largely the same, even if minor modifications take place during operational activities. Yet the amount and above all, the nature of the data is changing rapidly.

In the past, data was largely standardised due to the limited capacities and capabilities of the IT systems of that time, but today things look different. The constant manual adjustment of policies is becoming more complex and increasingly ties up personnel who can no longer perform other important tasks to their full extent. More complex data structures require more regular adjustments as the wrong choice of storage location can either burden the budget by using costly storage for irrelevant data or by disrupting operations through slower access to relevant data.

Ground-breaking solution using AI

So, how can this dilemma be solved? One way out is to use AI. With an automated method, adjustments can be made second-by-second without the need for manual intervention, allowing companies to use more cost-effective storage. Using machine learning, an AI engine can evaluate user behaviour and the nature of access to data and assign the storage location accordingly. In addition, it can project the patterns according to which accesses need to take place for future usage behaviour. This can also be used to make forecasts of the memory required and the performance required in the future, which can also be reflected in infrastructure and budget planning. An important goal here is to prevent the use of unnecessary resources.

Smart decisions via neural cache

For example, AI can be used via a neural cache, a technology that delivers lower latencies than flash by leveraging smart software algorithms. The machine learning algorithms scan the data pool and analyses data patterns to find hidden correlations. As a result, it decides which data is relevant for immediate access by applications or the user directly. Frequently used data is automatically stored in Random Access Memory (RAM) which is faster than Flash. Next is the ‘warm’ data, which is stored in Flash, and the less frequently used data is stored on Near-Line SAS drives, which are much more cost-effective.

In a storage array that combines Dynamic RAM (DRAM), Flash media and near-line SAS drives, the neural cache reduces latency and accelerates read/write access. Most applications are transactional, requiring at least two separate input/output (I/O) operations. One operation is always used to integrate the transaction to the logs, the other for the actual write operation of the data. This means that latencies can have an exorbitant effect on performance. Response times of the metadata layer thus affect the maximum performance of the application. Both read and write operations – insertions, changes, and deletions from the metadata structure – are processed with the same latency time.

These operations are performed without pre-processing such as pattern removal, compression, or encryption directly in the DRAM of the storage server. Meanwhile, a second copy of the write operation is made in the DRAM of another storage node with low latency RDMA and only then is a confirmation sent to the host. Writing directly to the DRAM connected to the server's central processing unit (CPU) results in lower overall latency than directly accessing an external flash device.

In addition, the use of a single large memory pool for accepting write access – unlike traditional architectures where the write cache is divided into smaller sections – ensures that larger write bursts can be maintained. Data that changes frequently can be overwritten with DRAM latency, allowing neural cache to intelligently decide which data blocks can be stored on which media. The longer retention of the data in the write cache means that CPU and back ends are relieved. The neural cache can also accelerate read operations by holding the most active data in the DRAM.

AI builds its experience by analysing large datasets of data and identifying patterns respectively features. It helps IT managers reduce their storage spending – which is already a top line-item in their budgets – and frees money to invest in innovation and transformation.

Share this article:
Share via emailShare via LinkedInPrint this page

Further reading:

Does your control room add value?
May 2019, Fidelity Security Group, G4S South Africa, Progroup , Editor's Choice, CCTV, Surveillance & Remote Monitoring, Integrated Solutions, IT infrastructure, Commercial (Industry)
Whether on- or offsite, control rooms are a critical aspect of security today and care must be taken in the design and rollout of these nerve centres.

Huawei’s Platform + Ecosystem strategy for 2019
May 2019, Huawei Technologies South Africa , News, Integrated Solutions, IT infrastructure, Conferences & Events
The annual Huawei Eco Connect event took place at the Sandton Convention Centre in Johannesburg and saw more than a 1000 attendees and housed exhibition stands from an array of partners.

As-a-Service solutions bolster information security
May 2019 , IT infrastructure, Integrated Solutions
The emerging concept of ‘Everything as a Service’ (XaaS) makes advanced technologies and solutions more cost effective and available to a broader spectrum of business.

Securing the Industrial Internet of Things
May 2019, Axiz , Cyber Security, IT infrastructure, Industrial (Industry)
The very benefits that makes the IIoT so compelling, makes it equally capable of damaging infrastructure operations and processes through bad actors.

Hybrid cloud and hyper-converged data protection
May 2019 , IT infrastructure, Cyber Security
Arcserve Unified Data Protection extends disaster recovery and backup technologies to prevent downtime and data loss for hyper-converged and SaaS-based workloads.

New wireless era with Wi-Fi 6
May 2019 , IT infrastructure, Products
Cisco recently announced new solutions that help customers embrace a new age of wireless connectivity incorporating Wi-Fi 6 (also known as 802.11ax), the new standard for Wi-Fi networks.

Anglo American’s underground Wi-Fi
May 2019 , IT infrastructure, Security Services & Risk Management
The introduction of underground Wi-Fi has improved productivity and working conditions for hundreds of miners at Anglo American’s coal mining operations in Mpumalanga.

Wi-Fi 6 will change your home network connectivity forever
May 2019, Duxbury Networking , Home Security, IT infrastructure, Residential Estate (Industry)
The new standard in wireless connectivity, Wi-Fi 6, gives you faster wireless experiences, with up to 40% higher peak data rates for a single device.

2019 Internet of Things (IoT) Barometer
March 2019 , Editor's Choice, Integrated Solutions, IT infrastructure
A majority of businesses that use IoT technology agree that it has either disrupted their industry or will do so in the next five years.

SECaaS a game changer for SA
April 2019, Specialised Exhibitions Montgomery , Integrated Solutions, IT infrastructure
SECaaS promises to comprehensively arm companies of all sizes with the tools they need to protect themselves from harmful cyberattacks.