An introduction to NIST FRVT

SMART Access & Identity 2024 Access Control & Identity Management

NIST’s Face Recognition Vendor Test (FRVT) is the most respected industry benchmark for companies building and using face recognition technology, and its benefit for the industry is undeniable. However, with the number of tests and the metrics’ complexity, the results can be challenging to parse and understand.

This article from Paravision and HID Global aims to give a high-level look into NIST FRVT and answer some of the frequently asked questions.

What is NIST FRVT?

The National Institute of Standards and Technology (NIST) is a non-regulatory agency of the US Department of Commerce that develops standards and guidelines for various industries, including biometrics.

The FRVT is a set of evaluations conducted by NIST to assess the performance of face recognition algorithms submitted by vendors from around the world. These evaluations aim to provide government agencies, industry, and the research community with unbiased, independent information about the performance of face recognition algorithms, empowering organisations to make informed decisions when selecting a face recognition system.

The NIST FRVT programme was established in 2000 to evaluate the performance of face recognition algorithms. The program has evolved over the years to include a variety of evaluations. These evaluations are designed to assess the performance of face recognition algorithms under a wide range of conditions, including varying lighting conditions, poses, and image qualities.

Different NIST FRVT tests

NIST FRVT tests are divided into two categories: Ongoing and Special. Ongoing tests are conducted regularly to evaluate the performance of face recognition algorithms submitted by vendors. In contrast, special tests are conducted on an as-needed basis to evaluate specific use cases or scenarios.

In all cases, NIST FRVT is performed on closed ‘Black box’ datasets. No vendors have access to any of the test images, which ensures that the submitted software is not trained specifically for NIST FRVT.

Ongoing NIST FRVT tests include:

FRVT 1:N: Considered the most challenging face recognition scenario, this test evaluates the performance of face recognition algorithms in a one-to-many matching, where a probe image is compared to multiple gallery images.

FRVT 1:1: Evaluates the performance of face recognition algorithms in a one-to-one matching scenario, where a probe image is compared to a single gallery image. The 1:1 test also includes specific results and metrics for demographic performance, providing accurate insight into how well facial recognition algorithms perform across genders, age groups, and countries of origin.

FRVT MORPH: Evaluates technologies detecting facial morphing (morphed/blended faces) in still photographs, as well as the resistance of face recognition algorithms against morphing.

FRVT QUALITY: Evaluates image quality algorithms by testing them on large sets of images and comparing their outputs with face recognition outcomes.

FRVT PAD: NIST’s newest upcoming test that evaluates the ability of face recognition algorithms to detect presentation attacks or spoofs.

Examples of special NIST FRVT tests include:

FRVT Paperless Travel: Tests face recognition algorithms in a paperless travel application through simulations of face recognition for boarding an aircraft and traversing an airport security point.

FRVT Face Mask Effects: Evaluates the performance of face recognition algorithms on masked faces. This test was created due to the widespread use of facial masks during the COVID-19 pandemic.

Out of the different types of NIST FRVT tests, the 1:1 and 1: N tests are considered to be the most general standard for facial recognition accuracy and performance.

Reading NIST FRVT results

Vendors can submit their algorithms multiple times to the ongoing tests, and the result tables often include multiple submissions from vendors, with individual report cards for each submission. To understand the results,

a reader must understand a few essential terms:

• False Negative Identification Rate (FNIR): Performance metric used in 1:N, showing the rate at which the face recognition system fails to identify a person who is present in the gallery.

• False Positive Identification Rate (FPIR): Performance metric used in 1:N, showing the rate at which the face recognition system incorrectly identifies a person as being present in the gallery when they are not.

• False Match Rate (FMR): Performance metric used in 1:1, showing the rate at which the face recognition system incorrectly matches two individuals.

• False Non-Match Rate (FNMR): Performance metric used in 1:1, showing the rate at which the face recognition system fails to match two individuals who are the same person.

• Gallery & probe types: Types of face images used for the testing, for example for visa, border, mugshot images and more.

It is crucial to understand the relationship between false positive and false negative matches to understand the performance of a face recognition algorithm. In an access control scenario, false positives are a security risk, as a false positive can mean a person’s face is matched with another person’s identity.

This is why the false positive rate is typically set at a very low threshold in NIST FRVT. In the same scenario, while not a security risk, false negatives increase inconvenience and processing time, as a person’s face is incorrectly marked as not matching their identity.

Necessary security thresholds differ between use cases, and even though a lower error rate is generally better, a balance should be struck between security and user experience.

Important considerations

NIST FRVT evaluations are vital for assessing the performance of face recognition algorithms. The variations of the tests used databases, and probe image types, and can provide an invaluable asset for companies comparing face recognition vendors and their algorithms. To best utilise NIST FRVT results, companies should consider how their use case compares with the different NIST FRVT testing types, probe images, and security thresholds.

It is also essential to understand that not all vendors offer their NIST-tested algorithms to partners or customers for commercial use, but might have separate algorithms for NIST testing and operational use. Organisations comparing face recognition vendors should always ask their partners if their NIST-tested algorithm is the same as that which is commercially available to partners.

Lastly, it is important to note that NIST FRVT occurs under specific conditions, which may or may not reflect the real world scenarios in which face recognition systems are used. In operational use, the performance of algorithms can be affected by factors such as lighting conditions, the quality of the camera, the network connection, and the processing power of the device, which might not be fully captured by NIST FRVT evaluations. These conditions may be better evaluated by other industry benchmarks such as the Department of Homeland Security’s Biometric Technology Rally, or internal use case-specific testing.

The DHS Biometric Technology Rally is an evaluation programme conducted by the United States Department of Homeland Security (DHS) to assess the performance of biometric technology, including face recognition, fingerprint recognition, and iris recognition.

This paper has been shortened, the full version is available at (or use the short link:*hid6).


Share this article:
Share via emailShare via LinkedInPrint this page

Further reading:

The future of access control is mobile
HID Global Mining (Industry)
The growing popularity of trusted ecosystems of cloud-connected access control devices, applications and trusted mobile identities has made mobile access easier to adopt than ever.

Defending against SIM swap fraud
Access Control & Identity Management
Mobile networks must not be complacent about SIM swap fraud, and they need to prioritise the protection of customers, according to Gur Geva, Founder and CEO of iiDENTIFii.

Access Selection Guide 2024
Access Control & Identity Management
The Access Selection Guide 2024 includes a range of devices geared specifically for the access control and identity management market.

Biometrics Selection Guide 2024
Access Control & Identity Management
The Biometrics Selection Guide 2024 incorporates a number of hardware and software biometric identification systems aimed at the access and identity management market of today.

Smart intercoms for Sky House Projects
Nology Access Control & Identity Management Residential Estate (Industry)
DNAKE’s easy and smart intercom solution has everything in place for modern residential buildings. Hence, the developer selected DNAKE video intercoms to round out upmarket apartment complexes, supported by the mobile app.

Authentic identity
HID Global Access Control & Identity Management
As the world has become global and digital, traditional means for confirming authentic identity, and understanding what is real and what is fake have become impractical.

Research labs secured with STid Mobile ID
Access Control & Identity Management
When NTT opened its research centre in Silicon Valley, it was looking for a high-security expert capable of protecting the company’s sensitive data. STid readers and mobile ID solutions formed part of the solution.

Is voice biometrics in banking secure enough?
Access Control & Identity Management AI & Data Analytics
As incidents of banking fraud grow exponentially and become increasingly sophisticated, it is time to question whether voice banking is a safe option for consumers.

Unlocking efficiency and convenience
OPTEX Access Control & Identity Management Transport (Industry)
The OVS-02GT vehicle detection sensor is the newest member of Optex’s vehicle sensor range, also known as ‘virtual loop’, and offers reliable motion detection of cars, trucks, vans, and other motorised vehicles using microwave technology.

Protecting our most vulnerable
NEC XON Access Control & Identity Management Products & Solutions
In a nation grappling with the distressing rise in child kidnappings, the need for innovative solutions to protect our infants has never been more critical. South Africa finds itself in the throes of a child abduction pandemic.