printer friendly version

An introduction to NIST FRVT

NIST’s Face Recognition Vendor Test (FRVT) is the most respected industry benchmark for companies building and using face recognition technology, and its benefit for the industry is undeniable. However, with the number of tests and the metrics’ complexity, the results can be challenging to parse and understand.

This article from Paravision and HID Global aims to give a high-level look into NIST FRVT and answer some of the frequently asked questions.

What is NIST FRVT?

The National Institute of Standards and Technology (NIST) is a non-regulatory agency of the US Department of Commerce that develops standards and guidelines for various industries, including biometrics.

The FRVT is a set of evaluations conducted by NIST to assess the performance of face recognition algorithms submitted by vendors from around the world. These evaluations aim to provide government agencies, industry, and the research community with unbiased, independent information about the performance of face recognition algorithms, empowering organisations to make informed decisions when selecting a face recognition system.

The NIST FRVT programme was established in 2000 to evaluate the performance of face recognition algorithms. The program has evolved over the years to include a variety of evaluations. These evaluations are designed to assess the performance of face recognition algorithms under a wide range of conditions, including varying lighting conditions, poses, and image qualities.

Different NIST FRVT tests

NIST FRVT tests are divided into two categories: Ongoing and Special. Ongoing tests are conducted regularly to evaluate the performance of face recognition algorithms submitted by vendors. In contrast, special tests are conducted on an as-needed basis to evaluate specific use cases or scenarios.

In all cases, NIST FRVT is performed on closed ‘Black box’ datasets. No vendors have access to any of the test images, which ensures that the submitted software is not trained specifically for NIST FRVT.

Ongoing NIST FRVT tests include:

• FRVT 1:N: Considered the most challenging face recognition scenario, this test evaluates the performance of face recognition algorithms in a one-to-many matching, where a probe image is compared to multiple gallery images.

• FRVT 1:1: Evaluates the performance of face recognition algorithms in a one-to-one matching scenario, where a probe image is compared to a single gallery image. The 1:1 test also includes specific results and metrics for demographic performance, providing accurate insight into how well facial recognition algorithms perform across genders, age groups, and countries of origin.

• FRVT MORPH: Evaluates technologies detecting facial morphing (morphed/blended faces) in still photographs, as well as the resistance of face recognition algorithms against morphing.

• FRVT QUALITY: Evaluates image quality algorithms by testing them on large sets of images and comparing their outputs with face recognition outcomes.

• FRVT PAD: NIST’s newest upcoming test that evaluates the ability of face recognition algorithms to detect presentation attacks or spoofs.

Examples of special NIST FRVT tests include:

• FRVT Paperless Travel: Tests face recognition algorithms in a paperless travel application through simulations of face recognition for boarding an aircraft and traversing an airport security point.

• FRVT Face Mask Effects: Evaluates the performance of face recognition algorithms on masked faces. This test was created due to the widespread use of facial masks during the COVID-19 pandemic.

Out of the different types of NIST FRVT tests, the 1:1 and 1: N tests are considered to be the most general standard for facial recognition accuracy and performance.

Reading NIST FRVT results

Vendors can submit their algorithms multiple times to the ongoing tests, and the result tables often include multiple submissions from vendors, with individual report cards for each submission. To understand the results,

a reader must understand a few essential terms:

• False Negative Identification Rate (FNIR): Performance metric used in 1:N, showing the rate at which the face recognition system fails to identify a person who is present in the gallery.

• False Positive Identification Rate (FPIR): Performance metric used in 1:N, showing the rate at which the face recognition system incorrectly identifies a person as being present in the gallery when they are not.

• False Match Rate (FMR): Performance metric used in 1:1, showing the rate at which the face recognition system incorrectly matches two individuals.

• False Non-Match Rate (FNMR): Performance metric used in 1:1, showing the rate at which the face recognition system fails to match two individuals who are the same person.

• Gallery & probe types: Types of face images used for the testing, for example for visa, border, mugshot images and more.

It is crucial to understand the relationship between false positive and false negative matches to understand the performance of a face recognition algorithm. In an access control scenario, false positives are a security risk, as a false positive can mean a person’s face is matched with another person’s identity.

This is why the false positive rate is typically set at a very low threshold in NIST FRVT. In the same scenario, while not a security risk, false negatives increase inconvenience and processing time, as a person’s face is incorrectly marked as not matching their identity.

Necessary security thresholds differ between use cases, and even though a lower error rate is generally better, a balance should be struck between security and user experience.

Important considerations

NIST FRVT evaluations are vital for assessing the performance of face recognition algorithms. The variations of the tests used databases, and probe image types, and can provide an invaluable asset for companies comparing face recognition vendors and their algorithms. To best utilise NIST FRVT results, companies should consider how their use case compares with the different NIST FRVT testing types, probe images, and security thresholds.

It is also essential to understand that not all vendors offer their NIST-tested algorithms to partners or customers for commercial use, but might have separate algorithms for NIST testing and operational use. Organisations comparing face recognition vendors should always ask their partners if their NIST-tested algorithm is the same as that which is commercially available to partners.

Lastly, it is important to note that NIST FRVT occurs under specific conditions, which may or may not reflect the real world scenarios in which face recognition systems are used. In operational use, the performance of algorithms can be affected by factors such as lighting conditions, the quality of the camera, the network connection, and the processing power of the device, which might not be fully captured by NIST FRVT evaluations. These conditions may be better evaluated by other industry benchmarks such as the Department of Homeland Security’s Biometric Technology Rally, or internal use case-specific testing.

The DHS Biometric Technology Rally is an evaluation programme conducted by the United States Department of Homeland Security (DHS) to assess the performance of biometric technology, including face recognition, fingerprint recognition, and iris recognition.

This paper has been shortened, the full version is available at https://www.hidglobal.com/sites/default/files/documentlibrary/Introduction_to_NIST_FRVT_WhitePaper_Paravision_HID.pdf (or use the short link: www.securitysa.com/*hid6).

Share this article:

Further reading: