Calibration of imaging systems typically involves the use of specially designed objects with known properties, called calibration targets. Standard calibration targets are, in general, used for calibrating the imaging pathway of fingerprint readers. However, there is no standard method for evaluating fingerprint readers in the operational setting where variations in finger placement by the user are encountered. The goal of this research is to design 3D targets for repeatable operational evaluation of fingerprint readers. 2D calibration patterns with known characteristics (e.g. sinusoidal gratings of pre-specified orientation and frequency, synthetic fingerprints with known singular points and minutiae) are projected onto a generic 3D finger surface to create electronic 3D targets (see Fig. 1). A state-of-the-art 3D printer (Stratasys Objet350 Connex) is used to fabricate the 3D targets with material similar in hardness and elasticity to the human finger skin. Our experimental results show that the (i) fabricated 3D targets can be imaged using three different (500/1000 ppi) commercial optical fingerprint readers, (ii) salient features in the 2D calibration patterns are preserved during the synthesis and fabrication of 3D targets, and (iii) intra-class variability between multiple impressions of the 3D targets captured using the optical fingerprint readers does not degrade the recognition accuracy. We also conduct experiments to demonstrate that the fabricated 3D targets can be used for operational evaluation of optical fingerprint readers (see Fig. 2).

Changes in human facial characteristics over time are inevitable. Hence, it is not surprising that face matching accuracy decreases with increasing elapsed time between two face images of the same person. With the deployment of automatic face recognition systems for de-duplication and law enforcement applications, it is crucial that we gain a better understanding of how facial aging due to time lapse affects recognition performance. Some people "age well" while the facial appearance of others changes drastically over time; this disparity mandates a subject-specific aging analysis. In this ongoing project, we use multilevel statistical models on a longitudinal database of 147,784 face images of 18,007 subjects (see Figs. 1 and 2) to quantify how time lapse affects match scores and recognition accuracy. A similar approach was used by Yoon and Jain [1] for fingerprint analysis and by Grother et al. [2] for iris analysis. Using two state-of-the-art COTS face matchers, we report (i) population-mean temporal trends in genuine scores and (ii) inter-subject variations, (iii) effects of other covariates (gender, race, face quality), and (iv) probability of true acceptance over time. Our preliminary findings suggest that false rejection rates of one of the COTS matchers remain stable up to approximately 10 years time interval.

Electronic health records (EHRs) capture comprehensive patient information in digital form from a variety of sources. Increasing availability of the EHRs has facilitated development of data and visual analytics tools, which significantly improved many aspects of healthcare analytics, especially clinical decision support and patient care management systems. The fundamental problems facing many medical data analytics tools include study of patient population, exploring complicated interactions among patients and their medical histories, and extracting phenotypes characterizing the patient population. Another challenge of EHRs is the visualization of the data in a way that clinicians can easily comprehend and interpret different patterns, and trends in patient populations. In this paper, we propose PHENOTREE a novel data-driven, hierarchical, and visually interactive phenotyping method via sparse principal component analysis (SPCA). Specifically, given the EHRs of a patient cohort the proposed framework can identify key features characterizing them using SPCA, and subtype those patients according to the identified key medical features. As such, the population can be iteratively refined (subtyped) according to the key medical features, and phenotypes at various levels of granularity can be formed. We provide an efficient SPCA algorithm and a visualization approach to illustrate the relations between diagnoses which helps to interpret the insights of patient populations in the cohort. The benefits of P HENO T REE are demonstrated through its application in the EHRs of two patient cohorts which are public and private datasets with 101,767 and 223,076 patients, respectively. Our evaluations show that PHENOT REE is beneficial to the detection of clinically meaningful hierarchical phenotypes.

Real-world machine learning applications involve many learning tasks that are inter-related. For example, in the healthcare domain, we need to learn a predictive model of a certain disease for many hospitals. The models for each hospital can be different because of the inherent differences in the distributions of the patient populations. However, the models are also closely related because of the nature of the learning tasks namely modeling the same disease. By simultaneously learning all the tasks, the multi-task learning (MTL) paradigm performs inductive knowledge transfer among tasks to improve the generalization performance of all tasks involved. When datasets for the learning tasks are stored in different locations, it may not always be feasible to move the data to provide a data-centralized computing environment, due to various practical issues such as high data volume and data privacy. This has posed a huge challenge to existing MTL algorithms. In this paper, we propose a principled MTL framework for distributed and asynchronous optimization. In our framework, the gradient update does not depend on and hence does not require waiting for the gradient information to be collected from all the tasks, making it especially attractive when the communication delay is too high for some task nodes. Empirical studies on both synthetic and real-world datasets demonstrate the efficiency and effectiveness of the proposed framework.

Investigations that require the exploitation of large volumes of face imagery are increasingly common in current forensic scenarios (e.g., Boston Marathon bombing), but effective solutions for triaging such imagery (i.e., low importance, moderate importance, and of critical interest) are not available in the literature. General issues for investigators in these scenarios are a lack of systems that can scale to volumes of images over 100K, and a lack of established methods for clustering the face images into the unknown number of persons of interest contained. As such, we explore best practices for clustering large sets of face images into large numbers of clusters as a method of reducing the volume of data to be investigated by forensic analysts (Fig. 1).

The popularity of face recognition has raised concerns about face spoof attacks (also known as biometric sensor presentation attacks), where a photo or video of an authorized person's face could be used to gain access to facilities or services. While a number of face spoof detection techniques have been proposed, their generalization ability has not been adequately addressed. In this project, we propose a new method for face liveness detection based on image quality analysis from 2D face spoof attacks (printed photos, video replays, etc.). Compared with widely used approaches based on face texture and motion analysis, the image quality analysis does not require face detection. Additionally, spoof detection based on image quality analysis should have better generalization ability than methods based on face texture or motion analysis. We also collect a face spoof database, MSU Mobile Face Spoofing Database (MSU MFSD), using two mobile devices (Google Nexus 5 and MacBook Air) with three types of spoof attacks (printed photo, replayed video with iPhone 5S, and iPad Air). The cross-database performance of individual face liveness detection methods is an important topic in this project.

In [1] we used image distortion analysis (IDA) to classify whether an input to a facial recognition system is of a genuine user or a spoof user. Four different features (specular reflection, blurriness, chromatic moment, and color diversity) are extracted to form the IDA feature vector. Experiments show our classifier based on IDA generalizes well, achieving state of the state performance, especially in cross-database testing, which portrays real world scenarios.

In [2] we address the problem of face spoof detection of replay attacks based on the analysis of aliasing in spoof videos. We analyze the moiré pattern aliasing that commonly appears during the recapture of video or photo replays on a screen in different channels (red, green, blue and grayscale) and regions (the whole face, detected face, facial component between the nose and chin). Multi-scale LBP and DSIFT features are used to represent the characteristics of moiré patterns. Experimental results on Idiap replay-attack and CASIA databases, as well as on a database we collected, Replay Attacks for Smartphones (RAFS), shows that the proposed approach is very effective in face spoof detection for both cross-database and intra-database testing scenarios. An Android application based on [2] has been developed and shows potential to be used for real world application.

Face retrieval is an enabling technology for many applications, including automatic face annotation, deduplication, and surveillance. In this project, we aim to build a face retrieval system which combines a k-nearest neighbor search procedure with a commercial-off-the-shelf (COTS) matcher in a cascaded manner (shown in Fig. 1). In particular, given a query face image, we first pre-filter the gallery set and find the top-k most similar faces for the query image by using deep facial features that are learned offline with a deep convolutional neural network. The top-k most similar faces are then re-ranked based on score-level fusion of the similarities between deep features and the COTS matcher.

The proposed retrieval system can address performance and scalability simultaneously. The retrieval system is evaluated on a large-scale web face database (5 million images). The gallery set is constructed with four different web face databases (Pubfig, LFW, WLFDB, and WebFace). All overlapping subjects are removed. The proposed system is compared with two commercial face matchers, and achieves the best performance.

	SOURCE	#SUBJECT	#IMAGES
QUERY SET	Pubfig	100	2,000
GALLERY SET	Pubfig	100	36,061
	LFW	5,446	11,173
	WLFDB	3,000	100,575
	WebFaces		4,881,187
TOTAL			5,000,000

Pubfig[1]	LFW[2]	WLFDB[3]	Web Faces

With a number of emerging applications requiring biometric recognition of children (e.g., tracking child vaccination schedules, identifying missing children and preventing newborn baby swaps in hospitals), investigating image capture of biometric traits of children and temporal stability of biometric recognition accuracy for children is important. Image capture and persistence of recognition accuracy of three of the most commonly used biometric traits (fingerprints, face and iris) has been investigated for adults. However, they have not been studied systematically for children in the age group of 0-4 years. Given that very young children are often uncooperative and do not comprehend or follow instructions, in our opinion, among all biometric modalities, fingerprints are the most viable for recognizing children. This is primarily because it is easier to capture fingerprints of young children compared to other biometric traits, e.g., iris, where a child needs to stare directly towards the camera to initiate iris capture. In this research, our goal is to investigate image capture and persistence of fingerprint recognition for children in the age group of 0-4 years. Based on preliminary results obtained for the data collected in the first phase of our study (see Figs. 1 and 2), use of fingerprints for recognition of 0-4 year-old children appears promising.

We study the problem of similarity learning and its application to image retrieval with large-scale data. The similarity between pairs of images can be measured by the distances between their high dimensional representations, and the problem of learning the appropriate similarity is often addressed by distance metric learning. However, distance metric learning requires the learned metric to be a PSD matrix, which is computational expensive and not necessary for retrieval ranking problem. On the other hand, the bilinear model is shown to be more flexible for large-scale image retrieval task, hence, we adopt it to learn a matrix for estimating pairwise similarities under the regression framework. By adaptively updating the target matrix in regression, we can mimic the hinge loss, which is more appropriate for similarity learning problem. Although the regression problem can have the closed-form solution, the computational cost can be very expensive. The computational challenges come from two aspects: the number of images can be very large and image features have high dimensionality. We address the first challenge by compressing the data by a randomized algorithm with the theoretical guarantee. For the high dimensional issue, we address it by taking low rank assumption and applying alternating method to obtain the partial matrix, which has a global optimal solution. Empirical studies on real world image datasets (i.e., Caltech and ImageNet) demonstrate the effectiveness and efficiency of the proposed method.

By the end of 2014, there were about 1.7 billion smartphone users worldwide. Mobile devices can carry large amounts of personal data but are often left unsecured. PIN locks are inconvenient to use and thus have seen low adoption (33% of users). While biometrics are beginning to be used for mobile device authentication, they are used only for initial unlock. Mobile devices secured with only login authentication are still vulnerable to theft in an unlocked state. This project studies a face-based continuous authentication system that operates in an unobtrusive manner. We present a methodology for fusing mobile device (unconstrained) face capture with gyroscope, accelerometer, and magnetometer data to correct for camera orientation and, by extension, the orientation of the face image. The project demonstrates (i) improvement of face recognition accuracy from face orientation correction, and (ii) efficacy of the prototype continuous authentication system. Additional face matching algorithms for matching unconstrained face images, and biometric traits such as motion, keystroke, and context will be studied.

Demographic estimation entails automatic estimation of age, gender and race of a person from his face image, which has many potential applications ranging from forensics to social media.Automatic demographic estimation, particularly age estimation, remains a challenging problem because persons belonging to the same demographic group can be vastly different in their facial appearances due to intrinsic and extrinsic factors. In this work, we present a generic framework (see Fig. 2) for automatic demographic (age, gender and race) estimation. Given a face image, we first extract demographic informative features via a boosting algorithm; we then employ a hierarchical approach consisting of between-group classification and within-group regression. Quality assessment is also developed to identify low-quality face images from which reliable demographic estimates are difficult to obtain. Experimental results on a diverse set of face image databases, FG-NET (1K images), FERET (3K images), MORPH II (75K images), PCSO (100K images), and a subset of LFW (4K images), show that the proposed approach has superior performance compared to the state of the art. We additionally use crowdsourcing to study the ability of human perception to estimate demographics from face images. A side-by-side comparison of the demographic estimates from crowdsourced data and the proposed algorithm provides a number of insights into this challenging problem.

A fundamental problem in machine learning is to assess the similarity or dissimilarity between objects, patterns, and data points. Distance metric learning addresses this fundamental issue by learning a distance metric from a set of training examples such that “similar” instances are close to each other while “dissimilar” instances are separated by a large distance. Distance metric learning has been applied to many tasks in computer vision, face detection and recognition, object tracking and video event detection. Nowadays high dimensional data appears in almost every application domain of machine learning. In order to capture diverse pattern in the visual content of images and temporal correlation among video frames, hundreds of thousands of features are extracted, leading to very high dimensional vector representations. Besides high dimensionality, the amount of data available for learning and mining has grown exponentially in recent years. The computational challenge of high dimensional DML arises from the fact that the number of parameters to be estimated increases quadratically in the number of dimensions. The computational challenge is further complicated by the constraint that the learned distance metric has to be a Positive Semi-Definite (PSD) matrix. To ensure a learned distance metric M to be PSD, we need to perform the eigenvalue decomposition of M, an expensive operation whose cost is cubic in the dimensionality. Another concern with high dimensional DML is the overfitting problem. Again, this is because the number of independent variables in distance metric grows quadratically in dimensionality. This project aims to develop efficient learning algorithms to explicitly address the computational challenges in large-scale high dimensional DML. Analysis for better understanding the generalization properties of regularized empirical minimization for high dimensional DML is also developed. In one of the studies conducted in this project, we developed a similarity learning algorithm [1]. We formulated the similarity learning as a matrix regression problem. High dimensionality problem is addressed by taking low rank assumption and solving partial matrix by alternating method. The large scale problem is tried to be solved my utilizing random projections. In the table, comparisons of mean average precisions and computational times for different metric learning methods and the method developed in our study which is denoted as SAM are given.

Lemurs are the world’s most endangered mammals. Analyzing individual lemur movements and interactions among a group of lemurs is essential for tracking the health of lemur populations. Current methods for identifying lemurs are limited to capturing and tagging individuals or visually identifying them via appearance. Tagging individual lemurs is expensive and disruptive to the population both in terms of causing injury and the possibility of group social dynamic shifts. Visual identification requires substantial expertise gained over time and produces results which are difficult to generalize.

Automatic facial recognition of humans is a mature technology and has many uses, from unlocking smartphones to finding suspects in surveillance video. A novel application of this technology, however, is recognizing other mammalian species. By creating a facial detection and recognition application to identify lemurs from images, this project will allow for simple and inexpensive tracking of individual lemurs in their habitat. The presence of distinctive facial features among lemurs allows for accurate recognition by extending and adapting techniques used in human face recognition to the facial anatomy of lemurs.

Kernel-based clustering algorithms achieve better performance on real world data than Euclidean distance-based clustering algorithms. However, kernel-based algorithms pose two important challenges:

• they do not scale sufficiently in terms of run-time and memory complexity, i.e. their complexity is quadratic in the number of data instances, rendering them inefficient for large data sets containing millions of data points, and
• the choice of the kernel function is very critical to the performance of the algorithm.

In this project, we aim at developing efficient schemes to reduce the running time complexity and memory requirements of these clustering algorithms and learn appropriate kernel functions from the data.

Approximate Kernel Clustering

In [1] and [2], we focus on reducing the complexity of the kernel k-means algorithm. The kernel k-means algorithm has quadratic runtime and memory complexity and cannot cluster more than a few thousands of points efficiently. In [1], we developed an algorithm called Approximate Kernel k-means which replaces the kernel similarity matrix in kernel k-means with a low rank approximate matrix, obtained by randomly sampling a small subset of the data set. The approximate kernel matrix is used to obtain the cluster labels for the data in an iterative manner (see Figure 1). The running time complexity of the approximate kernel k-means algorithm is linear in the number of points in the data set, thereby reducing the time for clustering by orders of magnitude (compared to the kernel k-means algorithm). For instance, the approximate kernel k-means algorithm is able to cluster 80 million images from the Tiny data set in about 8 hours on a single core, whereas kernel k-means requires several weeks to cluster them. The clustering accuracy of the approximate kernel k-means algorithm is close to that of the kernel k-means algorithm, and the difference in accuracy reduces linearly with the increase in the number of samples. The approximate kernel k-means algorithm can also be easily parallelized to further reduce the time for clustering. In [2], random projection is employed to project the data in the kernel space to a low-dimensional space, where the clustering can be performed using the k-means algorithm. Figure 2 shows some sample clusters from the Tiny data set obtained using these approximate algorithms.

Stream Kernel Clustering

Stream data is generated continuously and is unbounded in size. Stream data clustering, particularly kernel-based clustering, is additionally challenging because it is not possible to store all the data and compute the kernel similarity matrix. In [5], we devise a sampling based algorithm which selects a subset of the stream data to construct an approximate kernel matrix. Each data point is sampled as it arrives, with probability proportional to its importance in the construction of the low-rank kernel matrix. Clustering is performed in linear time using k-means, in a low-dimensional space spanned by the eigenvectors corresponding to the top eigenvalues of the kernel matrix constructed from the sampled points. Points which are not sampled are assigned labels in real-time by selecting the cluster center closest to them (Figure 3). We evaluated this algorithm on the Twitter stream, which was clustered with high accuracy at a speed of about 1 Mbps. Figure 4 shows sample tweets from the cluster representing the #HTML hashtag.

A computer aided system for histological prostate cancer diagnosis can assist pathologists by automating two stages. The first is to locate cancer regions in the large digitized tissue biopsy and the second is to assign grades to those regions. In this project we address both of the stages. In the first stage, besides well-known textural features, we use cytological features which are not utilized in the Gleason grading method to detect cancer regions in a whole slide tissue image at 20x magnification (the size is approximately 5,000x20,000 pixels). One of the most important cytological features is the presence of cancer nuclei (nuclei with prominent nucleoli) in the cancer regions. Both cytological and textural features are extracted from the tissue image and are combined to detect cancer regions.

In the second stage, motivated by the Gleason grading method, we aim at classifying a pre-selected region of interest (ROI) into one of the three common categories: benign, grade 3 and grade 4. In the benign ROIs, glands are well-defined with lumina regions in the center, circumscribed by epithelial nuclei and epithelial cytoplasm. In grade 3 ROIs, there appear small, circular glands with thin boundary. Finally, glands start to fuse with each other in grade 4 ROIs. As a result, glands in grade 4 regions do not maintain the well-defined structures as in benign regions. We propose a segmentation-based approach in which we (i) first segment glands from the tissue region, (ii) extract structural features from these segmented glands and (iii) classify the ROI into one of the three classes.

With the large scale deployment of biometric systems, a number of security and privacy issues are arising. It was shown that the stored fingerprint minutiae templates can be used to reconstruct the whole fingerprint and thus spoof fingerprints can be easily created to compromise the systems. Further, cross-linking of databases would be possible leading to privacy compromise. To curb such vulnerabilities of a biometric system, a number of template protection techniques are designed that either allow template revocation (called feature transformation based techniques) or keep the system secure even if the stored information is leaked (called biometric cryptosystems). Still significant advances are being made to design techniques that satisfy requirements of revocability as well as biometric cryptosystems and still have high matching performance.

Automatic Fingerprint Identification Systems (AFIS) have for a long time used only minutiae for fingerprint matching. But minutiae are only a small subset of fingerprint details routinely used by latent examiners for fingerprint matching. This has generated a lot of interest in extended feature set (EFS) with the aim of narrowing down the gap between the performance of AFIS and latent examiners. Typical extended features include dots, incipient ridges, pores, and ridge edge features, etc. See Fig. 1. Although some researchers reported the effectiveness of extended features in improving automatic fingerprint recognition accuracy by using their own private fingerprint databases and their own in-house fingerprint matchers, it is still not clear if and how much extended features can benefit AFIS when low quality fingerprint images (such as ink and latent fingerprints) are used and when commercial off-the-shelf (COTS) fingerprint matchers are used as the baseline. The goals of this project are: (i) design automatic extraction and matching algorithms for extended features (particularly, pores, dots, incipient ridges, and ridge edge protrusions), (ii) determine the utility of extended features in matching low quality fingerprints and latent fingerprints, (iii) explore how to incorporate extended features into existing AFIS, and (iv) make recommendations regarding the effectiveness of extended features. Fig. 2 shows the pores extracted by using our extraction algorithm in an example fingerprint image, and Fig. 3 shows the mated pores found by using our matching algorithm in a latent and its exemplar fingerprint. According to our preliminary results (see Fig. 4) of matching 131 partial inked fingerprint images against 4,180 rolled inked fingerprint images (all images are of 1,000 ppi), we observed that i) minutiae matchers usually work very well when there is a sufficient number of minutiae, and ii) the contribution of extended features is more effective for fingerprints which have few minutiae or low minutiae match scores.

Face recognition systems typically have a rather short operating distance with standoff (distance between the camera and the subject) limited to 1~2 meters. When these systems are used to capture face images at a larger distance (5 m), the resulting images contain only a small number of pixels on the face region, resulting in a degradation in face recognition performance. To address this problem, we propose dual camera system consisting of PTZ and static cameras to acquire high resolution face images up to a distance of 12 meters. The proposed camera system utilizes the coaxial and concentric configuration between the static and PTZ cameras to achieve distance invariance PTZ camera control. We also use a linear prediction model and camera control scheme to mitigate delays in image processing and mechanical camera motion. The proposed system has a larger standoff in face image acquisition and effectiveness in face recognition test.

The widespread deployment of Automated Fingerprint Identification Systems (AFIS) in law enforcement and border control applications has heightened the need for ensuring that these systems are not compromised. While several issues related to fingerprint system security have been investigated, including the use of fake fingerprints for masquerading identity, the problem of fingerprint alteration or obfuscation has received very little attention. Fingerprint obfuscation refers to the deliberate alteration of the fingerprint pattern by an individual for the purpose of masking his identity. Several cases of fingerprint obfuscation have been reported in the press (see Fig. 1). Fingerprint image quality assessment software (e.g. NFIQ) cannot always detect altered fingerprints since the implicit image quality due to alteration may not change significantly. The goals of this research include: (i) analyzing altered fingerprints based on a large altered fingerprint database provided to us by a law enforcement agency, (ii) detecting altered fingerprints automatically at a very low false positive rate (i.e., a natural/unaltered fingerprint is misclassified as altered fingerprint), (iii) restoring altered fingerprints in possible cases, and (iv) matching altered fingerprints to their unaltered mates in the database.

Relevant Publication(s)

S. Yoon, J. Feng, and A. K. Jain, "Altered Fingerprints: Analysis and Detection", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 34, No. 3, pp. 451-464, March 2012.

S. Yoon, Q. Zhao, and A. K. Jain, "On Matching Altered Fingerprints", ICB, New Delhi, India, March 29-April 1, 2012.

A. K. Jain and S. Yoon, "Automatic Detection of Altered Fingerprints", IEEE Computer, Vol. 45, No. 1, pp. 79-82, January, 2012.

J. Feng, A. K. Jain and A. Ross, "Detecting Altered Fingerprints", ICPR, Istanbul, Turkey, August 23-26, 2010.

Fingerprint

Extended Feature Set for Fingerprint Matching

There are fundamental differences in the way fingerprints are compared by forensic experts and current Automatic Fingerprint Systems (AFIS). For example, AFIS systems focus mainly on the quantitative measures of fingerprint minutiae (ridge ending and bifurcation points), while latent experts often analyze details of intrinsic ridge characteristics and relational information. This alternate process includes examination of an extended feature set of minutiae shape, dots, incipient ridges, local ridge quality, ridge tracing, etc. However, most of the features used by latent experts have not even been quantitatively defined for AFIS matching. This project aims to develop algorithms that automatically extract and match extended features.

A.K. Jain, Y. Chen and M. Demirkus, " Pores and Ridges: High Resolution Fingerprint Matching Using Level 3 Features", IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006.

Y. Chen, M. Demirkus and A.K. Jain, " Pores and Ridges: Fingerprint Matching Using Level 3 Features", Proc. of International Conference on Pattern Recognition (ICPR), Vol. 4, pp. 477-480, Hong Kong, August, 2006.

Multispectral Fingerprint Matching

Multispectral (MS) fingerprint imaging systems use different wavelengths of light to illuminate the surface and the subsurface layers of the finger skin and capture the reflected light. The resulting fingerprint images and the combination of these images provide more discriminatory and robust information about the characteristics of the fingerprint than those from a TIR based optical fingerprint sensor. In that sense, we analyze the performance of different fingerprint matching algorithms on MS fingerprint images and explore new features that can be extracted from each image band.

Individuality of Fingerprints

The question of fingerprint individuality can be posed as follows: Given a query fingerprint, what is the probability that the observed number of minutiae matches with a template fingerprint is purely due to chance? An assessment of this probability can be made by estimating the variability inherent in fingerprint minutiae. We develop a compound stochastic model that is able to capture three main sources of minutiae variability in actual fingerprint databases. The compound stochastic models are used to synthesize realizations of minutiae matches from which numerical estimates of fingerprint individuality can be derived. Experiments on the FVC2002DB1 and IBMHURSLEY databases show that the probability of obtaining a 12 minutiae match purely due to chance is 1.6×10−5 when the number of minutiae in the query and template fingerprints are both 46.

Y. Zhu, S. C. Dass, and Anil K. Jain, " Compound Stochastic Models for Fingerprint Individuality", Proc. of International Conference on Pattern Recognition (ICPR), Vol. 3, pp. 532-535, Hong Kong, August, 2006.

S. C. Dass, Y. Zhu and Anil K. Jain, " Statistical models for assessing the individuality of fingerprints", Fourth IEEE workshop on Automatic Identification Advanced Technologies, pages 1-7,2005.

S. Pankanti, S. Prabhakar, and A. K. Jain, "On the Individuality of Fingerprints", IEEE Transactions on PAMI, Vol. 24, No. 8, pp. 1010-1025, 2002. A shorter version also appears in Fingerprint Whorld, pp. 150-159, July 2002.

Fingerprint Mosaicking

It has been observed that the reduced contact area offered by solid-state fingerprint sensors do not provide sufficient information (e.g., minutiae) for high accuracy user verification. Further, multiple impressions of the same finger acquired by these sensors, may have only a small region of overlap thereby affecting the matching performance of the verification system. To deal with this problem, we suggest a fingerprint mosaicking scheme that constructs a composite fingerprint image using multiple impressions. In the proposed algorithm, two impressions of a finger are initially aligned using the corresponding minutiae points. This alignment is used by the well-known iterative closest point algorithm (ICP) to compute a transformation matrix that defines the spatial relationship between the two impressions. The transformation matrix is used in two ways: (a) the two impressions are stitched together to generate a composite image. Minutiae points are then detected in this composite image. (b) the minutia maps obtained from each of the individual impressions are integrated to create a larger minutia map. The availability of a composite template improves the performance of the fingerprint matching system as is demonstrated in our experiments.

A. K. Jain and A. Ross, " Fingerprint Mosaicking", Proc. International Conference on Acoustic Speech and Signal Processing (ICASSP), Orlando, Florida, May 13-17, 2002.

Hybrid Fingerprint Matcher

A fingerprint matcher that uses both minutiae and texture information present in fingerprints has been developed. A set of 8 Gabor filters are used to extract texture information inherent in fingerprints. Minutiae and/or core information is used to align two fingerprints. The hybrid matcher is shown to exhibit superior matching performance compared to a purely minutiae-based matcher.

A. Ross, A. K. Jain, and J. Reisman, " A Hybrid Fingerprint Matcher", Pattern Recognition, Vol. 36, No. 7, pp. 1661-1673, 2003.

A. Ross, J. Reisman and A. K. Jain, " Fingerprint Matching Using Feature Space Correlation", Proc. of Post-ECCV Workshop on Biometric Authentication, Copenhagen, Denmark, June 1, 2002.

A. K. Jain, A. Ross, and S. Prabhakar, " Fingerprint Matching Using Minutiae and Texture Features", Proc. International Conference on Image Processing (ICIP), Greece, October 7-10, 2001.

A. K. Jain, S. Prabhakar, L. Hong and S. Pankanti "Filterbank-based Fingerprint Matching", IEEE Transactions on Image Processing, Vol. 9, No.5, pp. 846-859, May 2000.

Fingerprint Classification

Fingerprint classification can provide an important indexing mechanism in a fingerprint database. An accurate and consistent classification can greatly reduce fingerprint matching time for large databases. In this paper, we present a fingerprint classification algorithm which is able to achieve an accuracy better than previously reported in the literature. We classify fingerprints into five categories: whorl, right loop, left loop, arch, and tented arch. The algorithm separates the number of ridges present in four directions (0, 45, 90, and 135 degrees) by filtering the central part of a fingerprint with a bank of Gabor filters. This information is quantized to generate a FingerCode which is used for classification. Our classification is based on a two-stage classifier which uses a K-nearest neighbor classifier in the first stage and a set of neural networks in the second stage. The classifier is tested on 4,000 images in the NIST-4 database. For the five-class problem, classification accuracy of 90% is achieved. For the four-class problem (arch and tented arch combined into one class), we are able to achieve a classification accuracy of 94.8%. By incorporating a reject option, the classification accuracy can be increased to 96% for the five-class classification and to 97.8% for the four-class classification when 30.8% of the images are rejected.

S. Dass and A. K. Jain," Fingerprint Classification Using Orientation Field Flow Curves", Proc. of Indian Conference on Computer Vision, Graphics and Image Processing, (Kolkata), pp. 650-655, December 2004.

A. K. Jain and S. Minut, "Hierarchical Kernel Fitting for Fingerprint Classification and Alignment", Proc. of International Conference on Pattern Recognition, Quebec City, August 11-15, 2002.

A. K. Jain, S. Prabhakar and L. Hong, " A Multichannel Approach to Fingerprint Classification", IEEE Transactions on PAMI, Vol.21, No.4, pp. 348-359, April 1999.

Distinguishing Identical Twins Using Fingerprints

Automatic identification methods based on physical biometric characteristics such as fingerprint or iris can provide positive identification with a very high accuracy. However, the biometrics-based methods assume that the physical characteristics of an individual (as captured by a sensor) used for identification are distinctive. Identical twins have the closest genetics-based relationship and, therefore, the maximum similarity between fingerprints is expected to be found among identical twins. We show that a state-of-the-art automatic fingerprint identification system can successfully distinguish identical twins though with a slightly lower accuracy than nontwins.

A. K. Jain, S. Prabhakar, and S. Pankanti, " On The Similarity of Identical Twin Fingerprints", Pattern Recognition, Vol. 35, No. 11, pp. 2653-2663, 2002.

Combination of Fingerprint Matchers

Different fingerprint matching algorithms may use different type of information extracted from the input fingerprints and hence complement each other. Integration of fingerprint matching algorithms is a viable way to improve the performance of a fingerprint verification system. In this paper, we use logistic transform to integrate the output scores from two different fingerprint matching algorithms. Each set of four parameters for a specified false acceptance rate (FAR) is obtained through supervised learning: the four parameters are adjusted so that the false rejection rate (FRR) is minimized for a given FAR. This results in optimizing a function with an unknown analytical form; hence the commonly used gradient-descent learning with an artificial neural network is not applicable. The optimization is solved by Brent's efficient numerical algorithm without the use of derivatives. Experiments conducted on a large fingerprint data set confirme the effectiveness of the proposed integration scheme.

A. K. Jain, S. Prabhakar and S. Chen, " Combining Multiple Matchers for a High Security Fingerprint Verification System", Pattern Recognition Letters, Vol 20, No. 11-13, pp. 1371-1379, 1999.

S. Prabhakar and A. K. Jain, " Decision-level Fusion in Fingerprint Verification" Pattern Recognition, Vol. 35, No. 4, pp. 861-874, 2002.

---

Face

3D Face Recognition

The performance of face recognition systems that use two-dimensional (2D) images is dependent on consistent conditions such as lighting, pose, and facial expression. A multi-view face recognition system is being developed, which utilizes three-dimensional (3D) information about the face, along with the facial texture, to make the system more robust to those variations. A procedure is presented for constructing a database of 3D face models and matching this database to 2.5D face scans which are captured from different views. 2.5D is a simplified 3D (x, y, z) surface representation that contains at most one depth value (z direction) for every point in the (x, y) plane. A robust similarity metric is defined for matching. To address the non-rigid facial movement, such as expressions, we present a facial surface modeling and matching scheme to match 2.5D test scans in the presence of both non-rigid deformations and large pose changes (multiview) to a neutral expression 3D face model. A geodesic-based resampling approach is applied to extract landmarks for modeling facial surface deformations. We are able to synthesize the deformation learned from a small group of subjects (control group) onto a 3D neutral model (not in the control group), resulting in a deformed template. A personspecific (3D) deformable model is built for each subject in the gallery w.r.t. the control group by combining the templates with synthesized deformations. By fitting this generative deformable model to a test scan, the proposed approach is able to handle expressions and large pose changes simultaneously.

X. Lu and A. K. Jain, " Deformation Modeling for Robust 3D Face Matching", Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR2006), Vol. 2, pp. 1377 - 1383, New York, NY, Jun. 2006.

X. Lu and A.K. Jain, " Integrating range and texture information for 3D face recognition", Proc. of WACV (Workshop on Applications of Computer Vision), pp. 156-163, Breckenridge, Colorado, January 2005.

X. Lu and A.K. Jain, " Deformation Analysis for 3D Face Matching", Proc. of WACV (Workshop on Applications of Computer Vision), pp. 99-104, Breckenridge, Colorado, January 2005.

X. Lu, D. Colbry and A. K. Jain, " Three-Dimensional Model Based Face Recognition", Proc. International Conference on Pattern Recognition (ICPR), vol. I, pp. 362-366, Cambridge, UK, August 2004.

X. Lu, D. Colbry and A. K. Jain, " Matching 2.5D Scans for Face Recognition", Proc. International Conference on Biometric Authentication (ICBA) , pp. 30-36, Hong Kong, July 2004.

X. Lu, R. Hsu, A. Jain, B. Kamgar-Parsi and B. Kamgar-Parsi, Face Recognition with 3D Model-Based Synthesis, Proc. International Conference on Biometric Authentication (ICBA), pp. 139-146, Hong Kong, July 2004.

Face Recognition in Video

Face recognition in video has gained wide attention as a covert method for surveillance to enhance security in a variety of application domains (e.g., airports). A video contains temporal information as well as multiple instances of a face, so it is expected to lead to better face recognition performance compared to still face images. However, faces appearing in a video have substantial variations in pose and lighting. These pose and lighting variations can be effectively modeled using 3D face models. Combining the advantages of 2D video and 3D face models, we propose a face recognition system that identifies faces in a video. The system utilizes the rich information in a video and overcomes the pose and lighting variations using 3D face model. The 3D face models are obtained from a 3D range sensor and stereographic reconstruction process. Experimental results have shown that both 3D face models provide better face recognition performance by compensating pose and lighting variations.

U. Park and A.K.Jain, " 3D Face Reconstruction from Stereo Video", Proc. First International Workshop on Video Processing for Security (VP4S-06) in Third Canadian Conference on Computer and Robot Vision (CRV06), June 7-9, Quebec City, Canada, 2006.

U. Park, H. Chen and A. K. Jain, " 3D Model-assisted Face Recognition in Video", Proc. of 2nd Workshop on Face Processing in Video, in conjuction with AI/GI/CRV05, pp. 322-329, Victoria, British Columbia, Canada, May 2005.

Face Detection

Human face detection is often the first step in applications such as video surveillance, human computer interface, face recognition, and image database management. We propose a face detection algorithm for color images in the presence of varying lighting conditions as well as complex backgrounds. Our method detects skin regions over the entire image, and then generates face candidates based on the spatial arrangement of these skin patches. The algorithm constructs eye, mouth, and boundary maps for verifying each face candidate. Experimental results demonstrate successful detection over a wide variety of facial variations in color, position, scale, rotation, pose, and expression from several photo collections.

R.-L. Hsu, Mohamed Abdel-Mottaleb and A. K. Jain, "Face Detection in Color Images", IEEE Transactions on PAMI, vol. 24, no.5, pp. 696-706, May 2002.

R.-L. Hsu, M. Abdel-Mottaleb, and A. K. Jain, " Face detection in color images", Proc. International Conference on Image Processing (ICIP) , Greece, October 7-10, 2001.

Sarat C. Dass and A. K. Jain, "Markov Face Models", The Eighth IEEE International Conference on Computer Vision (ICCV), Vancouver, Canada, July 9-12, 2001.

Face Modeling

3D Human face models have been widely used in applications such as face recognition, facial expression recognition, human action recognition, head tracking, facial animation, video compression/coding, and augmented reality. Modeling human faces provides a potential solution to the variations encountered on human face images. We propose a method of modeling human faces based on a generic face model (a triangular mesh model) and individual facial measurements containing both shape and texture information. The modeling method adapts a generic face model to the given facial features, extracted from registered range and color images, in a global to local fashion. It iteratively moves the vertices of the mesh model to smoothen the non-feature areas, and uses the 2.5D active contours to refine feature boundaries. The resultant face model has been shown to be visually similar to the true face. Initial results show that the constructed model is quite useful for recognizing profile views. sensors.

R.-L. Hsu and A. K. Jain, " Semantic face matching", Proc. IEEE Int'l Conf. Multimedia and Expo (ICME) , Lausanne, Switzerland, Aug. 2002.

R.-L. Hsu and A. K. Jain, " Face modeling for recognition", Proc. International Conference on Image Processing (ICIP) , Greece, October 7-10, 2001.

Combination of Face Matchers

Current two-dimensional face recognition approaches can obtain a good performance only under constrained environments. However, in real applications, face appearance changes significantly due to different illumination, pose, and expression conditions. Face recognizers based on different representations of the input face images have different sensitivity to these variations. Therefore, a combination of several face classifiers which can integrate the complementary information should lead to improved classification accuracy. We use the sum rule and RBF-based integration strategies to combine three commonly used face classifiers based on PCA, ICA and LDA representations. Experiments conducted on a face database containing 206 subjects (2,060 face images) show that the proposed classifier combination approaches outperform individual classifiers.

X. Lu, Y. Wang and A. K. Jain, " Combining Classifiers for Face Recognition", Proc. ICME 2003, IEEE International Conference on Multimedia & Expo, vol. III, pp. 13-16, Baltimore, MD, July 6-9, 2003.

---

Dental Biometrics

Dental Biometrics

The goal of forensic dentistry is to identify people based on their dental records, mainly as radiograph images. In this paper we attempt to set forth the foundations of a biometric system for semi-automatic processing and matching of dental images, with the final goal of human identification. Given a dental record, usually as a postmortem (PM) radiograph, we need to search the database of antemortem (AM) radiographs to determine the identity of the person associated with the PM image.We use a semi-automatic method to extract shapes of the teeth from the AM and PM radiographs, and find the affine transform that best fits the shapes in the PM image to those in the AM images. A ranking of matching scores is generated based on the distance between the AM and PM tooth shapes. Initial experimental results on a small database of radiographs indicate that matching dental images based on tooth shapes and their relative positions is a feasible method for human identification.

H. Chen and A. K. Jain, " Tooth Contour Extraction for Matching Dental Radiographs", Proc. ICPR 2004, vol. III, pp. 522-525, Cambridge, UK, August 2004.

G. Fahmy, D. Nassar, E. Haj-Said, H. Chen, O. Nomir, J. Zhou, R. Howell, H. H. Ammar, M. Abdel-Mottaleb and A. K. Jain, "Towards an Automated Dental Identification System (ADIS)", Proceedings of the International Conference on Biometric Authentication (ICBA), Hong Kong, July 2004.

A. K. Jain and H. Chen, " Matching of Dental X-ray Images for Human Identification ", Pattern Recognition, Vol. 37, No. 7, pp. 1519-1532, July 2004.

A. K. Jain, H. Chen and S. Minut, " Dental Biometrics: Human Identification Using Dental Radiographs", Proc. of 4th Int'l Conf. on Audio- and Video-Based Biometric Person Authentication (AVBPA), pp. 429-437, Guildford, UK, June 9-11, 2003.

---

Online Signature

Signature Verification

We describe a method for handwritten signature verification. The signatures are acquired using a digitizing tablet which captures both, dynamic and spatial information of the writing. After preprocessing the signature, several features are extracted. The authenticity of a writer is determined by comparing an input signature to a stored reference set (template) consisting of three signatures. The similarity between an input signature and the reference set is computed using string matching and the similarity value is compared to a threshold. Experiments on a database containing 1,232 signatures of 102 individuals show that user-specific thresholds yield better results. Several approaches to obtain the optimal threshold value from the reference set are investigated. The best result yields a false acceptance rate of 1.6% and a false reject rate of 2.8%.

A.K. Jain, Friederike D. Griess and Scott D. Connell, "On-line Signature Verification", Pattern Recognition, vol. 35, no. 12, pp. 2963--2972, Dec 2002.
Friederike D. Griess and Anil K. Jain, " On-line Signature Verification", MSU Technical Report TR00-15, 2000.

Biometric Template Selection and Update

The matching accuracy of a biometrics-based authentication system is affected by the stability of the biometric data associated with the users of the system. But due to factors such as improper interaction with the sensor, variations in environmental factors and temporary alterations of the biometric trait itself, biometric data have a large intra-class variability. Hence biometric template data can be significantly different from samples obtained during authentication. To alleviate this problem, multiple templates, instead of a single template, per user can be stored in a biometric database. In this project, we study techniques to automatically select and update templates by considering system performance along with storage and computational overheads associated with multiple templates.

U. Uludag, A. Ross and A. K. Jain, " Biometric Template Selection and Update: A Case Study in Fingerprints," Pattern Recognition, Vol. 37, No. 7, pp. 1533-1542, July 2004.

A. K. Jain, U. Uludag and A. Ross, " Biometric Template Selection: A Case Study in Fingerprints", Proc. of 4th Int'l Conf. on Audio- and Video-Based Biometric Person Authentication (AVBPA), pp. 335-342, Guildford, UK, June 9-11, 2003.

Image Quality

Image quality is one of the variables that has the largest effect on biometric system accuracy and is the major cause of high false reject rates (FRR) and failure to enroll (FTE) rates. Automatic image quality measures can be developed to (i) provide real-time feedback to reduce the number of poor quality submissions to the system, (ii) predict and improve the authentication performance using quality-dependent thresholds, and (iii) generate quality-based user (or modality) specific weights for multi-biometric fusion.

Y. Chen, S. Dass and A. Jain, " Localized Iris Image Quality Using 2-D Wavelets", Proc. of International Conference on Biometrics (ICB), pp. 373-381, Hong Kong, January, 2006.

J. Fierrez-Aguilar, Y. Chen, J. Ortega-Garcia and A. K. Jain, " Incorporating image quality in multi-algorithm fingerprint verification", Proc. of International Conference on Biometrics (ICB), pp. 213-220, Hong Kong, January, 2006.

Y. Chen, S. Dass and A. Jain, " Fingerprint Quality Indices for Predicting Authentication Performance", Proc. of Audio- and Video-based Biometric Person Authentication (AVBPA), pp. 160-170, Rye Brook, NY, July 2005.

K. Nandakumar, Y. Chen, S.C. Dass and A.K. Jain, " Quality-based Score Level Fusion in Multibiometric Systems", Proc. of International Conference on Pattern Recognition (ICPR), Vol. 4, pp. 473-476, Hong Kong, August, 2006.

Sample Size Requirement for Performance Evaluation

Authentication systems based on biometric features (e.g., fingerprint impressions, iris scans, human face images, etc.) are increasingly gaining widespread use and popularity. Often, vendors and owners of these commercialbiometric systems claim impressive performance that is estimated based on some proprietary data. In such situations, there is a need to independently validate the claimed performance levels. System performance is typically evaluated by collecting biometric templates from n different subjects, and for convenience, acquiring multiple instances of the biometric for each of the n subjects. Very little work has been done in (i) constructing confidence regions based on the ROC curve for validating the claimed performance levels, and (ii) determining the required number of biometric samples needed to establish confidence regions of pre-specified width for the ROC curve. To simplify the analysis that address these two problems, several previous studies have assumed that multiple acquisitions of the biometric entity are statistically independent. This assumption is too restrictive and is generally not valid. We have developed a validation technique based on multivariate copula models for correlated biometric acquisitions. Based on the same model, we also determine the minimum number of samples required to achieve confidence bands of desired width for the ROC curve. We illustrate the estimation of the confidence bands as well as the required number of biometric samples using a fingerprint matching system that is applied on samples collected from a small population.

S. C. Dass, Y. Zhu and Anil K. Jain, " Validating a Biometric Authentication System: Sample Size Requirements", IEEE Transactions of Pattern Analysis and Machine Intelligence, pp. 1902-1913, 2006.

(a)	(b)	(c)	(d)

A key asset in identification using face recognition technology is the extensive collection of face databases (photographs). The source of these databases range from ID card images (e.g. driver license and passport), visa applications (e.g. U.S. VISIT), and criminal mug shot photographs. Being populated with visible light photographs, commercial-of-the-shelf (COTS) face recognition systems (FRS) used to search these databases expects incoming visible light probe (query) photographs. However, many military and law enforcement scenarios exist where a probe face image is only available in an alternate sensing modality. Consider the following examples: (i) when acquiring a face image in environments with unfavorable illumination (such as night time) or in covert military and intelligence scenarios, infrared imaging must be used to capture a subjectв ¬ !" s face, and (ii) when no opportunity exists to acquire a face image, a forensic sketch must be drawn from a verbal description provide by a witness to a crime. In each of these scenarios the probe image will be from an alternate modality (infrared or sketch) than the images in the gallery (visible light photograph). This identification task, where the probe and gallery images are from different modalities, is called heterogeneous face recognition (HFR). Since only limited data is available on face images in no-visible spectrum (such as a database of infrared face images), accurate HFR systems are vital to enable identification in many critical situations. A COTS FRS performs poorly in heterogeneous face recognition. As a result, specialized matching systems must be developed to enable identification in these sensitive scenarios. This project aims at developing both generic solutions to multiple heterogeneous face recognition scenarios, as well as tailored solutions to specific HFR scenarios.

We are interested in the problem of multi-label learning with many classes, where the aim is assigning each instance to related categories (Figure 1). Our goal is to develop kernel methods for multi-label learning. Unlike the conventional approaches that implement multi-label learning as a set of binary classification problems, our goal is to develop direct approaches that would exploit relationships between the class labels, and we proposed multi-label ranking methods for this task (see references below). Moreover, since kernel learning, or more specifically Multiple Kernel Learning (MKL), has attracted considerable amount of interest in computer vision community, we proposed an efficient algorithm for multi-label multiple kernel learning (ML-MKL) – see reference below. We assume that all the classes under consideration share the same combination of kernel functions, and the objective is to find the optimal kernel combination that benefits all the classes.

We also consider a special type of multi-label learning where class assignments of training examples are incomplete. As an example, an instance whose true class assignment is (c1, c2, c3) is only assigned to class c1 when it is used as a training sample. We refer to this problem as multi-label learning with incomplete class assignment (Figure 2). We proposed a ranking based multi-label learning framework (MLR-GL) that explicitly addresses the challenge of learning from incompletely labeled data by exploiting the group lasso technique to combine the ranking errors.

Automatic feature extraction in latent fingerprints is a challenging problem due to poor quality of most latents, such as unclear ridge structures, overlapped lines and letters, and even overlapped fingerprints (see Fig. 1). In the current practice, latent examiners are required to mark minutiae and optionally extended features such as dots, incipient ridges, pore, etc. The goal of this research is to improve the automatic feature extraction accuracy in the enhanced latent fingerprints as well as to provide the enhanced images to the latent examiners for manual markups. In this research project, we are developing a semi-automatic latent fingerprint enhancement algorithm which only requires manually marked region of interest (ROI) and singular points (e.g. core and delta). Marking these features is generally less time-consuming and requires less expertise than marking minutiae. The proposed algorithm consists of two phases: (i) fingerprint orientation field estimation, which fits an orientation field model to the coarse orientation field estimated from skeleton provided by a commercial fingerprint SDK, and (ii) fingerprint ridge enhancement by Gabor filtering. Experimental results on NIST SD27 (a public domain latent fingerprint database) indicate that, by incorporating the proposed enhancement algorithm, the matching accuracy of the commercial matcher was significantly improved. Fig. 2 shows the examples where the proposed enhancement algorithm improves the retrieval rank of the mated fingerprint in the large background database with 15,758 images.

Latent fingerprint identification is of critical importance to law enforcement agencies in identifying suspects. While tremendous progress has been made in plain and rolled fingerprint matching, latent fingerprint matching continues to be a difficult problem. Latent fingerprints are inadvertent impressions left by fingers on surfaces of objects. Poor quality of ridge impressions, small finger area, and large non-linear distortion are the main difficulties in latent fingerprint matching, compared to plain or rolled fingerprint matching. We propose a system for matching latent fingerprints to rolled fingerprints that is needed in forensics applications. In addition to minutiae, we also use extended features, including singularity, ridge quality map, ridge flow map, ridge wavelength map, and skeleton. In order to evaluate the relative importance of each extended feature, these features are incrementally used in the order of their cost in marking by latent experts.

The evidential value of palmprints in forensic applications is clear, as about 30% of the latents recovered from crime scenes are from palms. While biometric systems for palmprint-based personal authentication in access control type of applications have been developed, they mostly deal with low resolution (about 100 ppi) palmprints and only perform full-to-full palmprint matching. We propose a latent-to-full palmprint matching system that is needed in forensic applications. Our system deals with palmprints captured at 500 ppi (the current standard in forensic applications) or higher resolution and uses minutiae as features to be compatible with the methodology used by latent experts. Latent palmprint matching is a challenging problem because latent prints lifted at crime scenes are of poor image quality, cover only a small area of the palm, and have a complex background. Other difficulties include a large number of minutiae in full prints (about 10 times as many as fingerprints), and the presence of many creases in latents and full prints. A robust algorithm to reliably estimate the local ridge direction and frequency in palmprints is developed. This facilitates the extraction of ridge and minutiae features even in poor quality palmprints. A fixed-length minutia descriptor, MinutiaCode, is utilized to capture distinctive information around each minutia and an alignment-based minutiae matching algorithm is used to match two palmprints.

Examples of forensic sketches (top row) and the corresponding photograph of the later identified subject (bottom row). Images used with permission from [L. Gibson, Forensic Art Essentials. Elsevier, 2008].

It is vital for law enforcement agencies to have the capability to identify unknown criminals using any various forms of available information. Forensic sciences often focus on this goal: given a latent fingerprint, DNA sample, or digital image of a suspect, accurately determine their identity. A profound improvement on this identification task has been observed through the development and progression of biometric technologies. Given (i) the large collection face image databases (e.g. mug shot, driver license, passport), and (ii) many opportunities to acquire face images from both the ubiquity of cameras and generation of forensic sketches, increasing the accuracy of face identification technology is of significant benefit to the criminal justice community.

Forensic sketches are drawn by a police artist based on a verbal description of the appearance of a subject provided by a witness or the victim. Commercial-of-the-shelf (COTS) face recognition systems (FRS) are not designed to match forensic sketches to photograph images, which limits the paradigm for utilizing forensic sketches to dissemination of the composite to media outlets and law enforcement agencies with the goal of someone recognizing the individual depicted in the image. Considering the (i) time spent generating a single forensic sketch, (ii) economic impact of disseminating the sketch to media outlets and law enforcement agencies, and (iii) egregious nature of crimes that are typically committed by culprits depicted in the composites (e.g. murder, sexual assault, armed robbery), progress in the ability to automatically match these sketches to the extensive face databases available is paramount. The goal of this research is to develop matching system specially designed to match forensic sketches to photograph databases.

Matching with the FaceSketchID System

Soft biometric traits embedded in a face (e.g., gender and facial marks) are ancillary information and are not fully distinctive by themselves in face recognition tasks. However, this information can be explicitly combined with face matching score to improve the overall face recognition accuracy. Moreover, in certain application domains, e.g., visual surveillance, where a face image is occluded or is captured in off-frontal pose, soft biometric traits can provide even more valuable information for face matching or retrieval. Facial marks can also be useful to differentiate identical twins whose global facial appearances are very similar. The similarities found from soft biometrics can also be useful as a source of evidence in courts of law because they are more descriptive than the numerical matching scores generated by a traditional face matcher. We propose to utilize demographic information (e.g., gender and ethnicity) and facial marks (e.g., scars, moles, and freckles) for improving face image matching and retrieval performance. An automatic facial mark detection method has been developed that uses 1) the active appearance model for locating primary facial features (e.g., eyes, nose, and mouth), 2) the Laplacian-of-Gaussian blob detection, and 3) morphological operators.

Aging variation poses one of the major problems to automatic face recognition systems. Most of the face recognition studies that have addressed the aging problem have focused on age estimation or aging simulation. However, research on age invariant face recognition is limited. Designing an appropriate feature representation and an effective matching framework for age invariant face recognition remains an open problem. In this paper, we propose a discriminative model to address face matching in the presence of age variation. In this framework, we first represent each face using two patch-based local feature representations, one based on scale invariant feature transform (SIFT) and the other based on multi-scale local binary patterns (MLBP). Since both SIFT-based features and MLBP-based features span a high-dimensional feature space, to reduce the feature dimensionality and avoid the over fitting problem, we use multi-feature discriminant analysis (MFDA) to process these two local feature spaces in a unified framework. The MFDA integrates two different random sampling techniques (random subspace and bagging) to improve the performance of linear discriminant analysis (LDA). By random sampling the training set as well as the feature space, several LDA-based classifiers are constructed and then combined to generate a robust decision via a fusion rule. Experimental results show that our approach outperforms a state-of-the-art commercial face recognition engine on MORPH, a large-scale public domain face aging dataset. We also compare the performance of the proposed discriminative model with a generative aging model. A fusion of discriminative and generative models further improves the face identification accuracy in the presence of aging.

Biometric systems based on a single source of information (unibiometric systems) suffer from limitations such as the lack of uniqueness and non-universality of the chosen biometric trait, noisy data and spoof attacks. In contrast, multibiometric systems fuse information from multiple biometric sources; an optimal combination of information can alleviate some of the limitations of unibiometric systems. Consequently, multibiometric systems achieve better performance compared to unibiometric systems and are being increasingly adopted in a number of applications. Some of the major issues in designing a multibiometric system are (i) determining the sources of biometric information to be fused, (ii) acquisition and processing sequence, (iii) type of information to be fused, (iv) optimal fusion methodology and (v) cost-benefit analysis. The goal of this project is to address these design issues systematically in order to maximize the performance of multibiometric systems.

The increase in twin births has created a requirement for biometric systems to accurately determine the identity of a person who has an identical twin. The discriminability of some of the identical twin biometric traits, such as fingerprints, iris and palmprints, is supported by anatomy and the formation process of the biometric characteristic, which state they are different even in identical twins due to a number of random factors during the gestation period. For the first time, we collected multiple biometric traits (fingerprint, face and iris) of 66 families of twins and we performed unimodal and multimodal matching experiments to assess the ability of biometric systems in distinguishing identical twins. Our experiments show that unimodal finger biometric systems can distinguish two different persons who are not identical twins better than they can distinguish identical twins; this difference is much larger in the face biometric system and it is not significant in the iris biometric system. Multimodal biometric systems that combine different units of the same biometric modality (e.g. multiple fingerprints or left and right irises) show the best performance among all the unimodal and multimodal biometric systems, achieving an almost perfect separation between genuine and impostor distributions.

Semi-supervised learning has attracted a significant amount of attention in pattern recognition and machine learning. Most previous studies have focused on designing special algorithms to effectively exploit the unlabeled data in conjunction with labeled data. Our contribution to semi-supervised learning literature is in two directions.

The first direction is applying semi-supervised learning in unsupervised learning settings, where there is no label information provided along with the data, but a weaker form of side-information is available in the form of pairwise constraints, i.e. information whether a given pair of data points belongs to the same cluster. However, most methods assume that the side information is provided by some oracle, which is not a very realistic setting. We adopt active learning framework as one of the possible ways to obtain constraint information from a user in the most effective, accurate and semi-automatic manner.

Secondly, unlike most of the previous studies on Semi-supervised learning that have focused on designing special algorithms to effectively exploit the unlabeled data in conjunction with labeled data, our goal is to improve the classification accuracy of any given supervised learning algorithm by using the available unlabeled examples. We call this as the Semi-supervised improvement problem, to distinguish the proposed approach from the existing approaches. We design a meta-semi-supervised learning algorithm that wraps around the underlying supervised algorithm, and improves its performance using unlabeled data. This problem is particularly important when we need to train a supervised learning algorithm with a limited number of labeled examples and a multitude of unlabeled examples. We present a boosting framework for semi-supervised learning, termed as SemiBoost. The key advantages of the proposed semi-supervised learning approach are: (a) performance improvement of any supervised learning algorithm with a multitude of unlabeled data, (b) efficient computation by the iterative boosting algorithm, and (c) exploiting both manifold and cluster assumption in training classification models.

Tattoo images on the human body have been routinely collected and used in law enforcement to assist in suspect and victim identification. However, the current practice of matching tattoos is based on keywords. Assigning keywords to individual tattoo images is both tedious and subjective. We have developed a content-based image retrieval system for a tattoo image database. The system automatically extracts image features based on the Scale Invariant Feature Transform (SIFT). Side information, i.e., body location of tattoos and tattoo classes, is utilized to improve the retrieval time and retrieval accuracy. Geometrical constraints are also introduced in SIFT keypoint matching to reduce false retrievals.