Extraction of Signatures from Document Images for Real World Applications

1 Introduction

Today, automatic systems are used in almost every field to facilitate different processes in our daily life. These uses range from simple vending machines through to sophisticated systems for automatically diverting postal mail. Research in the field of document image analysis has enabled the development of various automatic systems to analyze documents on the fly and extract different types of information for different purposes, e.g., optical character recognition for sorting of postal mails based on zip code [1], automatic extraction of names, addresses, numbers and dates from documents [2]. Along with the extraction of different types of information, researchers are aiming to develop automatic systems to verify the authenticity of questioned documents [3, 4, 5].

Signatures are a widely used authentication mechanism in many industries such as banking and law [6]. In the last three decades researchers have developed various offline (using only spatial information, such as scanned signature images) and online (using both spatial and temporal/dynamic information) signature verification systems. However, in most of the existing systems, it is assumed that signatures are readily available (segmented/extracted) from the document that originally contained them. In addition, existing publicly available datasets for the development and evaluation of signature verification systems also contain only signatures and not the complete documents containing signatures and/or other information. Contrary to this, in reality, signatures are contained on documents which also contain other background information. Such documents include bank checks, invoices, contracts, credit card pay slips and wills. In these real world scenarios, due to the lack of segmentation, the existing signature verification and identification systems cannot be used as is. This paper focuses on the challenges faced by researchers when developing a complete automatic document analysis system that is capable of performing signature segmentation/extraction from documents and then performing signature verification.

The remainder of this paper is organized into the following six sections labelled 2 through 7. Section 2 provides an overview of the systems currently used for segmentation of signatures from document images. Section 3 provides details on the proposed alternative method for signature segmentation. Section 4 presents details about the dataset used for evaluation of the segmentation method. Section 5 reports on the evaluation protocols and results of the proposed signature segmentation method. Section 6 discussed areas requiring further research and Section 7 concludes this paper.

2 Existing Segmentation Systems

To date, there has been very little research in the area of segmentation and extraction of signatures from documents, especially from document images. However, some research has been undertaken that focused on the segmentation of handwritten text from machine-printed text. In this section we will provide an overview of both cases, i.e., the existing systems available for segmentation of handwriting from machine-printed text, and those used for the segmentation of signatures from documents and bank checks.

2.3 Signature Segmentation from Document Images

There are many documents apart from bank checks that also contain signatures, e.g., contracts, invoices, and wills. Figure 2 shows that segmentation in such documents becomes more challenging as signatures can be located in different areas on the documents depending upon the document’s content. To deal with these situations, Zhu et al. [25] proposed a method for segmenting signatures from a complete document using a saliency map. In addition to a signature segmentation method, Zhu et al. [25] also introduced a publicly available dataset called Tobacco-800 which consists of complex document images containing information about signatures on printed text documents. Along with other information, Tobacco-800 contains patch level information (about which blocks contain signatures, without specific marking of signature pixels/strokes so that signatures overlapping with machine-printed text are not fine separated) for 900 signatures on complete documents. A complete document retrieval system is presented by Zhu et al. [26] where signature matching is combined with the detection framework discussed in the Zhu et al. 2007 [25] paper. Mandal et al. [27] proposed an approach for signature segmentation using conditional random fields. Results are reported on a subset of the Tobacco-800 dataset, i.e., 105 images out of the total 1290 documents contained in the dataset. One of the main problems with this approach is that it requires a large number of training samples. Furthermore, Mandal et al. [27] segments signatures on patch level, whereas in some cases some machine-printed text is touching the signature components which may cause problems in signature verification. To rectify this problem, Mandal et al. [28] recently proposed an alternative approach for segmentation of signatures by also segmenting those characters which are touching the signature strokes. Estaban et al. [29] detected the position of signatures in document images using the accumulative evidence technique. This method works by searching for regions that maximize the measure of correspondence (i.e., the slope of the strokes in the signature) with a given reference signature. The main problem with this method is that it requires one signature sample from each author. This limits its applicability in real world scenarios where it is not always possible to have signatures from only the already known persons, e.g., signatures on invoices received from an international supplier.

Figure 1.

Figure 2.

A major limitation with the majority of the above mentioned approaches of signature segmentation is that they have not been applied on real life problems. Furthermore, none of them has been tested on complete datasets such as the Tobacco-800 or any other publicly available dataset. Subsets of Tobacco-800 dataset have been used in different studies but it is not mentioned which images are included in any particular subset. This makes it difficult to analyze and compare the performance of these methods. Also for the methods tested, none of the researchers reported the efficiency of their systems in terms of time and complexity.

Furthermore, some commercial systems are available. These systems are capable of finding one or two signatures in bank checks as well as (IRD⁵) images and snippets, and applying signature verification on these segmented signatures, e.g., SignatureXpert-2⁶ by Parascript. The problem with commercial systems is that the data on which they are trained are never made publicly available. Also, details about how these systems work and the algorithms applied is proprietary information.

3 Dataset

Currently, to the best of the authors’ knowledge, there are two publicly available datasets that contain information about signature patches/zones. These are the Tobbaco-800 dataset [30] and the Maryland Arabic dataset [31]. The Tobbaco-800 dataset contains 1290 images containing the handwritten and machineprinted text in English as well as machine-printed logos. There are 900 labeled signatures in this dataset. The Maryland Arabic dataset contains 169 images containing handwritten English and Arabic text along with 149 labeled signatures.

To generate results comparable to the other approaches, such as those undertaken by Zhu et al. [30], we performed evaluations of our proposed system on the Tobbaco-800 dataset. This dataset contains only the ground truth information about the logos and signatures contained in a document on the patch level (i.e., which block in the image contains the signature and which block the logos). However, if the signature is overlapping printed text in that patch, only general information about the patch is supplied. As a result, it is not possible to evaluate the individual pixels in the image to determine which pixels are a part of signature and which are part of the printed text. As mentioned in Section 1, the document analysis community has only recently started considering the problem of signature segmentation. As a result, the datasets currently used only have the patch level ground truth information about signatures available. For these datasets to be useful to the current research being undertaken, the stroke/pixel level information is also required.

To compare our method with the method proposed by Mandal et al. [27], we have used a subset of images from the Tobbaco-800 dataset containing only machine-printed text and signatures.

4 Proposed Method for Signature Segmentation

This section provides an insight into the authors’ proposed method for signature segmentation from document images. This method is based on local features, i.e., Speeded Up Robust Features (SURF⁷). SURF is a part based approach that represents the image as a set of key points. As part based approaches extract key points/points of interest from the parts of image (which represent local features), it brings robustness against different variations in the image [34, 35].

For each of the SURF key points, a 128 bit descriptor is extracted which represents that particular key point. This descriptor is used to find similarities between different parts of the image. For extraction of SURF, we used a Hessian threshold of 400, i.e., all the key points having a Hessian threshold of less than 400 were ignored. This filtering removes unimportant features from the images.

For training purposes⁸, we used ten documents from the Tobaaco-800 dataset containing machine-printed text and signatures. To ensure the performance of the system, it is required that all the machine-printed text is separated from the signatures. As the Tobbaco-800 dataset does not include ground truth information for machineprinted text, the authors manually generated two new images for each document dividing it into the printed text and signatures image components. These generated images were then used for training the program. Pixels containing overlaid areas were extracted for both the printed text as well as signature image of the training set.

For all of the printed text components, the extracted printed text key points and their respective descriptors were added to a printed text features database. Similarly, for all of the signature components, the extracted signature key points and their respective descriptors were added to a signature features database. These two databases serve as the reference for matching the signature and printed-text features during testing. Figure 4 shows the training procedure used.

Figure 3.

Figure 4.

To segment a signature from a document containing both signatures as well as printed text, the connected components⁹ are extracted (as shown in Figure 3). Here, the SURF features are extracted for each of the connected component (full signature component, full printed text component, and the overlapping signature and printed text components). The descriptor of the key points are then compared with the descriptors of printed text keypoints and signature keypoints from the two reference databases. The Euclidean distance metric is used as a distance measure.

Finally for the classification of connected components, a majority voting approach is applied. If a connected component’s keypoint has less Euclidean distance to the signature keypoints reference database as compared to the printed text keypoints reference database, one vote is added to the signatures class and vice versa. The process is repeated until all of the connected components are assigned to one of the two classes (See Figure 4). Once all of the connected components are marked as printed text or signature, separate image for signature is generated. To segment the signature from the test document, the original image is cloned and bounding boxes of all connected components of printed text are filled with white color on that image, which in turn results in a segmented signature image.

As a post processing step, horizontal run length smearing is performed on the segmented signature image. Applying smearing merges all of the neighboring components. Connected components are extracted from smeared images and all of the small connected components are neglected. The remaining components are considered as signature patches. Figure 5 shows the extracted signatures from the document shown in Figure 3. One of the main advantages of our approach is that it requires a very limited number of training samples.

Figure 5.

5 Evaluation

To evaluate the performance of the authors’ proposed method the precision and recall measures are used. As mentioned in Section 3, the ground truth of the available dataset contains only patch level information of the signatures. Therefore, the authors also calculated the precision and recall on the patch level information for the images used. The signature is considered detected if there is at least a 40% of overlap between the ground truth and the detected signature patch.

The evaluation results of the proposed method are presented in Table 1. This method has a recall rate of 100%, which means that all the signatures were extracted successfully. A minor drawback of this method is, however, that the precision is currently quite low. One reason for this is that the images containing logos were also tested. The proposed method sometimes incorrectly marked logos as signature patches during the segmentation process. The authors however consider this drawback to be removable by adding the class “logo”.

Figure 7 shows some of the segmentation results of the proposed method. Qualitatively, the correctly segmented signatures are comparable to manually cropped signatures. Figure 7 also shows some examples of false positives.

Figure 6.

Figure 7.

As can be seen, the proposed method preforms quite well on a difficult subset. More than every second extracted patch is a signature, with all the signatures contained in the images successfully extracted. This outcome is very positive suggesting it will be a useful technique for document examiners.

6 Scope for Future Research

In order to automate the complete document analysis process, various methods need to be developed to extract different types of information from documents in a format usable for different applications. Likewise, reliable automatic segmentation methods that can be integrated with signature verification systems need to be developed to make it a viable resource for document examiners. To develop such methods, benchmark datasets on which signature segmentation systems can be evaluated in terms of recall (how many signatures/items are extracted from the document), precision (how many of the extracted items are actually signatures) and efficiency (both in terms of speed and complexity), need to be compiled. As discussed in Section 2, currently most of the existing signature segmentation systems are evaluated on subsets of Tobacco-800 dataset and (to the best of authors’ knowledge) there is no publicly available dataset specifically designed for signature segmentation. As discussed, the major disadvantage with the Tobacco-800 dataset is that it only contains patch level information about signatures, i.e., which block contains primarily signatures (meaning that some other information may also be there). This non-availability of datasets shows the current lack of research interest in signature segmentation. Since 2011, some researchers have considered this problem but still there is a lot more that needs to be done in this area.

The authors are currently working on developing a large dataset for signature segmentation and verification. Along with patch level information, this dataset will have also signature stroke information (i.e., information about each pixel that belongs to signatures or printed text) and would be usable for testing complete signature segmentation and verification frameworks for analysis of documents containing signatures. Furthermore, in this dataset multi-spectral information about signatures and other parts of documents will also be incorporated.

In addition to developing datasets specific to signature segmentation, another important area requiring research is in the development of a program capable of performing layout-free segmentation of signatures. As previously discussed, signatures are not always located in the same place on a document (as shown in Figure 2). This means that an automatic system needs to be capable of finding signatures without using any prior information about the layout of a document and probable location of a signature. Some efforts to achieve this have already been made by Mandal et al. [27, 28] and Ahmed et al. [38], but there is still a lot more to be done in terms of quality of extraction, precision, recall, and efficiency.

Another important aspect is to tune signature verification systems in such a way so that they are capable of distinguishing between genuine and forged signatures, even in the presence of some noise in signatures, e.g., overlapping characters or missing parts of signature. The existing signature verification systems assume that the questioned signature image contains no other information than the signature itself, which is not always the case (see Figure 2 for reference).

Figure 8 (b) shows a very common scenario where most of the existing signature verification systems will misclassify this signature as a forgery simply due to the presence of text in the signature image. This extraneous text is considered as a part of signature during verification by current automatic systems. Therefore, further development is required to tune the existing signature verification systems so as to make them more robust to noise and touching components. A probable solution could be to use only parts of the signature for verification rather than using the complete signatures [39, 40]. This, however, will raise further questions about which parts of signatures are to be used and why. In addition, a possibility is to use hyper-spectral imaging technology for distinguishing the signatures from background noise. Hyper-spectral imaging provides wavelength information about the underlying text/signatures represented from visible to near infrared regions. This information may allow the program to very accurately segment the signature from its surrounding, even at the pixel level [41].

Figure 8.

7 Conclusions and Future Work

In this paper, state-of-the-art automatic signature segmentation/extraction methods with the potential to be integrated with verification to perform authentication were discussed. The paper outlined the limitations of the currently available automatic signature segmentation systems in real world scenarios. The main drawback of the current systems is that documents often contain information other than just the signatures, e.g., background text, lines, and logos. In order to authenticate these documents by performing signature verification, the signature must firstly be segmented and extracted from the document. The paper assesses the various approaches that have been proposed for signature segmentation from document images and outlines their limitations. The paper also notes the current lack of databases suitable for the development and testing of complete document authentication systems involving signature segmentation and verification.

Furthermore, the authors have presented a part-based method for the extraction of signatures from documents based on the SURF key points method. The proposed method requires very limited training as compared to other contemporary approaches discussed in this paper. The experiments were performed on the Tobbaco-800 dataset where all of the available signatures were successfully extracted. However, some false positives were also detected mostly due to presence of other class information in the images such as logos. This limitation can be overcome by extending the method to identify other classes and to use advanced classifiers such as SVM. By incorporating these modifications into the system, the precision of the method will be vastly increased. Moreover, using the approach of removing the less distinctive features from all the classes (proposed by Ahmed et al. [42]) will hopefully further improve the results.

It is anticipated that the authors’ dataset containing signature patch and stroke level information will be completed in the near future. Once completed, it will be made publicly available. Emphasis will be given to incorporating examples of signatures overlapping text and graphics in the dataset to make it more reflective of real world situations. The patch information will be recorded in both the visible and near infrared spectra as recommended by Khan et al. [41]. Further, testing the methods proposed by Ahmed et al. [42] for separation of overlapping text/graphics to finely extract the signature strokes is also planned.