A Hybrid Fuzzy Approach for Human Eye Gaze Pattern Recognition.
ABSTRACT Face perception and text reading are two of the most developed visual perceptual skills in humans. Understanding which features
in the respective visual patterns make them differ from each other is very important for us to investigate the correlation
between human’s visual behavior and cognitive processes. We introduce our fuzzy signatures with a Levenberg-Marquardt optimization
method based hybrid approach for recognizing the different eye gaze patterns when a human is viewing faces or text documents.
Our experimental results show the effectiveness of using this method for the real world case. A further comparison with Support
Vector Machines (SVM) also demonstrates that by defining the classification process in a similar way to SVM, our hybrid approach
is able to provide a comparable performance but with a more interpretable form of the learned structure.
Conference Paper: Fuzzy rough signatures.[Show abstract] [Hide abstract]
ABSTRACT: We extend the idea of Fuzzy Signature to Fuzzy Rough Signature (FRS). The proposed Fuzzy Rough Signature is capable of handling most kind of uncertainty: epistemic and random uncertainty, vagueness due to indiscernibility, and linguistic vagueness that exists in both large as well as small sample data sets. Additionally, this system is capable of hierarchical organization of inputs and use of flexible aggregation selection will simplify the combinations of inputs from different sources.FUZZ-IEEE 2009, IEEE International Conference on Fuzzy Systems, Jeju Island, Korea, 20-24 August 2009, Proceedings; 01/2009
- [Show abstract] [Hide abstract]
ABSTRACT: The generalized Weighted Relevance Aggregation Operator (WRAO) is a non-additive aggregation function. The Ordered Weighted Aggregation Operator (OWA) (or its generalized form: Generalized Ordered Weighted Aggregation Operator (GOWA)) is more restricted with the additivity constraint in its weights. In addition, it has an extra weights reordering step making it hard to learn automatically from data. Our intension here is to compare the efficiency (or effectiveness) of learning these two types of aggregation functions from empirical data. We employed two methods to learn WRAO and GOWA: Levenberg–Marquardt (LM) and a Genetic Algorithm (GA) based method. We use UCI (University of California Irvine) benchmark data to compare the aggregation performance of non-additive WRAO and additive GOWA. We found that the non-constrained aggregation function WRAO was learnt well automatically and produced consistent results, while GOWA was learnt less well and quite inconsistently. KeywordsWRAO–OWA–GOWA–The Levenberg–Marquardt–Genetic algorithmsMemetic Computing 3(2):101-110.
Conference Paper: Gaze Pattern and Reading Comprehension.[Show abstract] [Hide abstract]
ABSTRACT: Does the way a person read influence the way they understand information or is it the other way around? In regard to reading of English text, just how much we can learn from a person’s gaze pattern? It is known that while reading, we inadvertently form rational connections between pieces of information we pick up from the text. That reflects in certain disruptions in the norms of reading paradigm and that gives us clues to our interest level in reading activities. In this paper, we validate the above statement and then propose a novel method of detecting the level of engagement in reading based on a person’s gaze-pattern. We organised some experiments in reading tasks of over thirty participants and the experimental outputs are classified with Artificial Neural Networks with an approximately 80 percent accuracy. The design of this approach is simple and computationally feasible enough to be applied in a real-life system. “Your eyes are the windows to your soul”Neural Information Processing. Models and Applications - 17th International Conference, ICONIP 2010, Sydney, Australia, November 22-25, 2010, Proceedings, Part II; 01/2010
A Hybrid Fuzzy Approach for Human Eye Gaze
Dingyun Zhu1, B.Sumudu.U. Mendis1, Tom Gedeon1, Akshay Asthana2and
1Department of Computer Science,
2Department of Information Engineering,
The Australian National University, Acton, Canberra, ACT 0200, Australia.
E-mail: email@example.com, firstname.lastname@example.org,
Abstract. Face perception and text reading are two of the most devel-
oped visual perceptual skills in humans. Understanding which features
in the respective visual patterns make them differ from each other is
very important for us to investigate the correlation between human’s vi-
sual behavior and cognitive processes. We introduce our fuzzy signatures
with Levenberg-Marquardt optimisation method based hybrid approach
for recognising the different eye-gaze patterns when a human is viewing
faces or text documents. Our experiment results show effectiveness of
using this method for the real world case. A further comparison with
Support Vector Machines (SVM) also demonstrates that by defining the
classification process in the similar way to SVM, our hybrid approach is
able to provide a comparable performance but with a more interpretable
form of the learned structure.
Key words: Eye-gaze Pattern, Fuzzy Signatures, WRAO, Levenberg-
Marquardt Optimisation, SVM.
Human-beings’ eyes and their movements are tightly coupled with human cog-
nitive processes, which has been found to be very informative and valuable in
various kinds of research areas. Furthermore, previous research has shown that
human beings’ eye-gaze patterns for observing different objects are also quite
significant for the understanding of cognitive and decision-making processes.
We have been working on developing effective, efficient and robust approaches
to generally provide a clear recognition or unambiguous interpretation of human
eye-gaze patterns in a variety of settings. In , we have successfully shown a
sophisticated use of eye-gaze information for inference of a user’s intention in a
game-like interactive task, which effectively eliminates the need of any physical
control from the human’s side, efficiently improving the communication between
the user and the virtual agents.
In this paper, we introduce our hybrid fuzzy approach: hierarchical fuzzy sig-
nature construction with Levenberg-Marquardt learning of generalised Weighted
Relevance Aggregation Operator (WRAO) for modeling recognition of human
eye-gaze patterns between face scanning and text scanning.
2Hierarchical Fuzzy Signatures
Hierarchical fuzzy signatures are fuzzy descriptors of real world objects. They
represent the objects with the help of sets of available quantities which are
arranged in a hierarchical structure expressing interconnectedness and sets of
non-homogeneous qualitative measures, which are the interdependencies among
the quantities of each set.
The fuzzy signature concept is an effective approach to solve the problem of
rule explosion in traditional fuzzy inference systems: constructing characteristic
fuzzy structures, modeling the complex structure of the data points (bottom up)
in a hierarchical manner [6, 3, 11].
Fuzzy signatures start with a generalized representation of fuzzy sets which
are regarded as Vector Valued Fuzzy Sets (VVFS) . A Fuzzy Signature is a
recursive version of VVFS where each vector can be another VVFS (called a
branch) or a atomic value (called a leaf):
A : X → [ai]k
Generally, fuzzy signatures result in a much reduced order of complexity, at
the cost of slightly more complex aggregation techniques. Unlike conventional
rule based hierarchical fuzzy systems, each branch in a fuzzy signature uses a
different aggregation function to represent the importance of that branch to its
parent, which is a final atomic value called ”degree of match”. Moreover, fuzzy
signatures are different to conventional decision trees as well, they use a bottom
up inference mechanism so that even with missing or noisy input data, this
structure is still able to find a final result.
The fuzzy signature concept has been successfully applied to a number of
applications, such as cooperative robot communication , personnel selection
models , etc. Figure 1 is an example of a fuzzy signature structure which was
constructed for a SARS pre-clinical diagnosis system .
3Levenberg-Marquardt Learning of WRAO for Fuzzy
The Weighted Relevance Aggregation Operator (WRAO)  is derived from the
generalisation of the weights and aggregations in Weighted Relevance Aggre-
gation (WRA), which introduces the weighted relevance of each branch to its
Fig.1. A Fuzzy Signature Exam-
higher branches of the fuzzy signature structure. In this way, WRAO is able
to enhance the accuracy of the results of fuzzy signatures by allowing better
adaptation to the meaning of the decision making process , and it can help
to reduce the number of individual fuzzy signatures by absorbing more patterns
into one structure.
The generalised Weighted Relevance Aggregation Operator (WRAO) of an
arbitrary branch aq...iwith n sub-branches, aq...i1, aq...i2,..., aq...in∈ [0,1], and
weighted relevancies, wq...i1, wq...i2,..., wq...in∈ [0,1] (see Figure 2), for a fuzzy
signature is a function g: [0,1]2n→ [0,1] such that,
The Levenberg-Marquardt (LM) method is not only a major learning algo-
rithm in neural network training functions, but also a widely used advanced
approach that outperforms simple gradient descent and gradient methods for
solving most of the optimisation based problems. This algorithm is a Sum of
Squared Error (SSE) based minimization method that is the function to be min-
imized is of the following special form :
f (s) =1
2? t − s ?
where t stands for the target vector, s for the predicted output vector of the
fuzzy signature, and ?? denotes the 2−norm. Also, it will be assumed that there
are m parameters to be learned and n records in the training data set, such that
n > m. The next update of the LM is the following equation:
u[k] = par[k] − par[k − 1](5)
where the vector par[k] contains all the parameters to be learned, i.e. all
the aggregation factors and weights of WRAO in the equation (3) for the kth
iteration. Then the next update of u[k] is defined as:
?JT[k]J [k] + αI?u[k] = −JT[k]e[k]
where J stands for the Jacobian matrix of the equation (4), I is the identity
matrix of J, and α is a regularisation parameter, which control both search
direction and the megnitude of the next update u[k].
4Eye Gaze Data Collection
An eye-gaze data collecting experiment was conducted. Ten volunteers (Gender:
5 male, 5 female; Occupation: 2 academic staff, 6 postgraduates, 2 undergrad-
uates) from the Australian National University community participated in the
Two sets of human face pictures, 20 in each, were selected for the face scan-
ning experiment. Another 5 text only documents with different lengths (minimal
half page, maximal one page) were also shown. In the experiment, all the face
pictures and documents were demonstrated as full screen scenes on a monitor.
Every participant was firstly asked to view one set of human face pictures
with about 5 seconds on each. The second stage of the experiment was to read
the 5 text documents to determine which were the most important sentences in
each one, no time restriction was imposed for the reading test so the participants
could conduct the reading with their usual speed. In the ranking of sentences
phase, only 1 participant ranked all the sentences, most participants ranked
only 3 sentences so we conclude that our instructions were interpreted as a text
scanning task. After that, the face scanning test was performed again with the
other set of pictures as the last stage.
There was no time break between any two stages, all the eye movement
data was collected by using Seeingmachines eye-tracking system with FaceLAB
software (Version 4.5, 2007) through the entire session of the experiment.
5 Fuzzy Signature Construction for Recognition of
Since people tend to concentrate their gaze fixations onto the interesting and
informative regions in the scene , we further filtered the original collected gaze
points into fixations which offered a much easier and more interpretable form
for the later data process. In addition, instead of considering all the fixations on
each of the test case (either face scanning or document reading), we only use
the first five fixations from every case. The reason for this is that it is possible
to interpret a plausible eye-gaze pattern from the early stage of face viewing
(as early as the first five fixations) . Moreover, the time period for reading a
document was obviously much longer than viewing a face in the data collecting
experiment, so the decision to use only the first five fixations also maintains a
more similar pattern for the future structure construction.
Fig.3. Two Samples of First Five Fixations Only Eye-gaze Patterns
To construct the fuzzy signature structure for learning, it is necessary to
figure out which essential feature in both of the possible patterns can show the
difference for recognition. Figure 3 illustrates the first five fixations for two eye-
gaze patterns from face viewing as well as text scanning respectively. The two
cases are obvious samples and this is actually not the usual source in all the data
records we collected in the experiment.
From these two cases, we can easily find the most obvious difference between
them is in the geometrical shapes, which shows that compared with face scan-
ning, participants’ gaze fixation locations for text scanning follow a very clear
horizontal pattern. On the other hand, although it is still difficult to address a
common gaze pattern for the face scanning, the plausible pattern has a much
more complicated geometrical shape than the simple form from text scanning,
because the informative regions (eyes, nose, mouth and cheeks, etc) in which an
observer is interested in a face are not aligned horizontally as are the sentences
in a document.
Fig.4. Fuzzy Signature Structure for Eye-gaze Pattern
According to the above point, we can further discover that the actually fea-
ture in both of the patterns rests on the vertical difference between two fixations
which are adjacent on time. Consequently, the constructed fuzzy signatures for
the recognition of two patterns can be formed to the structure in Figure 4.
The leaves of each sub-signature in the structure represent the fuzzy value
calculated by using Fuzzy C Mean (FCM) clustering method based on the ver-
tical difference between two adjacent fixations.
6Evaluation and Comparison
According to the constructed fuzzy signature structure for the recognition of
these two different patterns, we performed experiments to learn the weights and
aggregations by applying the Levenberg-Marquardt optimisation method as we
explained in the previous section. The following table shows the results of Mean
Squared Error (MSE) and Classification Error (CLE) for learning the weights of
WRAO for the training and test experiments respectively.
Table 1. Fuzzy Signatures Results for Eye-gaze Pattern Recognition
Experiment Mean Squared Error (MSE) Classification Error (CLE)
We need to clarify that the way we calculate the Classification Error (CLE)
for the fuzzy signature structure is actually based on the degree of difference (d)
between the predicted value and the initial desired value. In our eye-gaze pattern
case, we set three classes according to the degree of difference: Good (|d| ≤ 0.2,
not an error), Bad (0.2 < |d| ≤ 0.5, count 0.5 error) and Very Bad (|d| > 0.5,
count 1 error). So the final classification error rate would be the sum of all the
error numbers divided by the total number of records in the data set.
Table 1 shows that our hybrid fuzzy approach can perform around 80% ac-
curate predictions for the recognition of different eye-gaze patterns between face
and text scanning.
Furthermore, in order to have a performance comparison, the Support Vector
Machines (SVM)  based classifier was chosen to run through the same eye-gaze
fixation data set. Since the classification problem here is only for the recognition
between two different eye-gaze patterns, so we constructed a simple SVM based
classifier using the linear kernel to classify the data of vertical difference between
two adjacent fixations as we used in the previous experiment. For our fuzzy
signature structure, we also reduced the number of classes from previous three
(Good, Bad and Very Bad) to two (Good |d| ≤ 0.5 and Bad |d| > 0.5). Table 2
demonstrates the results of the experiments between fuzzy signatures and SVM.
From above results we can see for this particular pattern recognition problem,
the simple SVM based classifier constructed by using a linear kernel gives highly
Table 2. CLE Comparison Between Fuzzy Signatures and SVM
Experiment FS (3 classes) FS (2 classes) SVM (2 classes)
accurate classification results. Comparatively, by reducing the number of classes
for the fuzzy signatures from previous three (Good, Bad and Very Bad) to two
(Good |d| ≤ 0.5 and Bad |d| > 0.5)., the classification error rate reduces and is
comparable to that of SVM.
Beyond the results comparison, we also find that the way fuzzy signatures
model the classification problem is different from the way SVM models a classifi-
cation problem. SVM approaches the classifiaction problem through the concept
of margin, that is equal to the smallest distance between the decison boundary
and any of the samples . It computes the maximum margin hyperplane that
best defines the decision boundary. The samples that are closest to the decision
boundary lie on this maximum margin hyperplane and are know as support vec-
tors. Any other sample point in the data set plays no role in the classification
problem and is discarded. Once the decision boundary is known, the new data
points can be classified according to which side of the hyperplane it lies. On the
other hand, the construction of fuzzy signature is based on the expression of the
domain knowledge. Further, both the weight and aggregation learning processes
for WRAO offer us a clear view of what exactly produces the results inside the
structure, for instance, which aggregation functions are learned for each branch.
In addition, the value generated after every aggregation actually represents the
degree of match as the importance of the current branch to its parent, which is
also useful to discover which sub-branch makes the most contribution and which
are not significant factors from the domain for the problem modeling. So fuzzy
signatures provide a more interpretable expression of how well the output of the
structure approaches the target value or classification.
A fuzzy signature with a Levenberg-Marquardt learning based hybrid approach
has been introduced for modeling a real world study: recognising human eye-gaze
patterns to distinguish between face scanning and text scanning. The constructed
structure shows significant performance for the recognition in the experiment
From the further discussion of performance comparison with linear SVM, we
suggest that our structure is capable of producing a comparable result for the
classification, but as an effective approach, it can provide a more interpretable
representation signature pattern of a real world object from its construction as
well as the learning process.
 Althoff, R.R. and Cohen, N.J.: Eye-movement Based Memory Effect: A Reprocess-
ing Effect in Face Perception. In: Journal of Experimental Psychology: Learning,
Memory, and Cognition, Vol.25, pp.997-1010 (1999)
 Bishop, C.M.: Pattern Recognition and Machine Learning. Springer. (2006)
 Gedeon, T.D., Koczy, L.T., Wong, K.W. and Liu, P.: Effective Fuzzy Systems for
Complex Structured Data. In: Proceedings of IASTED International Conference
Control and Applications (CA’01), Banff, Canada, pp.184-187 (2001)
 Gedeon, T.D., Zhu, D. and Mendis, B.S.U.: Eye Gaze Assistance for a Game-
like Interactive Task. In: International Journal of Computer Games Technology,
Vol.2008, Article ID 623752, 10 pages (2008)
 Hsu, C., Chang, C. and Lin, C.: A Practical Guide to Support Vector Classification.
Technical Report, Department of Computer Science, National Taiwan University.
 Koczy, L.T., Vamos, T. and Biro, G.: Fuzzy Signatures. In: Proceedings of the
4th Meeting of the Euro Working Group on Fuzzy Sets and the 2nd International
Conference on Soft and Intelligent Computing (EUROPUSE-SIC’99), Budapest,
Hungary, pp. 210-217 (1999)
 Marquardt, D.: An Algorithm for Least Squares Estimation of Nonlinear Param-
eters. In: Journal of the Society for Industrial and Applied Mathematics, Vol.11,
No.2, pp.431-441 (1963)
 Mendis, B.S.U. and Gedeon, T.D.: A Comparison: Fuzzy Signatures and Choquet
Integral. In: Proceedings of IEEE World Congress on Computational Intelligence
(WCCI’08), Hong Kong, China, pp.1464-1471 (2008)
 Mendis, B.S.U., Gedeon, T.D., Botzheim, J. and Koczy, L.T.: Generalised Weighted
Relevance Aggregation Operators for Hierarchical Fuzzy Signatures. In: Proceed-
ings of the International Conference on Computational Intelligence for Mod-
elling, Control and Automation (CIMCA’06), Jointly with the International Con-
ference on Intelligent Agents, Web Technologies and International Commerce
(IAWTIC’06), Sydney, Australia, pp.198-203 (2006)
 Mendis, B.S.U, Gedeon, T.D. and Koczy, L.T.: Learning Generalized Weighted
Relevance Aggregation Operators Using Levenberg-Marquardt Method. In: Pro-
ceedings of the 6th International Conference on Hybrid Intelligent System (HIS’
06) and 4th Conference on Neuro-Computing and Evolving Intelligence (NCEI’
06), New Zealand, pp.34-39 (2006)
 Vamos, T., Koczy, L.T. and Biro, G.: Fuzzy Signatures in Data Mining. In: Pro-
ceedings of Joint 9th IFSA World Congress and 20th NAFIPS International Con-
ference, Vancouver, Canada, Vol.5, pp.2841-2846 (2001)
 Wong, K.W., Gedeon, T.D. and Koczy, L.T.: Construction of Fuzzy Signature
from Data: An Example of SARS Pre-clinical Diagnosis System. In: Processings of
IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2004), Budapest,
Hungary, pp.1649-1654 (2004)
 Yarbus, A.: Eye Movements and Vision. New York: Plenum Press. (1967)
 Zhu, D. and Gedeon, T.: Fuzzy Logic for Cooperative Robot Communication.
In: Proceedings of 2nd KES International Symposium on Agents and Multi-agent
Systems - Technologies and Applications (KES-AMSTA-2008), Incheon, Korea,
Lecture Notes in Computer Science, Vol.4953, pp.401-410 (2008)