Predicting the accuracy of multiple sequence
alignment algorithms by using computational
Francisco M. Ortun ˜o1,*, Olga Valenzuela2, Hector Pomares1, Fernando Rojas1,
Javier P. Florido3, Jose M. Urquiza4and Ignacio Rojas1
1Department of Computer Architecture and Computer Technology,2Department of Applied Mathematics,
University of Granada (UGR), 18071 Granada,3Medical Genome Project, Andalusian Human Genome
Sequencing Centre (CASEGH), 41092 Seville and4Chromatin and Disease Group, Bellvitge Biomedical
Research Institute (IDIBELL), L’Hospitalet, Barcelona 08907, Spain
Received April 16, 2012; Accepted September 11, 2012
Multiple sequence alignments (MSAs) have become
one of the most studied approaches in bioinfor-
matics to perform other outstanding tasks such as
structure prediction, biological function analysis or
next-generation sequencing. However, current MSA
algorithms do not always provide consistent solu-
tions, since alignments become increasingly difficult
when dealing with low similarity sequences. As
widely known, these algorithms directly depend on
specific features of the sequences, causing relevant
influence on the alignment accuracy. Many MSA
tools have been recently designed but it is not
possible to know in advance which one is the most
suitable for a particular set of sequences. In this
work, we analyze some of the most used algorithms
presented in the bibliography and their dependences
on several features. A novel intelligent algorithm
based on least square support vector machine is
then developed to predict how accurate each align-
ment could be, depending on its analyzed features.
This algorithm is performed with a dataset of 2180
MSAs. The proposed system first estimates the
promising methodologies are then selected in order
to align each set of sequences. Since only one
selected algorithm is run, the computational time is
not excessively increased.
Multiple sequence alignment (MSA) is a widely used
approach in the current molecular biology. This technique
involves the comparison of new sequences with well-
known ones, extracting their shared information and
their significant differences (1). MSA methods have trad-
itionally been essential for analyzing biological sequences
and designing applications in structural modeling, func-
tional prediction, phylogenetic analysis and sequence
database searching (2). Current MSA tools are also
applied to comparisons of protein structures (3), recon-
structions of phylogenetic trees (4) or predictions of mu-
tations (5) and interactions (6).
More recently, the interest of MSA methodologies
techniques. Current technologies provide a large amount
of data that must be analyzed, processed and assessed.
required to extract biological meanings from such infor-
mation. Thus, supervised learning algorithms have been
widely implemented in the analysis of genomic and prote-
omic experimental data. Additionally, recent experimental
methods also retrieve further biological data, which is
useful for extending the information included within align-
ment methods. Thus, current MSAs tools take advantage
of heterogeneous features, which are provided by recent
biological progress in functional, structural and genomic
researches, to obtain more accurate alignments within a
reasonable time (7). Therefore, MSAs are becoming one of
the more powerful and essential procedures of analysis (8).
Traditionally, alignment strategies are mainly incor-
porated in progressive algorithm and consistency-based
methods (7). Progressive algorithms assemble previously
built pairwise alignments through a clustering method and
store their evaluations in a library. Some well-known
programs using progressive strategies are ClustalW (9)
or Muscle (10). On the other hand, consistency-based
methodologies, e.g. T-Coffee (11) or MSAProbs (12),
*To whom correspondence should be addressed. Tel: +34 958 241778; Fax: +34 958 248993; Email: firstname.lastname@example.org
Published online 11 October 2012 Nucleic Acids Research, 2013, Vol. 41, No. 1e26
? The Author(s) 2012. Published by Oxford University Press.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by-nc/3.0/), which
permits non-commercial reuse, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact
consideration not only the previous pairwise alignments
but also if these alignments are consistent with the final
result. However, neither progressive nor consistency-based
methods build optimal alignments when sequences are dis-
tantly related.More recent
3D-COFFEE (13) or Promals (14), include further infor-
mation (structure, domains or homologies) in addition to
the provided sequences. Such features are usually found
by experimental resources in databases such as Protein
Data Bank (PDB) (15), Uniprot (16) or Pfam (17).
Nevertheless, the consumed time is still excessive for
these strategies and improvements are only relevant
when sequences are evolutionarily less related (7).
methodologies based on different strategies. Moreover,
each MSA tool usually depends on particular features;
thereby, there is no consensus about which one produces
more accurate alignments (18,19). A new intelligent algo-
rithm based on least square support vector machine
(LS-SVM) is proposed here in order to predict how accur-
ately each MSA tool will align a set of sequences.
Interesting features related to the sequences and their
products have been added from several resources in
order to make this prediction. To the best of our know-
ledge, there are no similar studies in the current bibliog-
raphy, which address the prediction of the alignment
accuracy. This algorithm also estimates which methodo-
logies are more significant to align those sequences. The
system has been created from 218 sets of sequences
provided by the BAliBASE benchmark (20) and their cor-
responding features. Since our algorithm applies a priori
features to predict the accuracy, only the best method is
run. Consequently, the CPU cost is not excessively
MATERIALS AND METHODS
In this work, a novel system called ‘Prediction of Accuracy
in Alignments based on Computational Intelligence’
(PAcAlCI) is developed. PAcAlCI is composed by four
independent modules (Figure 1). First, 218 groups of
sequences are aligned through 10 different methodologies,
producing a dataset of 2180 alignments (‘Input Dataset’
module). Alignments are then evaluated in order to
measure their accuracies. From these groups of sequences,
several features are also retrieved from various relevant
databases (‘Feature Extraction’ module). The most
useful features are progressively included in a subset
which is used by the subsequent algorithm (‘Feature
Selection’ module). Finally, selected features are added
to an LS-SVM model to predict alignment accuracies
and, subsequently, the most suitable methodologies
(‘LS-SVM Prediction’ module). The PAcAlCI system
was completely implemented with Matlab (Version
R2010b). The source code is available at http://www.ugr
A set of sequences must be considered in order to compare
different alignment algorithms and develop the proposed
prediction. Several datasets and techniques have usually
been developed to standardize the comparison of align-
ment results, e.g. Oxbench (21), HOMSTRAD (22),
Prefab (10) or BAliBASE (20). In this work, the
BAliBASE benchmark (v3.0) was chosen.
BAliBASE defines several groups of sequences that can
be easily aligned by standard algorithms. This dataset
includes a total of 218 sets of sequences that were
manually extracted from different databases, specifically
the PDB (15). This benchmark also provides a set of
handmade reference alignments (gold standard) in order
to compare them with the alignments obtained by other
tools. Thus, BAliBASE calculates a Sum-of-Pairs (SP)
score to evaluate such alignments. These SP scores are
used by our system to measure the quality of each
Sequences in BAliBASE are classified in the next
subsets: (i) equidistant PDB sequences with <35 insertions
and sharing <20% of identity between any pair of
sequences (RV11 and RV12 subsets); (ii) PDB orphan
sequences of families with >40% of identity and at least
one known 3D structure (RV20 subset); (iii) subfamilies of
sequences that share >40% of identity but <20% with
other subfamilies (RV30 subset); (iv) sequences with
>20% of identity and large terminal extensions (RV40
subset) and (v) sequences with >20% of identity and
internal insertions (RV50 subset). From these subsets, it
will be possible to establish, for example, which method-
ology is better when less related sequences are aligned or
what differences are found when the methodologies
Input featuresOutput Accuracies
Subset of relevant features
Figure 1. PAcAlCI scheme. The architecture is developed into four
modules: input dataset, feature extraction, feature selection and
e26 Nucleic Acids Research, 2013,Vol. 41,No. 1PAGE 2 OF 10
include additional information. These questions will be
solved in the ‘Comparison of MSA Methodologies’
Ten of the most relevant MSA tools are selected to be
included in PAcAlCI. These tools are classified according
to their implemented strategy: progressive techniques,
consistency-based methods or algorithms including add-
itional information (see summary in Table 1). Programs
were run with their default features. Among the progres-
sive methods, ClustalW (9), Muscle (10), Kalign (23),
Mafft (24) and RetAlign (25) were chosen. ClustalW
designs a tree-computing algorithm to find the alignment
by means of distance scores and a gap weighting scheme.
Muscle develops a strategy based on three stages, where a
quickly built alignment is refined with an iterative method
and a tree-dependent partitioning approach. Kalign uses
improve the distance calculation of the classical progres-
sive approach. Mafft identifies common homologies in se-
quences through a fast Fourier transform, significantly
reducing the computational cost. Lastly, RetAlign imple-
ments a progressive corner-cutting algorithm to identify
optimal alignments in a network of possible alignments.
included in PAcAlCI: T-Coffee (11), ProbCons (27) and
Fast Statistical Alignment (FSA) (28). T-Coffee develops
a standard consistency algorithm, building pairwise align-
ments and evaluating them against third sequences.
ProbCons defines a probabilistic consistency based on a
pair of hidden Markov models (pair-HMMs) to perform
a novel scoring scheme for the standard consistency
library. FSA estimates the insertion and deletion processes
in sequences through pair-HMMs to combine their
probabilities into alignments.
Finally, two more complex methodologies, namely
3D-Coffee (13) and Promals (14) were also applied.
3D-Coffee introduces structural information in the
standard T-Coffee evaluations from the PDB (15), per-
forming comparisons between each two structures and
each sequence with its structure. On the other hand,
combining sequences and homologies in profiles through
Databases and feature extraction
Features of BAliBASE sequences are extracted from
well-known biological databases. Such databases are con-
sulted to obtain interesting data which complement the
sequences and to build a complete set of features. The
final dataset will be composed by 23 features (see summary
in Table 2).
Some features related to sequences, domains, amino
acid types or structures have already been successfully
(18,29). However, the set of features is complemented
with further information based on other studies such as
protein interaction prediction (30) or protein model clas-
sification (31). Therefore, a more complete feature envir-
onment is presented in this work in order to study its
relevance to sequence alignments. Here below, each con-
sulted database is described, indicating which features
have been retrieved and their nomenclature in the
. BAliBASE (20) can be considered the first consulted
database, as it provides the sequences that are aligned.
Then, the featuresassociated
sequences are the number of sequences (f1), the
average length of sequences (f2) and the normalized
Table 2. Summary of features extracted from several databases
FeatureSource RangeType Rank
Number of sequences BAliBASE
AA in ?-helixa
AA in ?-stranda
f10 GO termsb
f11 MF-GO termsb
f12 CC-GO termsb
f13 BP-GO termsb
f14 Shared GO termsb
f16 Seq. with any
f18 Polar AAa
f19 Non-polar AAa
f20 Basic AAa
f21 Aromatic AAa
f22 Acid AAa
f23 MSA method
[66.13, 1630.11] Real
PDB[0.00, 0.75]Real 23
Biochemistry [0, 1]
Biochemistry [0, 1]
Biochemistry [0, 1]
Biochemistry [0, 1]
Biochemistry [0, 1]
2 [1, 10]
Twenty-three features were retrieved from different databases. The rele-
vance ranking was also measured according to the mRMR procedure.
aThese features are calculated as the percentage of amino acids
(AA) with that specific feature.
number of occurrences per sequence.
bThese features are calculated as the
Table 1. Summary of applied methodologies
Ten different methodologies were run to align multiple sequences.
Their versions and the applied strategies are also shown.
PAGE 3 OF 10Nucleic AcidsResearch, 2013, Vol.41,No. 1 e26
variance of the sequence length (f3). This information
determines whether there is any dependence between
alignment tools and the number/length of sequences.
Since BAliBASE classifies sequences according to
certain features (see the ‘Input Dataset’ section for
details), the subset, where each set of sequences is
included, is proposed as another feature (f4).
. Uniprot (16) consists of a wide repository of proteins
with accurate, consistent and rich annotation. Several
features are calculated from this database: the percent-
age of amino acids in ?-helix structures (f5), the per-
centage of amino acids in ?-strand structures (f6) and
the percentage of amino acids in the transmembrane
region (f7). Data associated with similar secondary
structures or locations usually indicate more related
sequences or regions.
. Pfam (17) identifies common functional regions in
families, also called domains. Domain features are per-
formed from this database as: average number of
domains per sequence (f8) and average number of
shared domains (between each pair of sequences) per
regions with related functionality. This functionality
can be useful to understand how some sequences
must be efficiently aligned or how close sequences
are in their families.
. The Gene Ontology Annotation
controlled vocabularies for the annotations of molecu-
lar attributes in different model organisms. These
ontologies organized as a directed acyclic graph
(DAG): molecular function (MF), cellular component
(CC) and biological process (BP). Features used in this
work from Gene Ontology Annotation (GOA) are
average number of annotated terms per sequence
(f10); average number of annotated terms for each
ontology per sequence: MF (f11), CC (f12) and BP
(f13) and average number of shared GO terms
(between each pair of sequences) per sequence (f14).
. PDB (15) includes information about experimentally
average number of annotated PDB structures per
sequence (f15), the percentage of sequences with struc-
tures (f16) and the average number of shared structures
(between each pair of sequences) per sequence (f17) are
proposed from this database.
into three structured
of eachprotein. The
Apart from these databases, other resources have been
applied in order to complete the set of features. For
instance, the classification of amino acids included in
(33) has been applied to define: the percentage of polar
uncharged amino acids [G,A,P,V,L,I,M] (f18), the percent-
age of non-polar aliphatic amino acids [S,T,C,N,Q] (f19),
the percentage of basic positively charged amino acids
[K,R,H] (f20), the percentage of aromatic amino acids
[F,W,Y] (f21) and the percentage of negatively charged
amino acids [D,E] (f22).
Finally, the MSA method being executed (see the
‘MSA methodologies’ section) is the last included
proposed system, as the purpose is to predict the
isdeterminant in the
features. Besides, the most suitable methods according
to the predicted accuracies are selected from that
feature. Also, the accuracies of each method are
included as outputs. As explained before, this accuracy
is called SP score by BAliBASE and it is defined as a
similarity value against the gold-standard references.
of eachmethod according toall these
Feature selection based on mutual information
The relevance of the previously proposed features is
analyzed through a feature selection procedure. Feature
selection algorithms allow reducing the number of
features, filtering out those irrelevant or redundant. One
of the well-known feature selection, called minimal-redun-
dancy-maximal-relevance (mRMR) (34), is applied in this
work. First, this approach calculates the relevance of the
features by using their mutual information. The obtained
relevance is then assessed through the subsequent
machine-learning procedure. The aim of mRMR is to
select a feature at a time with a first-order incre-
mental search, trying to avoid redundant features
(see Supplementary mRMR feature selection for details).
Discrete and continuous random variables are both con-
sidered in the mRMR algorithm. Such property is essen-
tial in the proposed set of features, since both types of
variables were included (‘real’ and ‘integer’ types in
Table 2). Besides, the output accuracy is also defined as
a continuous variable.
The mRMR method achieves a great accuracy in a
reduced time. Thus, the algorithm is useful to accurately
select features among a huge number of them. The
proposed features arethen
Subsequently, the LS-SVM model is trained and evaluated
progressively increasing the number of features from this
Least squares support vector machine
Features selected before are included in an LS-SVM
model (35) in order to estimate different alignment
accuracies. Subsequently, the algorithm also determines
which tools are more likely to obtain the best alignment
in term of accuracy. LS-SVMs models were generally
designed for prediction approaches, but they present
the most effective performance for regression problems.
Since our system includes continuous values of accuracy,
the proposed prediction is defined as a regression
problem. Additionally, as the order of input data in
LS-SVM is arbitrary, any change of the order would
not affect the modeling result (35,36). Therefore, the ap-
plication of LS-SVM would be an effective and faithful
A kernel model must also be selected to correctly design
the prediction method based on LS-SVM. The radial basis
function (RBF) kernel was chosen to be applied in the
proposed methodology. Additionally, LS-SVMs based
on RBF kernels must also be performed by two kinds of
hyper-parameters: the regulation parameter and the kernel
parameters. These hyper-parameters were optimized by
cross-validation in the proposed LS-SVM system (details
about kernels and hyper-parameters in LS-SVMs are
e26Nucleic Acids Research, 2013,Vol. 41,No. 1PAGE 4 OF 10
provided in the Supplementary LS-SVM models). The
LS-SVM algorithm is developed here from the Matlab
toolbox found in (37).
In order to assess the LS-SVM model, a 10-fold
cross-validation procedure is performed. This procedure
randomly divides the complete dataset (2180 problems)
into 10 subsets of 218 problems. Nine subsets are then
applied to train the proposed system. The training pro-
cedure includes the most relevant features and the pos-
terior accuracy for each problem in the subset. Thus,
hyper-parameters are tuned and the LS-SVM model is
estimated. Subsequently, the last subset is used to test
the estimated LS-SVM model. The accuracies from
such subset are then predicted and compared with
those already known. The training and test procedures
are repeated 10 times with the 10 different subsets. The
predicted accuracies are then validated by their errors
against real ones. The prediction error is measured by
means of the ‘mean relative error’ (MRE). Taking into
account this errorvalue,
proposed to select the most suitable methodologies. For
a specific set of sequences, those methodologies whose
accuracies exceed a confidence value (?s) are selected
(see the MRE and ?s equations in the Supplementary
a confidence intervalis
RESULTS AND DISCUSSION
Comparison of MSA methodologies
As described before, each MSA method proposes different
solutions depending on certain conditions or features. For
this reason, biologists and researchers do not agree with a
generally accepted solution (19). Some methods have been
developed to unify criteria and choose the most suitable
alignment tool (21,22), but this is currently an open issue.
In order to understand the performance of MSAs,
accuracies from several methodologies can be compared.
Previous reviews (7,8) have already compared accuracies
from BAliBASE subsets (SP scores) against the applied
strategy (progressive, consistency-based or approaches
with additional data). Generally, SP scores are quite
similar independently of the methodologies. Only when
more distant sequences are provided (<20% of identity),
accuracies are significantly higher in methods including
additional data. However, these strategies including add-
itional data are clearly in disadvantage in terms of
required time (7). Thus, we could suggest that, only in
special cases with less related sequences, additional data
are clearly useful.
This analysis supports the idea of using a system to
promising to obtain better alignments. Here, PAcAlCI
predicts accuracies to decide whether differences are
enough to select more sophisticated methods against
faster ones. Therefore, this system not only predicts the
most relevant methodologies, but it also estimates differ-
ences between alignment performances. We could then
decide which method constructs an accurate enough align-
ment according to its predicted accuracy.
methodologies are most
Selection of feature subset
The complete dataset was composed by 2180 different
inputs. Such inputs were retrieved from the 218 groups
of sequences of BAliBASE. They were then aligned by
the 10 previously proposed algorithms. For each input
alignment, a set of 23 features was also retrieved.
Output values were represented by the 2180 accuracies
calculated from the input alignments.
As described above, the mRMR algorithm (34) was
applied to select significant features. That procedure
returned a ranking of features according to their relevance
against calculated accuracies. An increasingly higher
subset of features was then included in the subsequent
system. According to this ranking (Table 2), the most
relevant features were ‘the number of domains’ (f8) and
‘the applied methodology’ (f23). Regarding the first one,
domains can be considered a measure of how deeply se-
quences are known. Domains are also associated with
functional relationships and they involve more conserved
sequence sections. Then, sequences that include more
number of domains will be harder to align and, subse-
quently, the system could provide accurate predictions.
On the other hand, the second feature is an essential
variable because it is including obligatory information.
This feature must always be included in order to know
for which methodology the prediction is done, developing
a robust and coherent system of prediction.
The features related to sequences, ‘the number of
sequences’ (f1) and ‘the average/variance of the length’
(f2,f3), were also ranked among first positions in the
ranking. These features are highlighted because the avail-
ability to obtain accurate alignments directly depends on
sequence properties. Other features less related to se-
quences but including amino acid information were
found in the first half of the ranking. Features such as
‘types of amino acids’ (f18? f22) or ‘the secondary struc-
ture’ (f5,f6) provide complementary information about the
composition and formation of sequences. Thus, they can
also be helpful to efficiently predict some similarities.
Additionally, it is also important to analyze the occur-
rences in BAliBASE of the selected features in order to
understand the obtained feature selection. BAliBASE se-
quences usually have known secondary structures (?-helix
or ?-strand) or GO terms. However, PAcAlCI was also
trained with a few datasets from BAliBASE where these
features are not known; thereby, cases without this infor-
mation were also considered. Thus, new datasets from
users not including that information can also be accurately
estimated, returning their predicted accuracies and a set of
suitable methods to use. On the other hand, datasets
associated with other features
regions) are considerably less included in BAliBASE.
Consequently, the significance obtained for such feature
was considered irrelevant and it was discarded from the
selection procedure (for example, the ‘transmembrane
amino acids’ feature was ranked in the 22nd position).
Prediction of alignment accuracy
Features previously analyzed were added to the subse-
quent LS-SVM model. PAcAlCI then predicted the
PAGE 5 OF 10 Nucleic AcidsResearch, 2013, Vol.41,No. 1e26
accuracy which each methodology returned for a set of
sequences. As far as we are concerned, similar accuracy
predictions in MSAs have not been addressed before.
PAcAlCI was performed using an incremental combin-
ation of features in ascendant relevance order. Such com-
bination was applied adding a feature at a time according
to the previous ranking. Finally, a 10-fold cross-validation
was performed to assess the algorithm. The prediction
error (MRE) was calculated for the training and test sets.
The evolution of the errors for each combination of
features is shown in Figure 2. According to such evolu-
tion, the error progressively decreases with higher number
of features. However, an almost optimal value is reached
from around 10 features. The prediction error is then kept
around 6% for the training data and 9% for the test data.
So, we could suggest that all features are not necessary to
obtain the optimal prediction. A smaller number of
features was then used to perform the system without
lack of accuracy. Specifically, the 10 most relevant
features were added to the LS-SVM model. According
to this configuration, accuracies predicted from four sets
of sequences are shown in Table 3. The total MRE value
returned by PAcAlCI was 0.0587 for the training set and
0.1012 for the test. This error is distributed along the 2180
predicted accuracies as shown in Figure 3.
Analyzing more deeply the proposed system, higher
error values are less frequent and they are usually
associated with low accuracies (see detail in Figure 3).
Alignments with low accuracies are less meaningful in
our system, as their performances are totally unaccept-
able. Consequently, they could not even be considered in
PAcAlCI. A minimal accuracy value, called ?, was then
defined as a threshold. Thus, the LS-SVM model only
kept the most accurate alignments, filtering out the re-
maining ones. This threshold allowed improving the sub-
sequent prediction. For example, if ? ¼ 0:5, the MRE
value using 10 input features decreased to 0.0340 in the
training set and to 0.0608 in the test. As appreciated, error
values were reduced by >2% and >4% for the training
and test set, respectively. These errors improved because
low values of accuracy, which led to highly wrong predic-
tions, were previously filtered (see the new distribution of
errors in Figure 4). Prediction errors are now considered
low enough to adequately determine differences between
methodologies. Then, the most suitable alignments can be
selected according to their predicted accuracies.
Selection of alignment methods
In occasions, several alignment methodologies obtain
quite similar accuracies; thereby, no one significantly
stands out from the rest. Consequently, as in other
few researches (29,38), the most promising MSA tools
can be selected according to several features. In this
case, a confidence interval was defined to decide those
methodologies which acceptably align a set of sequences.
The confidence interval covers those accuracy values that
Table 3. Accuracies obtained for four different sets of sequences
AlignmentMethod Real Acc.Pred. Acc.Rel. error
RV11 4th 3D-Coffee
Predicted accuracies are compared with those obtained by each meth-
odology in four different problems. Values in bold show accuracies
included in the confidence interval. The prediction error is also
05 10 1520 25
NUMBER OF FEATURES USED FOR LSSVM
MEAN ABSOLUTE PERCENTAGE ERROR
Figure 2. Evolution of the MRE. The number of features progressively
increases in ascendant relevance order. Training and test errors are
e26Nucleic Acids Research, 2013,Vol. 41,No. 1PAGE 6 OF 10
are higher than a confidence value ?s (see its formal
Those methodologies whose accuracies exceed such confi-
dence value were chosen as candidate methods.
This confidence interval was applied to real accuracies
and predicted accuracies. Two sets of suitable metho-
dologies were then retrieved (real and predicted sets).
Thus, thenumber ofselected
variable for each group, as it depends on how similar
accuracies were in that specific problem. Both groups
were then compared in order to know how many
methodologies were correctly selected in the predicted
set (see four examples in Figure 5). For example, using
accuracies from the performance of 10 features without
? threshold, the 83.55% of predicted methodologies
were also included in the real group. When accuracies
were predicted with the limitation ? ¼ 0:5, the percentage
85.89%. Therefore, the proposed system usually per-
formed an accurate group of outstanding methodologies.
As shown in the examples of Figure 5, methodologies
including additional information, namely 3D-Coffee and
Promals, were selected for sequences with low similarity
(RV11 subset). In these cases, more commonly used
0 0.10.20.30.4 0.5 0.60.7 0.80.91
0.30.40.50.6 0.7 0.80.91
Figure 4. Distribution of relative errors for training and test sets. Low accuracies were previously filtered to improve the LS-SVM prediction,
avoiding prediction with high errors (? ¼ 0:5).
0 0.10.2 0.30.4 0.50.6 0.70.80.91
Figure 3. Distribution of relative errors for training and test sets. The corresponding LS-SVM prediction was performed using 10 features.
PAGE 7 OF 10Nucleic AcidsResearch, 2013, Vol.41,No. 1e26
aligners (ClustalW, Kalign or Muscle) were not selected, as
they did not build accurate enough alignments. However,
these more complex methods (3D-Coffee and Promals) did
not significantly outperform other faster methods when se-
quences were more related. Thus, methodologies as Mafft,
T-Coffee, Kalign or ProbCons were also selected when se-
quences were highly related (>20% of similarity).
Consequently, we could again suggest that the prediction
system is working as expected. For instance, those datasets
including more than two domains per sequence selected
Kalign as a suitable method (in the 80.95% of cases),
whereas Mafft was appropriate for datasets with less
domains (78% of datasets selected it). Regarding the size
of the dataset, large datasets (>50 sequences or >400
amino acids of average length) usually picked both Mafft
and Kalign (90.17 and 71.9%, respectively), while
ProbCons was chosen for shorter datasets (62.89%).
Finally, Kalign also suited when the sequence lengths in
datasets have a high variability (a difference of >100
amino acids in average between sequence lengths) and
ProbCons for low variability (69.05% and 65.89% of
Although there are other expert systems to select
adequate MSA tools (38,39), PAcAlCI was compared
with AlexSys (29), as it performs a more similar strategy
(see comparison in Table 4). However, comparing both
methodologies can be complicated. Both systems develop
similar machine-learning approaches, but their objectives
are quite different. AlexSys defines a decision-tree
approach to predict whether sequences are ‘strongly’ or
‘weakly’ aligned with each specific method (classification
problem and binary solution). The best method among
those classified as ‘strong’ is then inferred according to
their success probability or their required CPU time.
This binary classification can be quite subjective in some
cases. Since accuracies over 0.5 are already classified as
‘strong’, quite different accuracies, e.g. 0.5 and 0.9, are
considered identical in the AlexSys approach. In a
different way, PAcAlCI first predicts accuracy values (re-
gression problem and real solution). The accuracy predic-
tion provides a relevant improvement in order to decide
whether it is worth aligning with a specific methodology.
Besides, suitable methods are also selected according to
the best accuracies. In general, AlexSys correctly predicts
the best aligner in a 45% of its test alignments. In another
45.5% of the alignments, the best aligner corresponded to
the second predicted method. In general, global success
rates in PAcAlCI are quite similar (83.55% or 85.89%
depending on the ? threshold), although the number of
suitable methods is usually higher in our prediction.
Regarding the included tools, PAcAlCI is composed by
a wider group of previous methodologies (10 approaches
compared with the six of AlexSys), including more
complex ones as 3D-Coffee or Promals.
Despite these differences, both methods may be con-
sidered complementary, as both perform accurate classi-
fiers but in different contexts. In any case, the final
decision of selecting the most suitable methodology
among the proposed ones can rely on the final user of
this system. Other criteria such as the complexity of par-
ameters in the methodologies or the required time could
also be taken into account in order to choose the correct
tool among the selected ones.
MSA is currently an open issue. Alignment tools must be
continually improved, as they are essential in the analysis
of huge amount of data provided by next-generation
sequencing and high-throughput experiments. Thus, new
trends in MSAs aim to integrate the major amount of
information while trying to significantly reduce the used
time. For this reason, efficient computational techniques
are increasingly implemented.
In this work, a complete study of MSA methodologies
has been developed. Relevant methodologies in this field
were first compared. Several types of methods were dis-
cussed and we have suggested that only in special cases
more sophisticated approaches including additional infor-
mation are really necessary. A novel intelligent system
(PAcAlCI) was then proposed based on the knowledge
acquired from this study.
PAcAlCI was designed in order to predict the accuracy
that each alignment method reaches for a specific set of
RV11 subset, 4 alignment
RV11 subset, 20 alignment
RV40 subset, 24th
RV50 subset, 10thalignment
Figure 5. Intersection of suitable and predicted methodologies (Venn
diagrams) corresponding to the four alignments whose accuracies are
shown in Table 3.
Table 4. Comparison between PAcAlCI and AlexSys
Number of aligners
Kind of problem
Values of prediction
45.0% (first aligner)
45.5% (second aligner)
Success rate83.6% (? ¼ 0)
85.9% (? ¼ 0:5)
PAcAlCI is qualitatively compared with AlexSys. The performance and
attributes of both procedures are shown.
e26Nucleic Acids Research, 2013,Vol. 41,No. 1PAGE 8 OF 10
sequences. This information gives us an idea of how ac-
curately each methodology works. The mRMR feature
selection technique was first applied to 23 features previ-
ously retrieved from several biological databases. We have
also described how the system can be performed with only
the 10 most relevant features to predict accuracies with a
reasonable efficiency. Finally, we have proposed the out-
standing methodologies which can be used for certain se-
quences according to their predicted accuracies. In this
sense, the proposed algorithm is able to successfully
select the most outstanding methods according to the pre-
viously predicted accuracies.
Supplementary Data are available at NAR Online:
Supplementary mRMR feature selection, Supplementary
LS-SVM models, Supplementary LS-SVM validation and
Supplementary References [40–47].
Government of Andalusia [Project P09-TIC-175476].
Funding for open access charge: the Government of
Andalusia [Project P09-TIC-175476].
SpanishCICYT [Project SAF2010-20558];the
Conflict of interest statement. None declared.
1. Attwood,T.K. and Parry-Smith,D.J. (2002) Introduction to
Bioinformatics. Pearson Education, Prentice Hall.
2. Pei,J. (2008) Multiple protein sequence alignment. Curr. Opin.
Struct. Biol., 18, 382–386.
3. Gelly,J.C., Joseph,A.P., Srinivasan,N. and de Brevern,A.G. (2011)
iPBA: a tool for protein structure comparison using sequence
alignment strategies. Nucleic Acids Res., 39, W18–W23.
4. Wang,L.S., Leebens-Mack,J., Wall,P.K., Beckmann,K.,
dePamphilis,C.W. and Warnow,T. (2011) The impact of multiple
protein sequence alignment on phylogenetic estimation. IEEE–
ACM Trans. Comput. Biol. Bioinform., 8, 1108–1119.
5. Hicks,S., Wheeler,D.A., Plon,S.E. and Kimmel,M. (2011)
Prediction of missense mutation functionality depends on both
the algorithm and sequence alignment employed. Hum. Mutat.,
6. Li,A.X., Marz,M., Qin,J. and Reidys,C.M. (2011) RNA–RNA
interaction prediction based on multiple sequence alignments.
Bioinformatics, 27, 456–463.
7. Kemena,C. and Notredame,C. (2009) Upcoming challenges for
multiple sequence alignment methods in the high-throughput era.
Bioinformatics, 25, 2455–2465.
8. Li,H. and Homer,N. (2010) A survey of sequence alignment
algorithms for next-generation sequencing. Brief. Bioinform., 11,
9. Thompson,J.D., Higgins,D.G. and Gibson,T.J. (1994) ClustalW:
improving the sensitivity of progressive multiple sequence
weighting, position-specific gap penalties and weight matrix
choice. Nucleic Acids Res., 22, 4673–4680.
10. Edgar,R.C. (2004) MUSCLE: multiple sequence alignment with
high accuracy and high throughput. Nucleic Acids Res., 32,
11. Notredame,C., Higgins,D.G. and Heringa,J. (2000) T-Coffee: a
novel method for fast and accurate multiple sequence alignment.
J. Mol Biol., 302, 205–217.
12. Liu,Y., Schmidt,B. and Maskell,D.L. (2010) MSAProbs: multiple
sequence alignment based on pair hidden Markov models and
partition function posterior probabilities. Bioinformatics, 26,
13. O’Sullivan,O., Suhre,K., Abergel,C., Higgins,D.G. and
Notredame,C. (2004) 3DCoffee: combining protein sequences and
structures within multiple sequence alignments. J. Mol. Biol., 340,
14. Pei,J. and Grishin,N.V. (2007) PROMALS: towards accurate
multiple sequence alignments of distantly related proteins.
Bioinformatics, 23, 802–808.
15. Berman,H.M., Westbrook,J., Feng,Z., Gilliland,G., Bhat,T.N.,
Weissig,H., Shindyalov,I.N. and Bourne,P.E. (2000) The Protein
Data Bank. Nucleic Acids Res., 28, 235–242.
16. Apweiler,R., Bairoch,A., Wu,C.H., Barker,W.C., Boeckmann,B.,
Ferro,S., Gasteiger,E., Huang,H.Z., Lopez,R., Magrane,M. et al.
(2004) UniProt: the Universal Protein knowledgebase. Nucleic
Acids Res., 32, D115–D119.
17. Finn,R.D., Mistry,J., Tate,J., Coggill,P., Heger,A., Pollington,J.E.,
Gavin,O.L., Gunasekaran,P., Ceric,G., Forslund,K. et al. (2010)
The Pfam protein families database. Nucleic Acids Res., 38,
18. Nuin,P.A.S., Wang,Z.Z. and Elisabeth,R.M. (2006) The accuracy
of several multiple sequence alignment programs for proteins.
BMC Bioinformatics, 7, 471.
19. Sierk,M.L., Smoot,M.E., Bass,E.J. and Pearson,W.R. (2010)
Improving pairwise sequence alignment accuracy using
near-optimal protein sequence alignments. BMC Bioinformatics,
20. Thompson,J.D., Koehl,P., Ripp,R. and Poch,O. (2005) BAliBASE
3.0: latest developments of the multiple sequence alignment
benchmark. Proteins, 61, 127–136.
21. Raghava,G.P.S., Searle,S.M.J., Audley,P.C., Barber,J.D. and
Barton,G.J. (2003) OXBench: a benchmark for evaluation of
protein multiple sequence alignment accuracy. BMC
Bioinformatics, 4, 47.
22. Stebbings,L.A. and Mizuguchi,K. (2004) HOMSTRAD: recent
developments of the homologous protein structure alignment
database. Nucleic Acids Res., 32, D203–D207.
23. Lassmann,T. and Sonnhammer,E.L. (2005) Kalign—an accurate
and fast multiple sequence alignment algorithm. BMC
Bioinformatics, 6, 298.
24. Katoh,K., Misawa,K., Kuma,K. and Miyata,T. (2002) MAFFT: a
novel method for rapid multiple sequence alignment based on fast
Fourier transform. Nucleic Acids Res., 30, 3059–3066.
25. Szabo,A., Novak,A., Miklos,I. and Hein,J. (2010) Reticular
alignment: a progressive corner-cutting method for multiple
sequence alignment. BMC Bioinformatics, 11, 570.
26. Wu,S. and Manber,U. (1992) Fast text searching allowing errors.
Commun. ACM, 35, 83–91.
27. Do,C.B., Mahabhashyam,M.S.P., Brudno,M. and Batzoglou,S.
(2005) ProbCons: probabilistic consistency-based multiple
sequence alignment. Genome Res., 15, 330–340.
28. Bradley,R.K., Roberts,A., Smoot,M., Juvekar,S., Do,J.,
Dewey,C., Holmes,I. and Pachter,L. (2009) Fast statistical
alignment. PLoS Comput. Biol., 5, 5.
29. Aniba,M.R., Poch,O., Marchler-Bauer,A. and Thompson,J.D.
(2010) AlexSys: a knowledge-based expert system for multiple
sequence alignment construction and analysis. Nucleic Acids Res.,
30. Wu,X.M., Zhu,L., Guo,J., Zhang,D.Y. and Lin,K. (2006)
Prediction of yeast protein–protein interaction network: insights
from the Gene Ontology and annotations. Nucleic Acids Res., 34,
31. Roslan,R., Othman,R.M., Shah,Z.A., Kasim,S., Asmuni,H.,
Taliba,J., Hassan,R. and Zakaria,Z. (2010) Utilizing shared
interacting domain patterns and Gene Ontology information to
improve protein–protein interaction prediction. Comput. Biol.
Med., 40, 555–564.
32. Camon,E., Magrane,M., Barrell,D., Lee,V., Dimmer,E., Maslen,J.,
Binns,D., Harte,N., Lopez,R. and Apweiler,R. (2004) The Gene
Ontology Annotation (GOA) database: sharing knowledge in
Uniprot with Gene Ontology. Nucleic Acids Res., 32, D262–D266.
33. Mathews,C.K., Holde,K.E.V. and Ahern,K.G. (2000)
Biochemistry. Benjamin Cummings Publishing, Redwood
PAGE 9 OF 10Nucleic AcidsResearch, 2013, Vol.41,No. 1 e26
34. Peng,H., Long,F. and Ding,C. (2005) Feature selection based on Download full-text
mutual information criteria of max-dependency, max-relevance,
and min-redundancy. IEEE Trans. Pattern Anal. Mach. Intell., 27,
35. Suykens,J.A.K., Van Gestel,T., De Brabanter,J., De Moor,B. and
Vandewalle,J. (2003) Least Squares Support Vector Machines.
World Scientific Pub. Co. Inc., Singapore.
36. Li,L., Su,H. and Chu,J. (2009) Sparse representation based on
projection method in online least squares support vector
machines. J. Control Theory Appl., 7, 163–168.
37. De Brabanter,K., Karsmakers,P., Ojeda,F., Alzate,C., De
Brabanter,J., Pelckmans,K., De Moor,B., Vandewalle,J. and
Suykens,J.A.K. (2011), LS-SVMLab: a MATLAB toolbox for least
squares support vector machines (v1.8). http://www.esat.kuleuven
.ac.be/sista/lssvmlab (21 September 2012, date last accessed).
38. Anderson,C.L., Strope,C.L. and Moriyama,E.N. (2011)
SuiteMSA: visual tools for multiple sequence alignment
comparison and molecular sequence simulation. BMC
Bioinformatics, 12, 184.
39. Thompson,J.D., Muller,A., Waterhouse,A., Procter,J.,
Barton,G.J., Plewniak,F. and Poch,O. (2006) MACSIMS: multiple
alignment of complete sequences information management system.
BMC Bioinformatics, 7, 318.
40. Estevez,P.A., Tesmer,M., Perez,C.A. and Zurada,J.M. (2009)
Normalized mutual information feature selection. IEEE Trans.
Neural Netw., 20, 189–201.
41. John,G., Kohavi,R. and Pfleger,K. (1994) Irrelevant features and
the subset selection problem. In International Conference on
Machine Learning, 121–129.
42. Bins,J. and Draper,B.A. (2001) Feature selection from huge
feature sets. In 8th IEEE International Conference on Computer
Vision, 2, 159–165.
43. Cover,T.M. and Thomas,J.A. (2006) Elements of Information
Theory. Wiley-Interscience, New York, NY, USA.
44. Kullback,S. (1997) Information Theory and Statistics. Courier
Dover Publications, New York, NY, USA.
45. Babich,G.A. and Camps,O.I. (1996) Weighted Parzen windows
for pattern classification. IEEE Trans. Pattern Anal. Mach. Intell.,
46. Hestenes,M.R. and Stiefel,E. (1952) Methods of conjugate
gradients for solving linear systems. J. Res. Natl. Bur. Stand., 49,
47. Rossi,F., Lendasse,A., Francois,D., Wertz,V. and Verleysen,M.
(2006) Mutual information for the selection of relevant variables
in spectrometric nonlinear modelling. Chemometrics Intell. Lab.
Syst., 80, 215–226.
e26 Nucleic Acids Research, 2013,Vol. 41,No. 1PAGE 10 OF 10