ArticlePDF Available

The Effectiveness of an Online Abstract Checker Application

Article

The Effectiveness of an Online Abstract Checker Application

Abstract

Writing an abstract is difficult for students and they need a tool to check their abstracts against a confirmed framework. The current paper is a report on the usefulness of the online abstract checker application to the users. The application ‘abstract checker 1.0’ was developed based on Santos’ (1996) Framework for writing a successful scientific abstract. The efficacy of the application was verified in a trial session which saw the participation of 32 postgraduate students from different faculties in Universiti Putra Malaysia (UPM). During the session, participants were given instructions on how to use the tool and their feedback on the performance of the application was collected through a questionnaire and notes from mutual talks with the researchers. The special features of the application are the highlighting of the sentences according to the moves and the Abstract Quality Index (AQI) as an indicator of the level of quality of the submitted abstract. The usefulness of the software was analyzed based on a five point Likert scale and the obtained score showed that generally students found the application very useful. However, more importantly, the feedback received from the participants were invaluable information for the researchers to further fine tune the software for a more efficient and trustworthy application.
RESEARCH ARTICLEXXXXXXXXXXXXXXXXX
* Email Address: syamsiah@upm.edu.my
Copyright © 2015 American Scientific PublishersAdvanced Science Letters
All rights reservedVol. XXXXXXXXX
Printed in the United States of America
The Effectiveness of an Online Abstract Checker
Application
Dariush Saberi1, Syamsiah Mashohor2*, Helen Tan1, Ain Nadzimah Abdullah1
1English Department, Faculty of Modern Languages and Communication, Universiti Putra Malaysia
2Department of Computer and Communication Systems, Faculty of Engineering, Universiti Putra Malaysia
Writing an abstract is difficult for students and they need a tool to check their abstracts against a confirmed framework. The
current paper is a report on the usefulness of the online abstract checker application to the users. The application ‘abstract
checker 1.0was developed based on Santos’ (1996) Framework for writing a successful scientific abstract. The efficacy of
the application was verified in a trial session which saw the participation of 32 postgraduate students from different faculties
in Universiti Putra Malaysia (UPM). During the session, participants were given instructions on how to use the tool and their
feedback on the performance of the application was collected through a questionnaire and notes from mutual talks with the
researchers. The special features of the application are the highlighting of the sentences according to the moves and the
Abstract Quality Index (AQI) as an indicator of the level of quality of the submitted abstract. The usefulness of the software
was analyzed based on a five point Likert scale and the obtained score showed that generally students found the application
very useful. However, more importantly, the feedback received from the participants were invaluable information for the
researchers to further fine tune the software for a more efficient and trustworthy application.
Keywords: Computer Science, Applied Linguistics, Abstract Checker, Abstract Quality Index.
1. INTRODUCTION
Writing a successful abstract is a difficult task as it
requires the writers not only to summarize their scientific
works but also they need to follow a predefined structure
as sanctioned in the academic genre. To overcome this
difficulty especially among novice writers, attempts were
made to develop an online application named ‘abstract
checker’. The application is developed based on Santos’
(1996)Framework1 for the writing of scientific abstracts.
Santos’ (1996) Framework was chosen as it encompasses
a comprehensive structure of moves which fulfill the
communicative purpose of the abstract writing.
The framework consists of five moves namely, Move
1 - situating the research, Move 2 - presenting objectives
of the research, Move 3- describing methodology, Move 4
- discussing the results and Move 5 - summarizing the
findings. This five move taxonomy covers the research
and background, the research methodology as well as a
summary and discussion of the findings. Based on Santos’
(1996) five moves, the application checks the rhetorical
moves in an uploaded abstract and reports the Abstract
Quality Index (AQI) of the abstract. The details of the
development process and the algorithm are reported in a
previously published article by the researchers2. To
evaluate the efficacy of the application, the current paper
reports on the software performance based on feedbacks
received from the users.
2. LITERATURE REVIEW
Computer based teaching and learning tools are one
of the topics that are actively developed in past years. The
educators and researchers are working together to develop
an effective computer-aided teaching and learning tools
due to many benefits. Among the benefits that are realized
are the computers programs help in giving feedback
Adv. Sci. Lett. X, XXXXXX, 2015RESEARCH ARTICLE
2
directly, students can perform self-evaluation and enhance
the student’s capability to learn3. It is believed that
instant feedback is often more educationally effective
than when delivered after a delay, possibly of days or
weeks, for human marking4.
The Intelligent Essay Assessor (IEA)5 is one of
software tools for scoring the quality of essay content.
The IEA uses Latent Semantic Analysis (LSA), which are
both a computational model of human knowledge
representation and a method for extracting semantic
similarity of words and passages from text. From the
performed simulations of psycholinguistic phenomena,
the tool shows that LSA reflects similarities of human
meaning effectively.
Therefore, these requirements are taken into account
in developing an abstract writing application that can
assess the quality of abstract by giving scoring and direct
feedbacks to users. The proposed tool is also aimed to
educate the users to improve their abstract periodically
based on the produced feedbacks and this imitates the
traditional way of teaching and learning.
3. METHODOLOGY
3.1 Dataset
The first version of the software is developed with a
framework consisting of five main stages namely, section
identification, keyword stemming, unique keyword rule,
principle move rule and AQI calculation. The data was
provided from a corpus study that analyzed the keywords in
1,000 thesis abstracts, collected from different faculties in
Universiti Putra Malaysia (UPM). 1000 postgraduate
abstracts were initially crawled from UPM Library’s website.
Of the 1000 abstracts, only those which contained all the 5
moves were selected. From this filtering process, 650
abstracts formed the sample for the study (325 in hard
sciences and 325 in soft sciences). Then, keywords pertinent
to each rhetorical move in the abstracts were identified 5
lists of keywords were prepared in both hard and soft
sciences. The comparison of the lists of keywords between
hard and soft sciences did not show any big difference but
the lists of keywords for each science were separated to
avoid any inconsistency. The keywords were incorporated in
the software afterward.
3.2Algorithm
Step 1 in the software development was ‘section
identification' in which defined the section as a sentence or a
paragraph depending on the length of abstract. Step 2 was
‘stemming of the keywords’. The collected keywords in the
linguistic phase were used in the framework. The list of
keywords went through a stemming phase with suffix
stripping for effective retrieval as reported by several
researchers in English language studies6-7. Afterward, the
keywords were stemmed manually and the regular
expression matching was used to match the words retrieved
from the abstract against the input. The algorithm is reported
previously in details2 and a part is explained here for
clarification.
Definition 1: slis a representation of section in an abstract,
whereby l= {1,2,3,...,maxl}.
Definition 2: i=1,2,3,4,5 represents five-move rule which
are Background (i=1), Objectives (i= 2), Methodology (i= 3),
Results (i= 4) and Conclusion (i= 5).
Definition 3: m is a move detected and it will vary from
m1until m5.
Definition 4: fmis a frequency of move detected in a
sentence, sl. Therefore, every move type has its own
frequency in a sentence. fmtotal is a frequency of particular
move’s occurrence in the whole abstract.
Definition 5: ws is a word retrieved from a sentence and kmjis
a keyword contained by the move’s list. mjrepresents the
move type, m and j is the index for a keyword in the list.
Any matching word wsand keyword kmjwill increase the fmfor
that particular sentence, sl.
Step 3 was ‘unique move rule’ that checks the
uniqueness of the keywords in the moves and analyses their
occurrences in multiple moves. fmin any section, slwhich
contributed by an identified keyword. Therefore, the j
number of keyword for each move type, m will be different
in different input abstract. The rule for removal of the
identified keyword, kmjwill be performed if the unique move
rule is activated (set to 1).
Principal Move Rule was the Step 4, in which the move
analysis overcomes the occurrences of multiple moves, fm in
a sentence, slwhich determine a single move is representing
a sentence. This is appropriate when a sentence is meant to
deliver specific information for one move, as in an ideal case.
Refer to previous article2for more details.
Definition 6: Pmlindicates the principal move for a sentence,
sland maximum value of fmfor respective move type m is
considered as a principal move. The value of fmtotaldependent
to which move that has been identified as Pml, whenever the
principal setting is activated (set to 1).
Step 5 is the last step and it is for AQI Calculation,
formulated in Equation 1:
 




(1)
Whereby wm is a weight given to every move, m and this is
depend on the importance of information for a specific type
of abstract. For engineering field, informative abstracts are
commonly used8 and require detail information on the aim
(m2), methodology (m3) and results (m4).
4. ONLINE APPLICATION
An online responsive interface was developed by the
researchers in the domain ‘abstractchecker.com'. As Figure 1
shows, the interface contains a form for the users to upload
their abstracts, guidelines for using the application and the
results obtained from uploading the abstract in the abstract-
checker.
RESEARCH ARTICLEXXXXXXXXXXXXXXXXX
3
Fig.1. Home page screenshot
As seen in Figure 1, the users are required to enter their
email address. They should also choose their field of study
from the drop-down menu which has two options 'Science
and Technology' and 'Social Science'. The abstract section is
used for the abstract input. The interface converts the input
data into simple text upon submission to remove special
characters and extra punctuations. As Figure 1 shows, a
color coding is used for the identification of each move and
its score. Table 1 shows the color representation of the
moves.
Table.1. Color coding for the moves identification
Move
Name
True Color
Color Code
1
Background
Dark blue
#2176AE
2
Objectives
Light blue
#57B8FF
3
Methodology
Brown
#B66D0D
4
Results
Light orange
#FBB13C
5
Conclusion
Dark orange
#FE6847
The scoring are scaled using color codes such as adequate
(100), satisfactory (70) and inadequate (30). The color
coding used for the scoring are provided in Table 2.
Table.2. Score coding for the checked abstract
Score
True Color
Color Code
100
Green
#28B62C
70
Yellow
#FF851B
30
Red
#FF4136
After successful submission, the result appears in details
with colored segments as shown in Figure2. Figure 2 shows
the colored segments in the checked abstract (upper textbox)
according to the identified moves represented by the
sentences.
Fig.2. The submitted abstract and results of abstract
checking.
Fig.3. AQI and moves representation results
Figure 3 shows an abstract with AQI=68 where move 5
(conclusion) was not found in the abstract and move 4
(results) is inadequate. Moves 1 (background), 2 (objectives)
and 3 (methodology) showed adequate results by obtaining
satisfactory result (in green color, valued with 100). The
graphical representation is believed to make the application
more user-friendly as it aids the users to find the respective
sections easily and to amend the parts for a re-checking.
These features will help users to identify the missing move
and inadequate move in their abstracts. From this feedback,
the user knows what to improve in the next abstract
submission.
Users can amend the abstract by editing the abstract in lower
textbox shown in Figure 2 and click the check button. A new
summary will be shown according to the corrections made.
5. TRIAL SESSION
In order to analyze the effectiveness of the application, a
trial session was conducted in the Universiti Putra Malaysia.
Participants were invited by announcement in the campus to
take part in the trial session. To participate, they were
required to bring a soft copy of their recent abstract. To
begin the trial session, the researchers first gave a detailed
explanation on how to work with the online application.
Then, the students were asked to upload their abstracts and
to report any problems they encounter while using the
application. They were also asked to fill a questionnaire as
Adv. Sci. Lett. X, XXXXXX, 2015RESEARCH ARTICLE
4
feedbacks after they had used the software.
5.1 Participants
In total, 32 students took part in the trial session from whom
5 were undergraduates, 9 were in their Masters and 23 were
PhD students. There was also a fair distribution between
hard and soft sciences participants as well. Distribution of
the participants based on their faculty affiliation is provided
in Table 3. Table.3. Participants Affiliation
Faculty
Bachelor
Master
PhD
Engineering
5
-
6
Modern Languages and
Communication
-
9
7
Educational Studies
-
-
1
Halal Products Research
Institute
-
-
1
Medical Science
-
-
1
Institute of Advanced
Technologies
-
-
1
Computer Science and
Informatics
-
-
1
5.2 Questionnaire
During the trial session, the participants were asked to report
their experience of the application using a questionnaire that
contained five questions. The focus was on the effectiveness
of the software along with the feedbacks on the problems in
the usage and interface. The effectiveness of the software
was analyzed based on the answers to the following
questions:
1) Do you find the abstract checker useful as a student?
(Scoring from 1 (poor) to 5 (excellent))
2) If it is useful, in what ways is the abstract checker
helpful to you in writing your abstract?
3) If it is not useful, please tell us what is/are the
problem/s you encounter while using the abstract
checker?
The participants were asked to give their suggestions for the
improvements in both technical calculation and the interface.
The questions are listed below:
4) How can this abstract checker be improved further?
Please give your suggestions.
5) Do you think the interface is helping you to
construct good abstract? Please suggest an
interface that you think will help you in improving
your abstract.
6. RESULTS
In response to Q1, the participants welcomed the idea. The
usefulness of the software was analyzed in a five points
scale from 1 (poor) to 5 (excellent) and the obtained average
score is 4.57 out of 5, which shows that students found it
very useful (excellent). 100% of the respondents found it
useful. Some comments from the participants are:
1. It give right flow of the writing the abstract
2. It helps us to know the structure of our projects
3. The interface is really good; it gives a very clear
percentage of each of the moves employed in the
abstract.
4. Interface is adequate and colors used are nice and clear.
In relation to the usefulness of the application the responses
from the participants touched on two aspects. The first was on
the abstract structure. The participants (n=10) responded that
the software has helped them to write the required rhetorical
sections of their abstracts. The second was on the quality of the
interface (n=7). The answers from the rest of the students (n=15)
were that they have found the software useful in terms of
writing an abstract which follows the structure of an accepted
academic writing convention.
The answers to Q2 concerned mainly on the improvements
that the software needs. The major points are about the AQI
and moves recognition. Some answers from the participants
are highlighted below
1. The problem that I faced is the abstract checker
highlight my methodology but it displays zero percent
(0%) for my methodology
2. Inaccurate in case of short abstracts
3. The students need to write a clear sentence to declare
the type of sentences for example to clarify whether
the sentence objectives or not
4. The percentage of each AQI factor is not clear
5. The checker is not able to identify the "background"
factor in a proper way.
The answers to the Q3are mainly on the technical problems
related to the move identification. This problem can be
attributed to the limited number of the keywords in the database.
This limitation is recognized and the linguistic team is working
on expanding the corpus and providing more keywords and
common patterns pertinent to each move. The software will be
updated to further improve the efficacy of the new version of
the application.
The respondents also suggested improvements (Q4 and Q5)
in their evaluation and they are related to the weaknesses of
the software. Here are the responses from the participants:
1. May be this checker can suggest and rephrase the word
2. I think various faculties needs to considered and the
way they present their research
RESEARCH ARTICLEXXXXXXXXXXXXXXXXX
5
3. More words need to be added to clearly distinguish
each move
4. Maybe this software could give suggestion a few
keyword that can be used to correct the abstract that
has been uploaded
5. If it can check more sentence structure, that will be
perfect
6. Give justification if there are some lacking
requirements
Most of the respondents expected the software to suggest
keywords for the moves. One student also recommended that
the application can be improved further to incorporate the
checking of short abstracts. Thus, improvement of the
software should take these suggestions into account.
To triangulate the study, the researchers took notes from the
informal interview with the participants. Their response
further affirmed the data obtained from the feedback. As
such findings from the trial session will be used for
improvement in the next version of the software.
7. CONCLUSION
To conclude, the analysis of the feedbacks from the
participants demonstrated that the framework and the
software application have been successful in helping novice
writers write better abstracts. The results obtained also help
researchers to improve on the scope of the stemmed
keywords and AQI accuracy. Therefore, the next version of
the software will include more keywords which were
extracted from the data. Specific keywords for the specific
moves will be fed into the software in order to increase the
uniqueness of the moves. This will help in a more precise
move identification procedure as well.
ACKNOWLEDGMENTS
This current project is sponsored by the Ministry of
Education Malaysia under Fundamental Research Grants
Scheme (FRGS).
REFERENCES
[1] M.B.D Santos. The textual organization of research
paper abstracts in applied linguistics. Text, 16,
4.(1996), pp. 481-499.
[2] Moshohor. S., Tunku Haifaa, T. O., Tan, H., Chan, S.
H., and Ain Nadzimah Abdullah. Engineering Abstract
Quality Index using Santos’ Move. Proc. of The
Second Intl. Conf. On Advances In Computing,
Control And Networking - ACCN 2015 (2015),
pp. 57-61.
[3] Ambar Sri Lestari. Application of Computer Based
Learning Model Tutorial as Medium of Learning.
American Journal of Educational Research, 3, 6.
(2015), pp. 702-706.
[4] J. V. Dempsey and M. P. Driscoll, L. K. Swindell.
Text-Based Feedback. Interactive instruction and
feedback, Educational Technology Publications, New
Jersey (1993), pp. 21-54.
[5] P. W. Foltz and D. Laham, T. K. Landauer. Automated
Essay Scoring: Applications to Educational
Technology. Proceedings of EdMedia: World
Conference on Educational Media and Technology,
(1999), pp. 939-944.
[6] M. F. Porter, An algorithm for suffix stripping.
Program, 14, 3 (1980), pp. 130 - 137.
[7] P. Willett. The Porter stemming algorithm: then and
now, Program, 40, 3 (2006), pp. 219 - 223.
[8] V. Rodrigues. How to write an effective title and
abstract and choose appropriate keywords.
http://www.editage.com/insights/how-to-write-an-
effective-title-and-abstract-and-choose-appropriate-
keywords. (Last access: 22nd February 2016)
... The participants were asked to report their usage experience in a questionnaire adapted from Saberi, Mashohor, et al. (2017) with five questions. The focus was ...
Thesis
Full-text available
Automatic paraphrasing has received much interest in the computational linguistics community in recent years. One type of paraphrasing is lexical substitution (McCarthy and Navigli, 2009), which replaces a word or short phrase with another. Paraphrasing can also involve manipulation of the clausal structure of a sentence, with a range of options that has been described as the “Cline of Metaphoricity” (Halliday and Matthiessen, 2014). According to Halliday, metaphoricity happens inside a sentence when the actions (i.e. verbs and adjectives) are converted into nouns, a process that is known as “nomianlization”. Sentences with multiple verb phrases are considered complex sentences. These complex clauses can be joined either paratactically (i.e. coordinate clauses) or hypotactically (i.e. subordinate clauses). Turning verbs into nouns through nominalization can turn a complex clause into a simple one (Halliday and Webster, 2004). Analysis of academic English corpora has shown that derived nouns and gerunds are very common in professional and academic English texts (Bhatia, 2014). The current study is an endeavor to design an assisted academic English writing system that automates the nominalization and lexical substitution in the ESL/EFL writings. The contributions of the system include an algorithm for noun generation using a neural language model, a model design for an assisted writing CALL system, mappings for nominalization and academic lexical substitution. The evaluations of the system performance in classroom and students’ formal writing show that the system has significant impact in providing assistance to the students in their writing.
Conference Paper
Full-text available
Clearly an abstract plays a key role in providing a synopsis of a study to editors and readers and encouraging them to read the paper. This qualitative study aimed at exploring the effect of using an abstract writing checklist on writing successful abstracts by a group of 27 MA students at a public university in Malaysia. An abstract writing checklist was developed by the researchers based on Hyland's (2000) rhetorical move framework. The students were an intact group of postgraduate students majoring in Applied Linguistics. They were prompted to write 250-word abstracts for intended submission to an international conference. After two weeks, they were briefed on the abstract writing checklist. This briefing session took no more than 30 minutes of going through the items of the checklist with the course lecturer (the second author) who answered the questions raised by the students. Then, the students were given two weeks to revise their abstracts following the checklist. The two drafts of the abstracts were compared qualitatively and the results indicated improvements in students' writing. The results show that self-assessment checklists help ESL learners improve the quality of their academic writing.
Article
Full-text available
Purpose: In 1980, Porter presented a simple algorithm for stemming English language words. This paper summarises the main features of the algorithm, and highlights its role not just in modern information retrieval research, but also in a range of related subject domains. Design: Review of literature and research involving use of the Porter algorithm. Findings: The algorithm has been widely adopted and extended so that it has become the standard approach to word conflation for information retrieval in a wide range of languages. Value: The 1980 paper in Program by Porter describing his algorithm has been highly cited. This paper provides a context for the original paper as well as an overview of its subsequent use.
Article
Research paper abstracts are an important site for the visibility of scientific endeavor. However, little research has been carried out on how abstracts can be characterized in terms of their textual organization and other key features. In addition, advice available in technical writing literature seems to be of little avail to the production of quality abstracts. To help remedy this deficiency, this study investigates the actual discourse organization of 94 abstracts in three leading journals from the field of applied linguistics. A move analysis reveals that abstracts follow a five-move pattern, namely: Move 1 motivates the reader to examine the research by setting the general field or topic and stating the shortcomings of previous study; Move 2 introduces the research by either making a descriptive statement of the article's main focus or by presenting its purpose: Move 3 describes the study design; Move 4 states the major findings; and Move 5 advances the significance of the research by either drawing conclusions or offering recommendations. This descriptive analysis concludes that actual practice does not coincide with the advice available in manuals. The proposed pattern may serve as a pedagogic tool to help researchers in writing informative abstracts and, beyond that, in entering the mainstream of research debate.
Article
The automatic removal of suffixes from words in English is of particular interest in the field of information retrieval. An algorithm for suffix stripping is described, which has been implemented as a short, fast program in BCPL. Although simple, it performs slightly better than a much more elaborate system with which it has been compared. It effectively works by treating complex suffixes as compounds made up of simple suffixes, and removing the simple suffixes in a number of steps. In each step the removal of the suffix is made to depend upon the form of the remaining stem, which usually involves a measure of its syllable length.
Application of Computer Based Learning Model Tutorial as Medium of Learning
  • Ambar Sri
Ambar Sri Lestari. Application of Computer Based Learning Model Tutorial as Medium of Learning.
Text-Based Feedback. Interactive instruction and feedback
  • J V Dempsey
  • M P Driscoll
  • L K Swindell
J. V. Dempsey and M. P. Driscoll, L. K. Swindell. Text-Based Feedback. Interactive instruction and feedback, Educational Technology Publications, New Jersey (1993), pp. 21-54.
Automated Essay Scoring: Applications to Educational Technology
  • P W Foltz
  • D Laham
  • T K Landauer
P. W. Foltz and D. Laham, T. K. Landauer. Automated Essay Scoring: Applications to Educational Technology. Proceedings of EdMedia: World Conference on Educational Media and Technology, (1999), pp. 939-944.
How to write an effective title and abstract and choose appropriate keywords
  • V Rodrigues
V. Rodrigues. How to write an effective title and abstract and choose appropriate keywords.