No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Towards an E-assessment tool for advanced software engineering skills
Abstract
E-assessment enjoys growing attention in university courses and is increasingly applied. Although automated feedback and grading for creative question types is a complex endeavor, more and more tools are developed to support e-assessment for miscellaneous question types. This paper focuses on an automated assessment of UML class diagrams. While there are already tools which support the analysis of student solutions for UML class diagrams, they only provide feedback on a low level. This paper motivates a need for assessing advanced computer science skills. For this, an analysis of student exams is conducted, focusing on exams which instruct the student to choose an appropriate design pattern for a given use case. Further, a prototype is presented which allows for an identification of design patterns, as a first step of a UML assessment tool. For this purpose, an algorithm for error-tolerant subgraph isomorphisms is adopted and extended which deals with multiple design patterns at the same time in order to match them to the students' solutions. Since it is possible to model design patterns in different sizes, a pattern template notation is presented which enables to specify the possible variants of each design pattern. A new algorithm then translates the pattern templates to be understood by the adopted matching algorithm. Finally, the identification process is evaluated empirically, showing the merits and the limitations of this approach.
... Request permissions from permissions@acm.org. SAC '19, and application of design patterns poses a challenge for students [10,17]. This motivates the development of teaching and learning techniques. ...
... Compared to other graph matching approaches, this identification process has to be done in an imprecise fashion in order to be able to cope with mistakes in the student's model. For this purpose, we build upon our previous research [17], in which we leveraged a graph matching technique by Messmer and Bunke [13] featuring error-correcting subgraph isomorphisms (ECSIs). With that, we compared a student's solution to multiple different design patterns in parallel, while efficiently finding the single best match. ...
... The distinction of the feedback into those two stages is done based on the following assumption. In our previous studies, we outlined the difficulties when comparing a student's solution with an exemplary solution in view of feedback generation, if they are based on different design patterns [17]. For an example, we could compare two models modelling the same use case, where the first applies the Decorator and the second one the Abstract Factory pattern. ...
In this paper, we introduce a formative e-assessment tool for building UML class diagrams. Students are tasked to model a UML class diagram for a use case, while choosing a suitable design pattern. The web-based tool is able to automatically assess student solutions and identify the design pattern they applied. This information is then used to advise students on their design pattern choice and give further feedback centred towards an exemplary solution. The paper outlines the applied algorithms for the identification process. Further, we analyse the performance of the design pattern identification based on a field experiment within a CS3 course. In the evaluation, we draw the comparison of our tool's identification quality with that of teachers from the CS3 class. Our results show that the approach is reasonable and helpful for students in order to learn how to select and apply design patterns in different scenarios.
... A UML class diagram test application have been proposed in [17] which aims to evaluate student diagrams developed for documenting an existing software. An e-assessment tool which makes use of design patterns is proposed by [18] for assessing UML class diagram. ...
... It is claimed by [18] that e-assessment is taking a foothold in higher and further education institutions despite its complexity in implementation. For instance, the Open University is expanding the use of its online quiz engines with the aim to facilitate the process of marking and providing feedback to its students [25]. ...
Unified modelling language (UML) is the accepted standard and modelling language for modeling in software development process. UML is widely used by most course tutors in teaching modules of software engineering and system analysis and design. Students taking such courses do submit assignments with UML diagrams such as use case, class, sequence, activity and so on. Different versions of such diagrams produced by the students for a given problem have to be assessed by the course tutor which is a challenging and time-consuming task. This paper presents a java-based tool which is developed based on a simple yet effective algorithm developed by the authors that will read student and tutors solution diagrams as inputs and evaluate and grade the diagrams automatically. The output of the tool is the score of the student diagram in respect of lecturer's final solution. The output is presented in two feedback files, one containing students' score for the lecturers and the other to be send to the student to note the areas that were incorrect. The tool has been tested and evaluated using a simple and assumed UML class diagram. The result shows that the tool functions effectively and can produce detail feedbacks for both students and tutors. The outcome of this paper contributes towards automating UML diagram evaluations.
... However, learning the application of such theoretic concepts is equally important. Researches show that identifying a suitable design pattern in a given scenario and applying it poses a challenge for students [3,6]. In our work, we design an interactive learning environment, which supports students in their process of learning design patterns. ...
... This way, we can cope with mistakes students make when modelling their solution, identify the mistakes, and provide feedback for those. For this purpose, we built upon the approach of Reischmann and Kuchen [6], which specify an mechanism for imprecise identification of design patterns within UML class diagrams. They use a technique by Messmer and Bunke [4], which is called error-correcting subgraph isomorphism. ...
In our research, we design an interactive learning environment for software engineering design patterns. The result is a web-based formative assessment tool, which asks students to choose a suitable design pattern for a given use case and to create a UML class diagram for it. A backend system is able to automatically analyse submissions according to the applied design pattern within each solution. The students are provided with contextualized feedback in order to help them improve their solution and to increase their understanding of design patterns. The paper describes our approach and illustrates the essential parts of the learning environment. Further, we review some insights gathered from a field experiment, in which the learning environment was applied within a software engineering class.
... Impressively, WebWolf accurately identified errors within the submissions [41]. This automated assessment approach extended its utility to the evaluation of UML class diagrams [42]. Furthermore, it provided significant benefits in terms of assessment efficiency and objectivity for educators [43]- [46]. ...
Background: Software engineering are courses comprising various project types, including simple assignments completed in supervised settings and more complex tasks undertaken independently by students, without the oversight of a constant teacher or lab assistant. The imperative need arises for a comprehensive assessment framework to validate the fulfillment of learning objectives and facilitate the measurement of student outcomes, particularly in computer science and software engineering. This leads to the delineation of an appropriate assessment structure and pattern. Objective: This study aimed to acquire the expertise required for assessing student performance in computer science and software engineering courses. Methods: A comprehensive literature review spanning from 2012 to October 2021 was conducted, resulting in the identification of 20 papers addressing the assessment framework in software engineering and computer science courses. Specific inclusion and exclusion criteria were meticulously applied in two rounds of assessment to identify the most pertinent studies for this investigation. Results: The results showed multiple methods for assessing software engineering and computer science courses, including the Assessment Matrix, Automatic Assessment, CDIO, Cooperative Thinking, formative and summative assessment, Game, Generative Learning Robot, NIMSAD, SECAT, Self-assessment and Peer-assessment, SonarQube Tools, WRENCH, and SEP-CyLE. Conclusion: The evaluation framework for software engineering and computer science courses required further refinement, ultimately leading to the selection of the most suitable technique, known as learning framework. Keywords: Computer science course, Software engineering course, Student assessment, Systematic literature review
... Os erros na modelagem de software poderiam classificarse em sintáticos e semânticos [15], mas existem diferentes critérios (como foram observados anteriormente). A literatura aponta a existência de diferentes tipos de erros como: relação, identificação, abstração, dependência, interpretação, confusão, redundância, inconsistência e omissão [24] [11] [12] [26]. Esses erros podem ser evitados com ferramentas de tratamentos, bem como regras ou algoritmos [22], porém, até o momento deste estudo, não há evidência de uma ferramenta que corrija todos os tipos de erros para todos os diferentes diagramas da modelagem de software. ...
This article adresses recurrent errors in software modeling made by students from Software Engineering courses. Software Modeling teachers and related literature agrees that there are certain patterns on errors made by students, although there is no detailed description of them. In this context, we evaluated 2 editions of Requirements Engineering and Systems Analysis of a Software Engineering course with a total of 41 students. Based on this analysis, student errors instances were compared with those found through a literature review, especially the ones involving text interpretation and abstraction difficulties, given these are essential skills for designing and representing models. Our results include a list of most common software modeling errors along with a description of each one, and the resulting catalog can support teachers who seek to develop techniques using error as a mediating instrument for teaching and learning as well as a fundamental resource to reduce them.
Nowadays, modeling exercises on software development objects are conducted in higher education institutions for information technology. Not only are there many defects such as missing elements in the models created by learners during the exercises, but the layout of elements in the class diagrams often differs significantly from the correct answers created by the instructors. In this paper, we focus on the above problem and propose a method to provide effective support to learners during modeling exercises by automatically converting the layout of the learner's class diagram to that of the instructor, in addition to indicating the correctness of the artifacts to the learners during the exercises. The proposed method was implemented and evaluated as a tool, and the results indicate that the automatic layout conversion was an effective feedback to the learners.
Context: Unified Modeling Language (UML) is a widely used modeling language for object-oriented systems development. However, software engineers either did not use the UML at all or used it only selectively and informally. One of the reasons for this is related to the difficulties software engineers face in learning UML. Objective: This study aims to investigate and classify the existing empirical evidence about the difficulties software engineers face when learning to design software using UML diagrams and to categorize the technologies that are being used to assist engineers in the modeling process. Method: We have carried out a systematic mapping study to collect the existing studies that present the difficulties of creating UML diagrams and the technologies proposed to minimize such difficulties. Then, we used a coding approach inspired by open coding and axial coding procedures from Grounded Theory to categorize the difficulties reported by the selected studies. Results: By analyzing 186 studies, we identified empirical evidence of difficulties faced by software engineers while creating UML diagrams. We also identified and classified the difficulties according to the diagram. The results showed that about 68% of the studies present technologies that can assist software engineers in modeling using UML diagrams. Conclusion: The results are helpful for researchers, instructors, and software professionals willing to investigate or implement tools to support engineers during UML model design. We characterize in an unbiased way the difficulties that prevent engineers from elaborating more complete and correct diagrams. The results of the systematic mapping allow us to observe state of the art on technologies that aim to improve the academic teaching and the training of professionals in the industry concerning software modeling using UML diagrams. However, from these results, it has been noticed that few studies report / present techniques that guide a more effective teaching process of UML diagrams.
The paper presents a web-based tool designed to give support to teaching and learning of UML class diagrams. The proposed UML environment is capable to correct automatically UML class diagrams exercises providing feedback to the student immediately. The tool is part of a more general framework, denoted ACME, which provides the main functionalities of an e-learning platform. The tool has been used in a first experimental group of an introductory database course.
Inexact graph matching has been one of the significant research foci in the area of pattern analysis. As an important way
to measure the similarity between pairwise graphs error-tolerantly, graph edit distance (GED) is the base of inexact graph
matching. The research advance of GED is surveyed in order to provide a review of the existing literatures and offer some
insights into the studies of GED. Since graphs may be attributed or non-attributed and the definition of costs for edit operations
is various, the existing GED algorithms are categorized according to these two factors and described in detail. After these
algorithms are analyzed and their limitations are identified, several promising directions for further research are proposed.
To date there has been very little work on the machine understanding of imprecise diagrams - diagrams drawn by students in response to assessment questions. While there have successful attempts at assessing text (essays) automatically, little success with diagrams has been reported. In this paper, we explain an approach to the automatic interpretation of graph-based diagrams based on a 5-stage framework. The paper reports on the evaluation of some experiments in automatically grading student diagrams produced under examination conditions which show good agreement with the performance of human markers. The paper also describes how the automatic marking algorithm is being used in a variety of teaching and learning tools.
A method to determine a distance measure between two nonhierarchical attributed relational graphs is presented. In order to apply this distance measure, the graphs are characterised by descriptive graph grammars (DGG). The proposed distance measure is based on the computation of the minimum number of modifications required to transform an input graph into the reference one. Specifically, the distance measure is defined as the cost of recognition of nodes plus the number of transformations which include node insertion, node deletion, branch insertion, branch deletion, node label substitution and branch label substitution. The major difference between the proposed distance measure and the other ones is the consideration of the cost of recognition of nodes in the distance computation. In order to do this, the principal features of the nodes are described by one or several cost functions which are used to compute the similarity between the input nodes and the reference ones. Finally, an application of this distance measure to the recognition of lower case handwritten English characters is presented.
In this paper, we adopt a template matching method to detect design patterns from a software system by calculating their normalized cross correlation. Because design patterns document flexible design ideas, there can be various ways of implementing them. In our approach, not only the exact matches of pattern instances are detected from system source code, but also the variations of pattern candidates can be identified. Based on our method, we provide tool support and perform experiments on different large open-source systems.
Automated checks for software artefacts like UML diagrams used in automated assessment or tutoring systems do often rely on direct comparisons between a solution and a sample solution. This approach has drawbacks regarding flexibility in face of different possible solutions which are quite common in modeling tasks. This paper presents an alternative technique for checking UML class diagrams based on graph queries which promises to be more flexible.
Design Patterns are proven solution to common recurring design problems. Design Pattern Detection is most important activity that may support a lot to re-engineering process and thus gives significant information to the designer. Knowledge of design pattern exists in the system design improves the program understanding and software maintenance. Therefore, an automatic and reliable design pattern discovery is required. Graph theoretic approaches have been used for design pattern detection in past. Here we are applying an algorithm which decomposes the graph matching process into K phases, where the value of K ranges from 1 to the minimum of the numbers of nodes in the two graphs to be matched. The effectiveness of this algorithm results from the use of small values of K, and significantly reduces the search and space and producing very good matching between graphs. The same algorithm we are here using for design pattern detection from the system design.
Graphs are a powerful and universal data structure useful in
various subfields of science and engineering. In this paper, we propose
a new algorithm for subgraph isomorphism detection from a set of a
priori known model graphs to an input graph that is given online. The
new approach is based on a compact representation of the model graphs
that is computed offline. Subgraphs that appear more than once within
the same or within different model graphs are represented only once,
thus reducing the computational effort to detect them in an input graph.
In the extreme case where all model graphs are highly similar, the
run-time of the new algorithm becomes independent of the number of model
graphs. Both a theoretical complexity analysis and practical experiments
characterizing the performance of the new approach are given
Investigates the influence of the cost function on the optimal
match between two graphs. It is shown that, for a given cost function,
there are an infinite number of other cost functions that lead, for any
given pair of graphs, to the same optimal error correcting matching.
Furthermore, it is shown that well-known concepts from graph theory,
such as graph isomorphism, subgraph isomorphism, and maximum common
subgraph, are special cases of optimal error correcting graph matching
under particular cost functions
We propose a new algorithm for error-correcting subgraph
isomorphism detection from a set of model graphs to an unknown input
graph. The algorithm is based on a compact representation of the model
graphs. This representation is derived from the set of model graphs in
an off-line preprocessing step. The main advantage of the proposed
representation is that common subgraphs of different model graphs are
represented only once. Therefore, at run time, given an unknown input
graph, the computational effort of matching the common subgraphs for
each model graph onto the input graph is done only once. Consequently,
the new algorithm is only sublinearly dependent on the number of model
graphs. Furthermore, the new algorithm can be combined with a future
cost estimation method that greatly improves its run-time performance
The identification of design patterns as part of the reengineering process can convey important information to the designer. However, existing pattern detection methodologies generally have problems in dealing with one or more of the following issues: identification of modified pattern versions, search space explosion for large systems and extensibility to novel patterns. In this paper, a design pattern detection methodology is proposed that is based on similarity scoring between graph vertices. Due to the nature of the underlying graph algorithm, this approach has the ability to also recognize patterns that are modified from their standard representation. Moreover, the approach exploits the fact that patterns reside in one or more inheritance hierarchies, reducing the size of the graphs to which the algorithm is applied. Finally, the algorithm does not rely on any pattern-specific heuristic, facilitating the extension to novel design structures. Evaluation on three open-source projects demonstrated the accuracy and the efficiency of the proposed method
Error correcting graph matching: On the influence of the underlying cost function. IEEE transactions on pattern analysis and machine intelligence
- H Bunke
H. Bunke. Error correcting graph matching: On the
influence of the underlying cost function. IEEE transactions on pattern analysis and machine intelligence,
21(9):917-922, 1999.
A new algorithm for error-tolerant subgraph isomorphism detection. IEEE transactions on pattern analysis and machine intelligence
- B T Messmer
- H Bunke
B. T. Messmer and H. Bunke. A new algorithm for
error-tolerant subgraph isomorphism detection. IEEE
transactions on pattern analysis and machine intelligence, 20(5):493-504, 1998.
Learning and automatically assessing graph-based diagrams
- P G Thomas
- K Waugh
- N Smith
P. G. Thomas, K. Waugh, and N. Smith. Learning and automatically assessing graph-based diagrams.
In Beyond Control: learning technology for the social
network generation. Research Proceedings of the 14th
ALT-C. Citeseer, 2007, pp. 61-74.