Figure 1 - uploaded by Dit-Yan Yeung
Content may be subject to copyright.
Source publication
Most of the calculator programs found in existing penbased mobile computing devices, such as personal digital assistants (PDA) and other handheld devices, do not take full advantages of the pen technology offered by these devices. Instead, input of expressions is still done through a virtual keypad shown on the screen, and the stylus (i.e., electro...
Context in source publication
Similar publications
Face detection and recognition have numerous multimedia applications of broad interest, one of which is automatic face annotation. There exist many robust algorithms tackling these problems but most of these algorithms are computationally demanding and have only been implemented in PC- or server- based environments. In this demonstration we show a...
Citations
... Smithies et al. [37] G M B H Toyozumi et al. [142] M B H Chan and Yeung [143] -B H Phan et al. [61] -I H Bott and Laviola Jr [144] G* B H LaViola Jr and Zeleznik [145] G -H Zeleznik et al. [146] G I H MathBrush [147] G M B H Microsoft Office [148] M B H Microsoft Math Solver [149] M* B H MyScript Calculator [150] G I P Samsung S Note [151] M B P SMath [152] M I P Such mode removes the ambiguities related to the choice of instance for editing, reduces the number of elements on the screen. At present, there are no end-to-end solutions that support printed representation mode. ...
... Year Avail. Platform Purpose and brief functionality description PenCalc [143] 2001 P D Simple solving MathPad 2 [145] 2004 P D Animated visualization, solving, simplifying, factoring E-Chalk [156] 2005 -B Lecturer assistant with computational functionality and plotting Anthony et al. [157] 2007 P D Tutoring systems in algebra learning AlgoSketch [158] 2008 P D Pen-based interface for 2D algorithmic description language, which supports iterations, conditions, trace tool. Newton's Pen [159] 2008 P D Diagrams and equilibrium equations PenProof [160] 2010 P D Geometry proving system which correlate figures and ME VectorPad [161] 2010 P -Operations on vectors and 2D/3D visualization SetPad [162] 2012 P D Explore discrete math problems by sketching LogicPad [163] 2012 P D Visualization and verification of Boolean Algebra PhysicsBook [164] 2014 P D Pen-based tutoring system for physics domain that supports sketch recognition, understanding, and animation. ...
Handwritten mathematical expressions are an essential part of many domains, including education, engineering, and science. The pervasive availability of computationally powerful touch-screen devices, similar to the recent emergence of deep neural networks as high-quality sequence recognition models, result in the widespread adoption of online recognition of handwritten mathematical expressions. Also, a deeper study and improvement of such technologies is necessary to address the current challenges posed by the extensive usage of distance learning, and remote work due to the world pandemic. This paper delineates the state-of-the-art recognition methods along with the user’s experience in pen-centric applications for operating with handwritten mathematical expressions. Recognition methods have been categorized into classes, with a description of their merits and limitations. Particular attention is paid to end-to-end approaches based on encoder-decoder architecture and multi-modal input. Evaluation protocols and open benchmark datasets are considered as well as the comparison of the recognition performance, based on open competition results. The use of handwritten math recognition is illustrated by examples of applications for various fields and platforms. A distinctive part of the survey is that we also considered how UI design relies on the use of different recognition approaches, which is aimed at helping potential researchers improve the performance of the introduced approaches toward the best responses in practical applications. Finally, this paper presents the prospective survey of future research directions in handwritten mathematical expression recognition and their applications.
... Competitions on recognizing online HMEs (OHMEs) have been ongoing under the series of CROHME [1] with improved recognition performance. With this progress, many e-learning interfaces based on pen-based devices have been studied [2][3][4] and employed in practical applications. If the recognition result is verified and confirmed by a learner, either online or offline HME recognition can be incorporated into self-learning and e-testing applications. ...
To help human markers mark many answers in the form of online handwritten mathematical expressions (OHMEs), this paper proposes bag-of-features for clustering OHMEs. It consists of six levels of features from low-level pattern features to high-level symbolic and structural features obtained from a state-of-the-art OHME recognizer. Then, it introduces distance-based representation (DbR) to reduce the dimensionality of our proposed feature spaces. Moreover, it presents a method for combining the proposed features to improve the performance. Experiments using the k-means++ algorithm are conducted on a set of 3,150 OHMEs (Dset_50) and an answer dataset (Dset_Mix) of 200 OHMEs intermixed between real patterns and synthesized patterns for each of 10 questions. When the number of clusters is set as the true number of categories, the best purity around 0.99 is produced by bag-of-symbols with DbR for Dset_50, which is better than state-of-the-art methods for clustering offline patterns converted from their OHMEs. The combination of both low-level and high-level features with DbR achieves a purity of around 0.777, increases to more than 0.90 and reduce the marking cost by more than 0.35 point than manually marking OHME answers by adjusting the number of clusters for Dset_Mix.
... In terms of complexity, the method aims to split expressions into smaller ones, so that even using a parser of high complexity, the time for parsing the short-length expression would be low. The method has been also applied for the understanding of handwritten mathematical expressions [7] with very high accuracy. ...
... However, most methods achieve relatively high scores. The method by Chan and Yeung [7], which applies topdown parsing, outperforms all other methods achieving average score 4.10/5.00, followed by the method by Chen, Shimizu and Okada [10] which also applies top-down parsing and achieves average score 3.70/5.00. ...
... The top-down parsing approaches with the best scores are [7] and [11]. The first one relies on backtracking which does not guarantee very good efficiency in the general case [22]. ...
... Some of the products developed along the competitions are commercially available in the market such as the products by MyScript 1 and Wiris 2 . Several tablet-based e-learning interfaces have been researched [8,9,10] and employed for practical applications. ...
... A bottom-up analysis is applied in (Dimitriadis et al., 1991) using horizontal/vertical projections. Similarly, bounding boxes can be used to group strokes (Chan & Yeung, 2001). Other spatial constraints have also been used; a new stroke is judged belonging (or not) to the same symbol of the previous one based on a distance measurement (Tapia & Rojas, 2003). ...
Despite the recent advances in handwriting recognition, handwritten two-dimensional (2D) languages are still a challenge. Electrical schemas, chemical equations and mathematical expressions (MEs) are examples of such 2D languages. In this case, the recognition problem is particularly difficult due to the two dimensional layout of the language. This paper presents an online handwritten mathematical expression recognition system that handles mathematical expression recognition as a simultaneous optimization of expression segmentation, symbol recognition, and 2D structure recognition under the restriction of a mathematical expression grammar. The originality of the approach is a global strategy allowing learning mathematical symbols and spatial relations directly from complete expressions. A new contextual modeling is proposed for combining syntactic and structural information. Those models are used to find the most likely combination of segmentation/recognition hypotheses proposed by a 2D segmentation scheme. Thus, models are based on structural information concerning the symbol layout. The system is tested with a new public database of mathematical expressions which was used in the CHROME competition. We have also produced a large base of semi-synthetic expressions which are used to train and test the global learning approach. We obtain very promising results on both synthetic and real expressions databases, as well as in the recent CHROME competition.
... Other systems such as MathPaper , Algo-Sketch xThink's MathJournal (2003) allow the sketching and writing of mathematics, but rely on in-context menus to allow users to perform manipulations. Littin's (1995) recognition and parsing system, the Natural Log system (Matsakis, 1999), FFES (Smithies et al., 2001), PenCalc (Chan and Yeung, 2001), inftyEditor (Suzuki et al., 2004), and JMathNotes and the related E-Chalk system Rojas, 2003, 2005) are simple equation entry and editing programs without the added benefit of sketching or graphing. Many of the earlier systems on this list are out of date and not maintained. ...
This paper presents the interaction design of, and demonstration of technical feasibility for, intelligent tutoring systems that can accept handwriting input from students. Handwriting and pen input offer several affordances for students that traditional typing-based interactions do not. To illustrate these affordances, we present evidence, from tutoring mathematics, that the ability to enter problem solutions via pen input enables students to record algebraic equations more quickly, more smoothly (fewer errors), and with increased transfer to non-computer-based tasks. Furthermore our evidence shows that students tend to like pen input for these types of problems more than typing. However, a clear downside to introducing handwriting input into intelligent tutors is that the recognition of such input is not reliable. In our work, we have found that handwriting input is more likely to be useful and reliable when context is considered, for example, the context of the problem being solved. We present an intelligent tutoring system for algebra equation solving via pen-based input that is able to use context to decrease recognition errors by 18% and to reduce recognition error recovery interactions to occur on one out of every four problems. We applied user-centered design principles to reduce the negative impact of recognition errors in the following ways: (1) though students handwrite their problem-solving process, they type their final answer to reduce ambiguity for tutoring purposes, and (2) in the small number of cases in which the system must involve the student in recognition error recovery, the interaction focuses on identifying the student’s problem-solving error to keep the emphasis on tutoring. Many potential recognition errors can thus be ignored and distracting interactions are avoided. This work can inform the design of future systems for students using pen and sketch input for math or other topics by motivating the use of context and pragmatics to decrease the impact of recognition errors and put user focus on the task at hand.
... al recognizes simple handwritten expressions [99]. In 2001, Chen and Yeung published a paper on the first penbased calculator [30]. In 2002, the FFES/DRACULAE pen-based equation editor 2 [135,165] was distributed as an open-source prototype. ...
... Unlike the other approaches described in this section, operator-driven decomposition constructs an operator tree (Figure 4b) directly from the symbol layout, rather than first producing a symbol layout tree. The earliest example of a simple pen-based math calculator made use of this method [30]. Lee and Wang [88] use a similar approach to recover symbol layout, using operator dominance to group symbols vertically, followed by determining horizontal adjacencies between symbols. ...
Document recognition and retrieval technolo-gies complement one another, providing improved ac-cess to increasingly large document collections. While recognition and retrieval of textual information is fairly mature, with wide-spread availability of Optical Char-acter Recognition (OCR) and text-based search engines, recognition and retrieval of graphics such as images, fig-ures, tables, diagrams, and mathematical expressions are in comparatively early stages of research. This pa-per surveys the state of the art in recognition and re-trieval of mathematical expressions, organized around four key problems in math retrieval (query construc-tion, normalization, indexing, and relevance feedback), and four key problems in math recognition (detecting expressions, detecting and classifying symbols, analyz-ing symbol layout, and constructing a representation of meaning). Of special interest is the machine learn-ing problem of jointly optimizing the component algo-rithms in a math recognition system, and developing effective indexing, retrieval and relevance feedback al-gorithms for math retrieval. Another important open problem is developing user interfaces that seamlessly integrate recognition and retrieval. Activity in these important research areas is increasing, in part because math notation provides an excellent domain for study-ing problems common to many document and graphics recognition and retrieval applications, and also because mature applications will likely provide substantial ben-efits for education, research, and mathematical literacy.
... L'entrée d'expressions est simplement faite à l'aide d'un clavier virtuel. Chan et al. [33] ont proposé une calculatrice intelligente à base d'écriture manuscrite, permettant l'entrée d'expressions en les écrivant directement sur l'écran par un stylo. De plus, l'utilisateur peut définir des variables pour stocker des résultats intermédiaires. ...
... Une analyse ascendante attribue chaque trait à un groupe, puis chaque groupe est considéré comme un . Une méthode similaire a été proposée dans [33], où les traits qui ont des boîtes englobantes se chevauchant sont regroupés. En qui se situent dans une même fenêtre sont également regroupés. ...
... Une solution est de couvrir plusieurs domaines des mathématiques. Par exemple, quatre domaines ont été choisis dans [33] : l'algèbre élémentaire, les fonctions trigonométrique, la géométrie analytique, et les intégrales indéfinies. Un ensemble de 60 expressions est écrit par 10 scripteurs pour un total de 600 expressions. ...
Les travaux présentés dans le cadre de cette thèse portent sur l'étude, la conception, le développement et le test d'un système de reconnaissance de structures manuscrites bidimensionnelles. Le système proposé se base sur une architecture globale qui considère le problème de reconnaissance en tant qu'optimisation simultanée de la segmentation, de la reconnaissance de symboles, et de l'interprétation. Le premier cadre d'applications a été celui d'un système de reconnaissance d'expressions mathématiques manuscrites. La difficulté du problème se situe aux trois niveaux évoqués. La segmentation est complexe du fait de la grande liberté de composition d'une expression, avec notamment la possibilité de symboles multi-traits non séquentiels ; la reconnaissance doit affronter un nombre élevé de classes et en particulier, gérer les situations de formes non-apprises ; l'interprétation peut-être ambiguë du fait du positionnement spatial approximatif. La solution proposée repose sur la minimisation d'une fonction de coût global qui met en compétition des coûts de reconnaissance et des coûts structurels pour explorer un vaste espace de solutions. Les résultats obtenus sont très compétitifs et prometteurs comparés à ceux de la littérature. Nous avons finalement montré la généricité de notre approche en l'adaptant à la reconnaissance d'un autre type de langage 2D, celui des représentations graphiques de type organigramme.
... A lot of study groups have over the past decades researched how to illustrate the feature information of freehand writing [2]. Also, their research has a widespread application in various environments [3], [4], [5]. Chan and Yeung [5] proposed a feature representation method to recognize handwriting by using fundamental properties of symbol consisting of a primitive set. ...
... Also, their research has a widespread application in various environments [3], [4], [5]. Chan and Yeung [5] proposed a feature representation method to recognize handwriting by using fundamental properties of symbol consisting of a primitive set. This method has a weakness, which makes it difficult to recognize complex symbols or general sketches. ...
... The directional information of input strokes is one of the significant features, in addition to the spatial information of writing. Thus, this factor is often used to recognize shape in several research area [9], [5], [8] [6]. When feature of shape is represented using directional information, one significant factor has to be considered. ...
We propose a powerful shape representation to recognize sketches drawn on a pen-based input device. The proposed method is robust to the sketching order by using the combination of distance map and direction histogram. A distance map created after normalizing a freehand sketch represents a spatial feature of shape regardless of the writing order. Moreover, a distance map which acts a spatial feature is more robust to shape variation than chamfer distance. Direction histogram is also able to extract a directional feature unrelated to the drawing order by using the alignment of the spatial location between two neighboring points of the stroke. The combination of these two features represents rich information to recognize an input sketch. The experiment result demonstrates the superiority of the proposed method more than previous works. It shows 96% recognition performance for the experimental database, which consists of 28 freehand sketches and 10 on-line handwritten digits.
... For mathematical computation, sketch-based interfaces have also been developed by quite a few people and companies. Chan and Yeung developed a simple pen-based calculator PenCalc [9] , and Thimbleby and Thimbleby also developed a calculator with a gesture-based interface supporting animation for simple math calculations [10]. MathBrush [11] recognizes mathematics and drives a symbolic algebra system with it. ...
We present MathPaper, a system for fluid pen-based entry and editing of mathematics with support for interactive computation.
MathPaper provides a paper-like environment in which multiple mathematical expressions and even algorithms can be entered
anywhere on the page. Mathematical expressions can also be modified using simple deletion and dragging gestures with real-time
recognition and computation feedback. In addition, we support extended notations and gestures for controlling computational
assistance, simplifying input, and entering algorithms, making MathPaper a user-friendly system for mathematical sketching
and computation.
