Figure 2 - uploaded by Patrick FELIX
Content may be subject to copyright.
Source publication
Testing conceptually consists of three activities: test case generation, test case execution and verdict assignment. Using online testing, test cases are generated and simultaneously executed (i.e. the complete test scenario is built during test execution). This paper presents a framework that automatically generates and executes tests "online" for...
Context in source publication
Citations
... Online testing refers to the testing performed on production software [60,63], as opposed to offline testing, which is done on a development environment, before the software is released. Online testing can be regarded both as a passive monitoring activity [60,61] or as an active testing process [65,80]. In the former, the system is observed under real-world usage aiming to detect inconsistencies or anomalies [59,83]. ...
... In the former, the system is observed under real-world usage aiming to detect inconsistencies or anomalies [59,83]. In the latter, the system is continuously stimulated with new test cases, possibly derived from the feedback obtained from previous test results [65,100]. In this paper, we adopt the second definition and refer to online testing as an active testing process consisting in generating and executing test cases (i.e., API requests) in the APIs in production. ...
... In the research arena, some papers address online testing of RESTful APIs (i.e., testing web APIs in production) [70,72,78,86], but with a limited number of test cases or during a few hours, and typically using a single testing technique. Online testing has also been applied to service-oriented architectures (SOA) [65,67,80], including non-functional testing [62,94], and other less related domains such as mobile applications [66], embedded systems [53] and software product lines [81], among others. ...
Online testing of web APIs—testing APIs in production—is gaining traction in industry. Platforms such as RapidAPI and Sauce Labs provide online testing and monitoring services of web APIs 24/7, typically by re-executing manually designed test cases on the target APIs on a regular basis. In parallel, research on the automated generation of test cases for RESTful APIs has seen significant advances in recent years. However, despite their promising results in the lab, it is unclear whether research tools would scale to industrial-size settings and, more importantly, how they would perform in an online testing setup, increasingly common in practice. In this paper, we report the results of an empirical study on the use of automated test case generation methods for online testing of RESTful APIs. Specifically, we used the RESTest framework to automatically generate and execute test cases in 13 industrial APIs for 15 days non-stop, resulting in over one million test cases. To scale at this level, we had to transition from a monolithic tool approach to a multi-bot architecture with over 200 bots working cooperatively in tasks like test generation and reporting. As a result, we uncovered about 390K failures, which we conservatively triaged into 254 bugs, 65 of which have been acknowledged or fixed by developers to date. Among others, we identified confirmed faults in the APIs of Amadeus, Foursquare, Yelp, and YouTube, accessed by millions of applications worldwide. More importantly, our reports have guided developers on improving their APIs, including bug fixes and documentation updates in the APIs of Amadeus and YouTube. Our results show the potential of online testing of RESTful APIs as the next must-have feature in industry, but also some of the key challenges to overcome for its full adoption in practice.
... Concerning the problem of test case generation, in field testing a test case has a broader scope than just the test input data. The actual novel challenges in generating field tests descend from identifying and reproducing with each test case the relevant interactions with the field, whereas [28,74,78,79] Test Suite Adaptation [38,47,48] Profile Data [34] Field Triggers [72] offline functional IO Data Pattern [81] Metamorphic Relations [10,76] Built-in Tests [54,55,75,93,94,99] Test planning and management [42,43,77] Adaptation and Reconfiguration [62,64] nonfunctional Security Specifications [24,25] online functional Choreographies and Service-based specifications [2,5,6,12,22,98] Finite-State Models [18,20,33,71,95] Metamorphic Relations [21] Graph grammars [83] Event Interface [101] Fault Injection [102] Test planning and management [1,14,27,42,43,59,60] Adaptation and Reconfiguration [32,44,53,62,64] Isolation [17] nonfunctional Stochastic Models of User Behavior [82,89] Security Specifications [12,13,26,46] Usability Models [69] Δ-grammars for QoS [83] Timed-automata [71,95] Fault Injection [3,102] Operational Profile [73] Adaptation and Reconfiguration [32,67,70] a priori we do not expect that new, ad-hoc techniques need to be specified for generating the test inputs data. We cluster techniques in four categories [85] depending on the information used to generate the test cases, as is also done for in-house testing: (i) Specification-based techniques derive test cases from formal or informal requirements specification; (ii) Structure-based techniques use structural information, mostly the code structure, to derive test cases; (iii) Fault-based techniques use models of potential faults (fault models) to derive test cases that address the faults represented in the fault model; and (iv) techniques working with pre-existing test cases exploit already available test cases. ...
... Only 20 of 46 approaches (43%) explicitly describe the proposed oracles. The majority of approaches that explicitly define an oracle use some form of specification-based oracles: component specifications [12], SUT models or specifications [6], and BPEL specifications [20]. Several approaches refer to metamorphic relations [10,21,46,74,76]. ...
Field testing refers to testing techniques that operate in the field to reveal those faults that escape in-house testing. Field testing techniques are becoming increasingly popular with the growing complexity of contemporary software systems. In this article, we present the first systematic survey of field testing approaches over a body of 80 collected studies, and propose their categorization based on the environment and the system on which field testing is performed. We discuss four research questions addressing how software is tested in the field, what is tested in the field, which are the requirements , and how field tests are managed , and identify many challenging research directions.
... Many other approaches have advanced the state of the art regarding automated model-driven functional testing of Web services [28,48,52,56,86], and of composition of Web services through WS-BPEL processes (see [77] for a survey), but they are limited to unit black-box testing, combined with the "assisted automation" of either test case generation or test execution, or test case selection, as currently offered in available state-of-the-art tools such as SoapUI [5], Oracle SOA Suite [69], and Parasoft [16]. In particular, to the best of our knowledge, there is no available solution for the full automation across the spectrum of the critical tasks (test input and oracle generation, test execution/arbitration, dynamic test case prioritisation, test planning) of end-to-end testing of large-scale, multi-stakeholder services architectures. ...
This paper presents the approach to functional test automation of services (black-box testing) and service architectures (grey-box testing) that has been developed within the MIDAS project and is accessible on the MIDAS SaaS. In particular, the algorithms and techniques adopted for addressing input and oracle generation, dynamic scheduling, and session planning issues supporting service functional test automation are illustrated. More specifically, the paper details: (i) the test input generation based on formal methods and temporal logic specifications, (ii) the test oracle generation based on service formal specifications, (iii) the dynamic scheduling of test cases based on probabilistic graphical reasoning, and (iv) the reactive, evidence-based planning of test sessions with on-the-fly generation of new test cases. Finally, the utilisation of the MIDAS prototype for the functional test of operational services and service architectures in the healthcare industry is reported and assessed. A planned evolution of the technology deals with the testing and troubleshooting of distributed systems that integrate connected objects.
... Several testing technics (e.g. [126,127,128]) have been proposed in the last few years. This is because the SOA and especially Web services have been adopted by the industry to develop mission critical applications for different domains, such as robotics, enterprise software and pervasive applications [129,130]. ...
... We can find several approaches in recent literature (e.g. [126,127,128,150]) that uses this technique to verify the behavior of a running system. Both approaches in [126,127] use active testing techniques, which assume that the tester can interact with a SOA (sub-) system at runtime. ...
... [126,127,128,150]) that uses this technique to verify the behavior of a running system. Both approaches in [126,127] use active testing techniques, which assume that the tester can interact with a SOA (sub-) system at runtime. In [127], authors propose an MBT approach based on a model of the orchestrator designed using IOLTSs (Input Output Labeled Transition Systems [151]). ...
As software systems are pervasive and play an important role in everyday life, the users are becoming more and more demanding. They mainly require more reliable systems that automatically adapt to different use cases. To satisfy these requirements, technical frameworks and design methods, upon which the systems development is based, must meet specific objectives mainly modularity, flexibility, and consistency. Service-Oriented Architecture (SOA) is a paradigm that offers mechanisms to increase the software flexibility and reduce development costs by enabling service orchestration and choreography. SOA promises also reliability through the use of services contracts as an agreement between the service provider and consumer. Model-driven SOA is a novel and promising approach that strengthens SOA with Model-Driven Engineering (MDE) technics that ease the specification, development, and verification of Service-Oriented Applications by applying abstraction and automation principles. Despite the progress to integrate MDE to SOA, there are still some challenging problems to be solved: (1) Rigorous verification of SOA system specifications. This is a challenging problem because to model SOA systems designers need more than one viewpoint, each of which captures a specific concern of the system. These viewpoints are meant to be semantically consistent with each other. This problem is called horizontal consistency checking and it is an important step to reduce inconsistencies in SOA models before transforming them into other forms (code generation, test cases derivation, etc.). (2) Transformation of systems specifications into executable artifacts. Despite the maturity of SOA, the transformation of system specifications into executable artifacts is usually manual, fastidious and error-prone. The transformation of services choreographies into executable orchestrations particularly remains a problem because of the necessity to take into account critical aspects of distributed systems such as asynchrony and concurrency when executing centralized orchestrations. (3) Runtime verification. Even after verifying Horizontal consistency at design time, there could be unexpected and unspecified data interactions that are unknown during design-time. For this reason, we still need consistency verification at runtime to handle such unforeseen events. This problem is called Vertical consistency checking. This thesis work proposes a Model-driven SOA approach to address the above-mentioned challenges. This approach includes a two-step model-driven methodology to horizontally and vertically verify the consistency of SOA systems specifications described using the SoaML standard from the Object Management Group (OMG). The horizontal consistency checking problem, which is the first challenge, is solved by means of static analysis of the system specification at the design level. The second challenge is solved by specifying the transformation from a choreography specification model to an executable orchestration implementing the choreography logic. Our transformation takes into consideration the asynchronous nature of the communications between distributed services. The vertical consistency checking problem, which is the third challenge, is solved by our approach thanks to offline analysis that allows consistency verification between both design and runtime levels. The entire methodological proposal was implemented as an extension to the open source UML modeling tool Papyrus.
... The framework presented in [Cao et al. 2010] generates and executes automatically tests for conformance testing of a composite of Web Services described in BPEL. This approach was combined with passive testing, which verifies time traces with respect to a set of constraints [Cao et al. 2011]. ...
Purpose
– This paper aims to address the issue of long-term stability of services and systems depending on service-oriented architecture that has become a popular architecture in systems development and is often implemented using Web services. However, the dependency, especially on externally provided services, can impact the reliability of a system. This is often caused by the loose coupling also implying a less stringent policy for change management and notifications. Therefore, the authors characterise the types of changes that can happen in remote services and propose the concept of resilient web services (RWSs) as an example on how to upgrade existing services to better support the long-term stability of services and systems.
Design/methodology/approach
– Having analysed several use cases where systems broke because of external dependencies not correctly maintained, the authors derived requirements for RWSs.
Findings
– By means of a prototype implementation and evaluation of this solution in a case study, the feasibility of the approach was verified. Several scenarios of changes in WSs were simulated, correctly identified and responded to.
Originality/value
– The authors propose a set of extensions to existing standards such as Web Services Description Language to improve the long-term availability of services in SOAs. A prototype implementation was developed for service monitoring and RWSs.
... Indeed, testing a protocol at runtime may be performed during a normal use of the system without disturbing the process. Several online testing techniques have been studied by the community in order to test systems or protocol implementations [14], [10], [2]. These methods provide interesting studies and have their own advantages, but they also have several drawbacks such as the presence of false negatives, space and time consumption, often related to a needed complete formal model, etc. ...
Testing a protocol at runtime in an online way is a complex and challenging work. It requires the ability to handle numerous messages in a short time, and also requires the same offline testing preciseness. Meanwhile, since online testing is a long term continuously process, the tester has to undergo severe conditions when dealing with large amount of nonstop traces. In this paper, we present a novel logic-based online passive testing approach to test, at runtime, the protocol conformance and performance through formally specified properties with new definitions of verdicts. In order to evaluate and assess our methodology, we experimented our approach with several Session Initiation Protocol properties in a real IP Multimedia Subsystem environment. Relevant verdicts and discussions are provided.
... Indeed, testing a protocol at run-time may be performed during a normal use of the system without disturbing the process. Several online testing techniques have been studied by the community in order to test systems or protocol implementations [10,13,14]. These methods provide interesting studies and have their own advantages, but they also have several drawbacks such as the presence of false negatives, space and time consumption and often related to a needed complete formal model. ...
... In [14], the authors presented a framework that automatically generates and executes tests for conformance testing of a composite of Web services described in BPEL. The proposed framework considers unit testing and it is based on a timed modeling of BPEL specification, and an online testing algorithm that assigns verdicts to every generated state. ...
... In the following loop (Line [8][9][10][11][12][13][14][15][16][17][18][19], if prototype doesn't have dependencies then this will never go inside. In the loop, the program will check all the captured packets with the dependencies. ...
While today’s communications are essential and a huge set of services is available online, computer networks continue to grow and novel communication protocols are continuously being defined and developed. De facto, protocol standards are required to allow different systems to interwork. Though these standards can be formally verified, the developers may produce some errors leading to faulty implementations. That is the reason why their implementations must be strictly tested. However, most current testing approaches require a stimulation of the implementation under tests (IUT). If the system cannot be accessed or interrupted, the IUT will not be able to be tested. Besides, most of the existing works are based on formal models and quite few works study formalizing performance requirements. To solve these issues, we proposed a novel logic-based testing approach to test the protocol conformance and performance passively. In our approach, conformance and performance requirements can be accurately formalized using the Horn-Logic based syntax and semantics. These formalized requirements are also tested through millions of messages collected from real communicating environments. The satisfying results returned from the experiments proved the functionality and efficiency of our approach. Also for satisfying the increasing needs in real-time distributed testing, we also proposed a distributed testing framework and an online testing framework, and performed the frameworks in a real small scale environment. The preliminary results are obtained with success. And also, applying our approach under billions of messages and optimizing the algorithm will be our future works
... Offline testing aims at collecting set of protocol traces while running (through interfaces, ports or points of observations (P.O)) and then checking some properties through these traces afterwards. Several model based offline testing techniques have been studied by the community in order to passively test systems or protocol implementations [16,12,14,2]. Nevertheless, though offline testing still raises many interesting issues [10], online testing approaches bring out these same issues plus the challenges that are inherent to online testing. ...
... Online conformance testing In the online testing area, there are lots of work focus on conformance testing. In [2], the authors present a framework that automatically generates and executes tests "online" for conformance testing of a composite of Web services described in BPEL. The proposed framework considers unit testing and it is based on a timed modeling of BPEL specification, and an online testing algorithm that generates, executes and assigns verdicts to every generated state in the test case. ...
Testing a protocol at runtime in an online way is a complex and challenging work. It requires the same precise-ness in conformance testing and efficiency in performance testing, where conformance testing is a functional test which verifies whether the behaviors of the protocol satisfy defined requirements, and performance testing is a qualitative and quantitative test which checks whether the performance requirements of the protocol have been satisfied under certain conditions. As a matter of course, it raises an interesting issue of converging these two kinds of testing by using the same formal approach, and applying the approach online. In this paper, we present a novel logic-based online testing approach to test the protocol conformance and performance through formally specified properties. In order to evaluate and assess our methodology, we developed a prototype and experimented it with a set of Session Initiation Protocol properties in a real IP Multimedia Subsystem environment. Finally, the relevant verdicts and discussions are provided.
... Integration of expensive services from external providers is not the intended scope. We point out that related approaches which perform online testing of services[53,54,55,56]use similar assumptions. For instance,[53]utilizes online testing for service discovery, binding, and composition. ...
This paper addresses integration testing of data‐centric and event‐based dynamic service compositions. The compositions under test define abstract services that are replaced by concrete candidate services at runtime. Testing all possible instantiations of a composition leads to combinatorial explosion and is often infeasible. We consider data dependencies between services as potential points of failure and introduce the k‐node data flow test coverage metric, which helps to significantly reduce the number of test combinations. We formulate a combinatorial optimization problem for generating minimal sets of test cases. On the basis of this formalization, we present a mapping to the model of FoCuS, a coverage analysis tool. FoCuS efficiently computes near‐optimal solutions, which are used to automatically generate test instances. The proposed approach is applicable to various composition paradigms. We illustrate the end‐to‐end practicability based on an integrated scenario, which uses two diverse composition techniques: on the one hand, the Web Services Business Process Execution Language and on the other hand, WS‐Aggregation, a platform for event‐based service composition. Copyright © 2013 John Wiley & Sons, Ltd.
... They applied a protocol conformance testing technique based using symbolic transition system (STS) to service orchestration testing. Cao et al. [29] proposed a framework that automatically generates and executes tests for conformance testing of service orchestration specified in BPEL. They applied a protocol conformance testing technique using timed extended finite state machine (TEFSM). ...
Testing distributed service-oriented applications (SOAs) is more challenging than testing monolithic applications since these applications have complex interactions between participant services. Test engineers can observe test results only through a front service that handles request messages sent by test engineers. Message exchanges between participant services are hidden behind the front service and cannot be easily observed or controlled through the front service. For this reason, testing SOAs suffer from limited observability and controllability problem.