[show abstract][hide abstract] ABSTRACT: The ability to comprehensively evaluate the quantitative and qualitative performance of an intelligent system is critical to accurately predicting how it will perform in various situations. The design of such evaluations is often as much of a research challenge as is the design of the intelligent systems themselves. Over the past decade, the Intelligent Systems Division, a part of the National Institute of Standards and Technology, has been at the forefront of assessing the performance of the various intelligent systems. This article gives a broad overview of some of the evaluation efforts that have been pursued by the Intelligent Systems Division over the past few years, including performance evaluation of emergency response robots, sensor systems on unmanned ground vehicles, speech-to-speech translation systems, and the development of performance metrics for mixed-palletizing through the use of a simulation environment.
[show abstract][hide abstract] ABSTRACT: As new technologies develop and mature, it becomes critical to provide both formative and summative assessments on their performance. Performance assessment events range in form from a few simple tests of key elements of the technology to highly complex and extensive evaluation exercises targeting specific levels and capabilities of the system under scrutiny. Typically the more advanced the system, the more often performance evaluations are warranted, and the more complex the evaluation planning becomes. Numerous evaluation frameworks have been developed to generate evaluation designs intent on characterizing the performance of intelligent systems. Many of these frameworks enable the design of extensive evaluations, but each has its own focused objectives within an inherent set of known boundaries. This paper introduces the Multi-Relationship Evaluation Design (MRED) framework whose ultimate goal is to automatically generate an evaluation design based upon multiple inputs. The MRED framework takes input goal data and outputs an evaluation blueprint complete with specific evaluation elements including level of technology to be tested, metric type, user type, and, evaluation environment. Some of MRED's unique features are that it characterizes these relationships and manages their uncertainties along with those associated with evaluation input. The authors will introduce MRED by first presenting relationships between four main evaluation design elements. These evaluation elements are defined and the relationships between them are established including the connections between evaluation personnel (not just the users), their level of knowledge, and decision-making authority. This will be further supported through the definition of key terms. An example will be presented in which these terms and relationships are applied to the evaluation design of an automobile technology. An initial validation step follows where MRED is applied to the speech translation technology whose evaluation design was inspired by the successful use of a pre-existing evaluation framework. It is important to note that MRED is still in its early stages of development where this paper presents numerous MRED outputs. Future publications will present the remaining outputs, the uncertain inputs, and MRED's implementation steps that produce the detailed evaluation blueprints.
[show abstract][hide abstract] ABSTRACT: The Army Research Laboratory (ARL) Robotics Collaborative Technology Alliance (CTA) conducted an assessment and evaluation of multiple algorithms for real-time detection of pedestrians in Laser Detection and Ranging (LADAR) and video sensor data taken from a moving platform. The algorithms were developed by Robotics CTA members and then assessed in field experiments jointly conducted by the National Institute of Standards and Technology (NIST) and ARL. A robust, accurate and independent pedestrian tracking system was developed to provide ground truth. The ground truth was used to evaluate the CTA member algorithms for uncertainty and error in their results. A real-time display system was used to provide early detection of errors in data collection.
[show abstract][hide abstract] ABSTRACT: The National Institute of Standards and Technology (NIST) Intelligent Control of Mobility Systems (ICMS) Program provides
architectures and interface standards, performance test methods and data, and infrastructure technology needed by the U.S.
manufacturing industry and government agencies in developing and applying intelligent control technology to mobility systems
to reduce cost, improve safety, and save lives. The ICMS Program is made up of several areas including: defense, transportation,
and industry projects, among others. Each of these projects provides unique capabilities that foster technology transfer
across mobility projects and to outside government, industry and academia for use on a variety of applications. A common theme
among these projects is autonomy and the Four Dimensional (3D+time)/Real-time Control System (4D/RCS) standard control architecture
for intelligent systems that has been applied to these projects.
[show abstract][hide abstract] ABSTRACT: As unmanned ground vehicles take on more and more intelligent tasks, determination of potential obstacles and accurate estimation of their position become critical for successful navigation and path planning. The performance analysis of obstacle mapping and unmanned vehicle positioning in outdoor environments is the subject of this paper. Recently, the National Institute of Standards and Technology's (NIST) Intelligent Systems Division has been a part of the Defense Advanced Research Project Agency LAGR (Learning Applied to Ground Robots) Program. NIST's objective for the LAGR Project is to insert learning algorithms into the modules that make up the NIST 4D/RCS (Four Dimensional/Real-Time Control System) standard reference model architecture which has been successfully applied to many intelligent systems. We detail world modeling techniques used in the 4D/RCS architecture and then analyze the high precision maps generated by the vehicle world modeling algorithms as compared to ground truth obtained from an independent differential GPS system operable throughout most of the NIST campus. This work has implications, not only for outdoor vehicles but also, for indoor automated guided vehicles where future systems will have more and more onboard intelligence requiring non-contact sensors to provide accurate vehicle and object positioning.
[show abstract][hide abstract] ABSTRACT: Tactical behaviors for autonomous ground and air vehicles are an area of high interest to the Army. They are critical for the inclusion of robots in the Future Combat System (FCS). Tactical behaviors can be defined at multiple levels: at the Company, Platoon, Section, and Vehicle echelons. They are currently being defined by the Army for the FCS Unit of Action. At all of these echelons, unmanned ground vehicles, unmanned air vehicles, and unattended ground sensors must collaborate with each other and with manned systems. Research being conducted at the National Institute of Standards and Technology (NIST) and sponsored by the Army Research Lab is focused on defining the Four Dimensional Real-time Controls System (4D/RCS) reference model architecture for intelligent systems and developing a software engineering methodology for system design, integration, test and evaluation. This methodology generates detailed design requirements for perception, knowledge representation, decision making, and behavior generation processes that enable complex military tactics to be planned and executed by unmanned ground and air vehicles working in collaboration with manned systems.
[show abstract][hide abstract] ABSTRACT: This report describes an architecture and software system for automatically performing process planning and control code generation for cutting and inspecting prismatic piece parts. This "Feature-Based Inspection and Control System" (FBICS) consists of seven processes joined by interprocess communication. FBICS starts with a feature-based description of a part. Planners are provided at three hierarchical control levels that consider, in turn: (1) an entire part, (2) work done in a single part fixturing, and (3) work done on a single part feature. FBICS implements the RCS architecture. For data handling, FBICS uses ISO 10303 (STEP) methodology, standards, and tools. FBICS includes a solid modeler and 3D interactive graphics. Control code is written in RS274 for machining and DMIS for inspection.
[show abstract][hide abstract] ABSTRACT: We describe a project to collect and disseminate sensor data for autonomous mobility research. Our goals are to provide data of known accuracy and precision to researchers and developers to enable algorithms to be developed using realistically difficult sensory data. This enables quantitative comparisons of algorithms by running them on the same data, allows groups that lack equipment to participate in mobility research, and speeds technology transfer by providing industry with metrics for comparing algorithm performance. Data are collected using the NIST High Mobility Multi-purpose Wheeled Vehicle (HMMWV), an instrumented vehicle that can be driven manually or autonomously both on roads and off. The vehicle can mount multiple sensors and provides highly accurate position and orientation information as data are collected. The sensors on the HMMWV include an imaging ladar, a color camera, color stereo, and inertial navigation (INS) and Global Positioning System (GPS). Also available are a highresolution scanning ladar, a line-scan ladar, and a multicamera panoramic sensor. The sensors are characterized by collecting data from calibrated courses containing known objects. For some of the data, ground truth will be collected from site surveys. Access to the data is through a web-based query interface. Additional information stored with the sensor data includes navigation and timing data, sensor to vehicle coordinate transformations for each sensor, and sensor calibration information. Several sets of data have already been collected and the web query interface has been developed. Data collection is an ongoing process, and where appropriate, NIST will work with other groups to collect data for specific applications using third-party sensors.
[show abstract][hide abstract] ABSTRACT: The 4D/RCS architecture provides a reference model for military unmanned vehicles on how their software components should be identified and organized. It defines ways of interacting to ensure that missions, especially those involving unknown or hostile environments, can be analyzed, decomposed, distributed, planned, and executed intelligently, effectively, efficiently and in coordination. To achieve this, the 4D/RCS reference model provides well defined and highly coordinated sensory processing, world modeling, knowledge management, cost/benefit analysis, behavior generation, and messaging functions, as well as the associated interfaces. The 4D/RCS architecture is based on scientific principles and is consistent with military hierarchical command doctrine.
[show abstract][hide abstract] ABSTRACT: This is an analysis of standards related to dimensional metrology, with recommendations regarding standards development. The analysis focuses on the degree to which existing and developing standards provide a complete set of non-overlapping specifications for information needed to perform dimensional metrology. The analysis identifies four major software systems and their interfaces. It also identifies 15 dimensional metrology activities, each of which is expected to correspond to a software module. The major systems are made up of sets of modules. Twenty-two application programming interfaces and data formats for dimensional metrology are identified and discussed. The analysis also discusses issues regarding languages in which standards may be written.
[show abstract][hide abstract] ABSTRACT: This paper1 describes an architecture for a system for machining and inspecting mechanical piece parts and an implementation of it called the Feature-Based Inspection and Control System (FBICS). In FBICS, the controller of a machining center or coordinate measuring machine uses a standard feature-based description of the shape of the object to be made as a principal input for machining and/or inspection. FBICS is a hierarchical control system and performs automated hierarchical process planning. FBICS serves: (1) to demonstrate feature-based inspection and control in an open-architecture control system; (2) as a testbed for solving problems in feature-based manufacturing; and (3) to test the usability of STEP methods and models.
[show abstract][hide abstract] ABSTRACT: Abstract This is an analysis of standards related to dimensional metrology, with recommendations,regarding standards development. The analysis focuses on the degree to which existing and developing standards provide a complete set of non-overlapping specifications for information needed to perform dimensional metrology. The analysis
[show abstract][hide abstract] ABSTRACT: This paper describes a reference architecture that is applicable to multiple classes of large-scale, complex real-time control systems. An associated tool, Generic Shell, is also described. Generic Shell employs a set of code templates that facilitate system development and integration. A case study is presented. 1
[show abstract][hide abstract] ABSTRACT: The hierarchical real-time control system (RCS) reference model architecture that is under development at the Na- tional Institute of Standards and Technology (NIST) aims at designing and developing intelligent control for large and complex systems. RCS is based on several general and fundamental principles of complex systems. RCS is a reference model architecture, supporting systems or system components that are developed using Artificial Intelli- gence (AI) and expert systems techniques, neural nets, fuzzy logic, reactive behavior, or other innovative solutions. RCS provides a mechanism to support knowledge engineering process, an important aspect of expert systems. The RCS methodology, the process of applying RCS to control system development, is described. This process unifies several engineering principles including: multiple spatial and temporal resolutions, behavior-oriented, object- oriented, and functional decomposition. The methodology combines a knowledge-based system-design proce- dure and a template-based implementation approach. The authors describe three control systems, either com- pleted or still under development, that progressively describe the advances on the implementation approach and demonstrate the richness of the RCS architecture.
[show abstract][hide abstract] ABSTRACT: The U.S. Army Laboratory Command, as part of its Robotics Initiative, is developing a testbed for cooperative, real-time control of unmanned land vehicles. The system requires the development and integration of many elements which allow the vehicles to perform both autonomous and teleoperated functions. The National Institute of Standards and Technology is supporting this program by developing the vehicle control system using an architecture based on the Realtime Control System (RCS). RCS is a hierarchical, sensory-based control system, initially developed for the control of industrial robots and automated manufacturing systems. In this application, RCS controls all vehicle mobility functions, coordinates the operations of the other subsystems on the vehicle, and communicates between the vehicle and the remote operator control station. This paper reviews the overall control system architecture and the design of the mobility and communication functions. Introduction Artificial intelli...
[show abstract][hide abstract] ABSTRACT: NIST is applying their Real-time Control System (RCS) methodology for control of ground vehicles for both the U.S. Army Research Lab, as part of the DOD's Unmanned Ground Vehicles program, and for the Department of Transportation's Intelligent Vehicle / Highway Systems (IVHS) program. The actuated vehicle, a military HMMWV, has motors for steering, brake, throttle, etc. and sensors for the dashboard gauges. For military operations, the vehicle has two modes of operation: a teleoperation mode - where an operator remotely controls the vehicle over an RF communications network; and a semi-autonomous mode called retro-traverse - where the control system uses an inertial navigation system to steer the vehicle along a prerecorded path. For the IVHS work, intelligent vision processing elements replace the human teleoperator to achieve autonomous, visually guided road following.