Marvin C. McCallum’s research while affiliated with University of Washington and other places

This report describes the final phase of a research project that examined some key technical requirements of the International Maritime Organization's (IMO) Seafarer's Training, Certification and Watchkeeping Code (STCW Code), as amended in 1995. During this final phase, the issues associated with the conducting of valid and reliable assessments were examined on simulators, on training ships, and, in particular, on commercial vessels. Assessment procedures were adaptable to the circumstances of each of these settings. On commercial ships, regular officers with little training in the process and with little time to devote to preparation, conducted successful assessments using self-contained and ship-specific procedures. Thorough prior preparations of the assessment "package" to be handed over to them were found to be a necessary prerequisite. The different assessment settings (e.g., simulators, training ships, or commercial vessels) offer different strengths for assessing proficiencies, and are discussed in the report.

The present study explored the requirements for developing and documenting performance-based assessments that would conform to the amended Seafarers' Training, Certification and Watchkeeping (STCW) Code. The study had two major objectives: (1) specify and refine a method for developing performance-based assessments of mariner proficiency; and (2) develop an assessment for Automatic Radar Plotting Aid (ARPA) operator proficiency. A five-step method for developing performance-based assessments of mariner proficiency was created on the basis of a review of STCW requirements and the Instructional Systems Development literature, then applied in the development of an ARPA operator proficiency assessment. Overall, the proposed assessment development method was found to be useful in guiding the development of an objective and valid set of performance-based assessment procedures. It was recommended that future efforts be directed toward refinement of the method, development of additional assessment examples, and development of practical guidance and training for the maritime community.

This report describes a process for evaluating the capability of simulators to support performance-based assessment of mariner proficiencies. The step-by-step approach produces an evaluation protocol for examining the following capabilities of a subject simulator: to provide flexible exercise programming to the assessor; to replicate the characteristics of actual equipment; to simulate the operational conditions required to demonstrate the desired performance; and to support thorough debriefing of the assessment candidate. The general method is illustrated with a case study, examining the capability of two PC based simulators to support the assessment of mariner performance in Automatic Radar Plotting Aid (ARPA) operation. The approach can be generalized to the evaluation of simulators of other marine equipment and of equipment in other industries, and to the evaluation of simulators for training. The method is compliant with the requirements of the International Maritime Organization (IMO) Seafarers' Training, Certification and Watchkeeping (STCW) Code and with the best practices of Instructional Systems Development (ISD).

The present study developed casualty investigation procedures that focused on communications problems. These procedures were applied by U.S. Coast Guard (USCG) Investigating Officers in their investigation of 589 marine casualties over a seven-month period. Analysis of the resulting casualty reports determined that communications is a prevalent causal factor in marine casualties, being a factor in 18 percent of critical vessel casualties, 28 percent of critical personnel injuries, and contributing to 19 percent of critical marine casualties overall. The investigations procedures also identified characteristics and causes of communications problems. The single largest problem involved mariners who did not communicate when appropriate. Two types of faulty assumptions were usually the cause of this: they either misinterpreted the situation and did not perceive a threat, or they incorrectly assumed that others were aware of the problem and would take care of it. Training in developing team situation awareness is suggested to combat the first problem. Better crew resource management, specifically empowering crew members to speak up when a threat is perceived, would correct the second problem and potentially reduce communications-related casualties by 29 percent.

This report presents a systematic method for developing reliable and valid performance-based assessments (PBA) of mariner proficiencies. The method is compliant with: the requirements of the International Maritime Organization's (IMO) Seafarers' Training, Certification and Watchkeeping Code (STCW Code), as amended in 1995; with all relevant U.S. Coast Guard Navigation and Vessel Inspection Circulars (NVIC); with Instructional Systems Development (ISD); and with the best practices of the maritime industry. The method guides the developer through five steps: (1) specify assessment objectives, (2) determine assessment methods, (3) specify assessment conditions, (4) develop proficiency criteria, and (5) prepare the assessment materials. The report provides materials for a developer's workshop, a stand-alone assessment development manual, and four sample assessments that have been developed by project participants. Guiding materials for the actual conduct of such assessments are provided in a separate report (R&DC-204).

The contribution of ineffective communications to marine casualties was studied. A standardized procedure was used to guide casualty investigations by Coast Guard investigators over a six-month period. Analyses of the resulting 589 casualty reports were conducted to characterize the extent and nature of communications problems in critical marine casualties. The results suggest that increased efforts to address these problems through improvements in operational procedures and training could reduce occurrences of such casualties. The study procedures were revised for future use in marine casualty investigations.

The present study addressed the applicability of investigation, reporting, and analysis procedures that focus on the role of mariner skill and knowledge limitations in marine casualties. An initial set of procedures was used during a trial implementation by Coast Guard Investigating Officers in their investigation of 389 marine casualties. The procedures were found to be effective. Analysis of the resulting casualty reports provided a basis for determining the prevalence and characteristics of skill and knowledge limitations in marine casualties, as well as the identification of investigator recommendations corresponding to specific causes. Skill and knowledge limitations were found to contribute to 22% of critical casualties, with errors in bridge operations being the predominant cause. The investigation tools allowed for the identification of the specific operational activities which were performed incorrectly and led to the casualty. This information provides the maritime community with specific areas in which improvements are needed in mariner education and shipboard operating procedures. Following the assessment of the initial procedures and analysis of the casualty data, a set of revised investigation tools was developed.

The goal of this project was to develop a structured process for evaluating the capability of simulators to support mariner performance assessment. In this approach, simulator evaluation objectives and criteria are based on performance-based mariner assessment requirements. The approach produces a simulator evaluation protocol that addresses four areas of simulator capability: (1) providing flexible exercise programming to the assessor; (2) replicating the characteristics of actual equipment; (3) simulating the operational conditions required to demonstrate the desired mariner performance; and (4) supporting thorough debriefing of the assessment candidate. A case study showed that our evaluation process was feasible. The process proved to be both sensitive and valuable in differentiating the capabilities of two simulators used to support mariner assessment objectives. This approach can be generalized to the evaluation of simulators of equipment used in other industries.

This paper documents an initial study of the feasibility and potential value of developing an Interactive Rules of the Road Tester. Using an existing desk-top ship bridge simulator, a series of interactive exercises were developed to match the test objectives of a paper and pencil multiple-choice test currently administered to mariners by the U.S. Coast Guard. Both tests were administered to 100 U.S. Merchant Marine Academy students and biographical data relevant to Rules of the Road knowledge and application were obtained for each student. Analyses indicated that the separate test formats measured different aspects of students' competencies and that a test that combines both written items and interactive exercises has the potential to increase the overall level of test validity. Such a system is see as a cost-effective means of improving the assessment of mariner's knowledge and application of the Rules of the Road.

Determining the crew complement needed on a ship is a complex and important process. The level of automation, type of cargo, and type of trade route all affect crew requirements on commercial ships. Technological change and economic pressure is fueling an increased interest in sailing with smaller crews. This in turn has prompted a critical look at traditional methods for determining crew size. The Coast Guard recently developed a computer model to evaluate crew size based on the shipboard tasks required to be performed. This tool, the Crew Size Evaluation Model (CSEM), is a task network model which simulates shipboard activities by specifying when each task occurs and assigning qualified crew members to perform them. The report describes the rigorous validation and sensitivity analyses used to evaluate CSEM. It was found that CSEM is able to provide a valid and flexible analysis of the effects of many different operational factors on crew size.