Abstract The notion of user preference is adopted for the representation, interpretation and measurement of the relevance or usefulness of documents. User judgments on docu- ments may be formally described by a weak order (i.e., user ranking) and measured using an ordinal scale. Within this framework, a new measure of system performance is suggested based on the distance between user ranking and
... [Show full abstract] system ranking. It only uses the relative order of documents and therefore confirms to the valid use of an ordinal scale measuring relevance. It is also applicable to multi-level relevance judg- ments and ranked system output. The appropriateness of the proposed measure is demonstrated through an axiomatic approach. The inherent relationships between the new measure and many existing measures provide further supporting evidence. 1