7.3.3 Relevance assessments 

The final aspect of information seeking we shall address, although briefly, is the process of making 

relevance assessments. RF algorithms require users to assess a sample of the retrieved documents but 

the criteria under which a user makes a relevance assessment can be subject to a number of factors. In 

this section, we shall introduce some of these factors. 

One of the main factors is the order in which documents are shown to the user. Several studies, e.g. 

[FM95, EB88], point to the importance of the position of a document in a ranking when assessing the 

relevance of the document. Relevance assessments are relative: viewing one relevant document can 

change the user's perception of the relevance of subsequently viewed documents. Tiamiyu and 

Ajiferuke, [TA88], also looked at the effect that the order in which relevance assessments are made can 

have on retrieval performance. They suggest three types of dependence that can exist in retrieval;  

i. 

independence. Each document should be considered as an independent relevance assessments,  

ii. 

complementarity relationship. The information contained within two documents sums to more 

than the sum of relevance ratings of each document together. 

iii. 

substitutability relationship. The information in one document can substitute for the 

information in another document.  

They show, theoretically, that the presence of different types of relationships can, although, giving same 

recall precision results, give a very different result for user satisfaction. This also brings up the question 

of whether we should treat all relevance assessments as a single set of assessments. Draper, [Dra00], for 

example makes the point that users typically assess individual documents as relevant, not a group of 

documents, whereas RF systems treat relevant documents as a set of related items. 

Janes, [JJ91], also demonstrates that different representations of documents (title, abstract, full text) 

can affect relevance assessments, meaning how the document is presented can affect how likely it is to 

be assessed relevant. 

Relevance assessments are often treated as binary assessments: a document is either relevant or not 

relevant. However, in practice, documents may be regarded as more or less relevant than each other: 

relevance assessments are often partial assessments

37

. Spink et al, [SGB98], examined relevance 

assessments from four separate studies of information seeking to examine the role of partial relevance 

assessments. In particular they looked at whether the use of partial relevance assessments correlated 

with other aspects of searching. The most conclusive finding was the number of partially relevant items 

was often positively correlated with a change in search topic or criteria for relevance: the more partial 

relevance assessments at a given stage in a search, the more uncertain is the user s current information 

need.  

This study concentrated mainly on users at the initial search stage, when information needs are more 

likely to be variable. However, partial relevance assessments as an indicator of search stage or search 

status may be useful in defining what type of documents should be retrieved. For example we may wish 

to increase retrieval of loosely related material at certain stages, and suppress retrieval to only highly 

relevant material at other stages. 

A further important factor in determining how users will make relevance assessments is the task the 

user is trying to complete. Users with different tasks will obviously mark different documents relevant, 

but a user with a long running task may change their criteria for relevance over time. Spink [Spi96] for 

example, reports on a study of when and how academics use IR systems over the course of a research 

project. The majority of users search at the beginning of project and many search again throughout the 

project. One reason for searching at later stages of projects is to check new updated references   

rerunning same searches against new data   but many users modify their search terms over time, either 

as their information problems change or they obtain information from new sources. Although the 

searches are similar and the basic topic of the searches are broadly the same, the reasons for searching 

and the type of information being sought is different leading to different relevance assessments. 

37

 In this context a partial assessment means a document is only somewhat relevant to the topic or the user is not 

sure of the document's relevance. This is distinguished from the situation where only part of the document is 

relevant. 

 44