Vakkari, [Vak00a, Vak00b], also examined long running searches to examine how relevance 

assessments changed over time. In his study he demonstrated that not only did subjects chose different 

documents at different stages in their task, they also used different search tactics and strategies

38

. 

Vakkari provided support for Spink's observation that high numbers of partial assessments correlates 

with a lack of ability to discriminate relevant and non relevant. This may occur at the start of a search, 

for example. He also found evidence to indicate that when a user has a good idea of what constitutes 

relevant material he is less likely to make a high number of relevance assessments 

These studies are important for RF because they point to the fact that not all relevance assessments are 

equal: users make assessments for different reasons and with different amounts of knowledge. RF 

techniques developed so far tend not to make these distinctions or incorporate this kind of knowledge. 

8 Conclusion

RF has proved to be a useful and pragmatic solution to the uncertainty of describing an information 

need. It has further, in test collection evaluations, been shown to be a relatively stable procedure: it 

works in most cases, a wide range of algorithms give approximately the same performance and how the 

algorithmic parameters should be set are fairly well understood. Although we have not discussed non 

text documents, such as images or speech, in this paper the same basic principle of selecting good 

discriminators of relevance can be used for different media to implement RF functionality.  

The conceptual simplicity of RF   users only have to recognise useful material, not describe it   neatly 

hides the complexity and variety of the query modification features behind the interface. However, 

there is a growing awareness that RF is not sufficient on its own to improve retrieval.  RF is useful in 

that it is conceptually simple but it does not yet provide adequate support for the range of strategies and 

tactics demonstrated by the user in research such as [Bat90]. RF may only be part of the interaction 

process and will require integration with other functionalities. 

Further, although RF is simple for the user to employ, the interaction decisions involved in RF can be 

obscure. That is, RF generally does not give the user enough context on which to based their relevance 

decisions, e.g. how many documents should be marked as relevant, how relevant should a document be 

before being marked as relevant, what does not relevant mean? Although RF research has answers to 

some of these questions (e.g. more relevance information is generally better), getting the user to provide 

the necessary input data is not easy, and making the process of assessing relevance more difficult may 

result in less interaction not more.  

Therefore we argue that the strength of RF shown in non interactive situations should exploited in the 

interactive situation by paying much more attention to the users of RF techniques and how they 

incorporate RF into their searching. Finally, we note that RF is not only a potentially useful technique 

for improving the quality of a searching but is also a very useful for technique for investigating how 

people search. Only by studying how people actually interact with systems can we understand how to 

build more usable and useful search systems. 

Acknowledgements 

We would like to express our thanks to Keith van Rijsbergen and Joemon Jose who gave many useful 

comments on this survey. We would also like to acknowledge the helpful suggestions made by the 

anonymous reviewer. 

References 

[Aal92] I. J. Aalbersberg. Incremental relevance feedback.  Proceedings of the Fifteenth Annual 

International ACM SIGIR Conference on Research and Development in Information Retrieval. pp 11 

22. Copenhagen. 1992. 

[BYRN99] R. Baeza Yates and B. Ribeiro Neto. Modern information retrieval. Addison Wesley. 

1999. 

38

 As measured using Kuhlthau's categorisation of searching, [Kuh91, Kuh93] 

 45