this paper except that the new query terms are selected manually. This was used fairly evenly across 

searches, i.e. intermediaries and users employed it in approximately the same number of searches but 

intermediaries tended to use the technique more often within a search. This type of feedback was used 

far less often than content or magnitude feedback.  

iii. 

magnitude feedback. Magnitude feedback refers to feedback based on the size of the retrieved 

set of documents. Judgements were that the retrieved set was too large, too small or just about right. 

This type of feedback was the most common observed feedback type. Intermediaries used this type of 

feedback in all searches; users initiated magnitude feedback in around three quarters of the searches. 

However the intermediary made around 81% of all the magnitude feedback decisions. Thus, it appears 

that the intermediaries were more concerned with the size of the retrieved set than the users who were 

more interested in the relevance of the documents, a point also noted by Shenouda, [She91]. This kind 

of feedback is not exclusive to Boolean systems, e.g. White et al [WJR02] reports similar findings on 

best match systems in a small study of searches on Web search engines. 

iv. 

tactical review feedback. This type of feedback (6% of total feedback instances) was based 

around search strategies. Specifically these involved the intermediary examining the search history to 

make a decision about how to proceed with the search, e.g. to avoid repeating a previous search. This 

may not be an operation that is likely to be performed by an inexperienced user of the system. 

v. 

terminology review. This type of feedback, corresponding to around 1% of feedback instances, 

involved the intermediary or user making a strategic decision by looking at terms in inverted file. For 

example the intermediary may search for alternative spellings of query terms. 

The importance of studies such as this is that they indicate that users often want to give information on 

more than just the content of the documents: relevance feedback is not the only important feedback but 

is often the only feedback that is considered or offered by the system. 

7.3.2 Sources of query terms 

Relevance feedback is not the only source of query terms after a user has performed an initial retrieval. 

The user may add more terms or may select terms from other sources, such as a dictionary. The relative 

effectiveness of feedback terms against terms from other sources has been addressed in a study by 

Spink and Saracevic [SS93, SS97]. 

Spink and Saracevic investigated the use and effectiveness of search terms gathered from various 

sources (query, user interaction, term RF, thesaurus, intermediary) during 40 online mediated Boolean 

searches. The search logs were analysed for the first occurrence of each query term, this was taken to be 

the source occurrence of the term for the purposes of classifying the source of the term. Repeated uses 

of the same term were ignored.  

The users were responsible for most (62%) of the search terms but only 38% of the terms came from 

the user's initial query statements. That is the majority of search terms came from the interaction with 

the IR system (after the users have written their information needs). 19% of search terms came from the 

thesaurus, and only 11% of search terms came from term relevance feedback. This is a rather low 

percentage of terms coming from the relevant documents, particularly as the intermediary selected 

terms comprised the majority (65%) of terms chosen using feedback. Term relevance feedback was not 

automatic in this study: terms were chosen manually from the relevant items, which may cause the low 

reported percentage of use. Relevance feedback was only used in about half the searches; again, this 

would seem to be a low percentage.  

The single most successful source of query terms for retrieving relevant documents was the users' query 

statement. Terms from this source retrieved half of the relevant items. Term relevance feedback was 

poor at retrieving relevant items, either on its own or in combination with other sources of terms. 

However, RF, unlike other sources of query terms such as the thesaurus, was more likely to improve 

rather than degrade a search's performance. RF appeared to be used in certain circumstances, for 

example it was often used later in search when there was more interaction or when the user had 

exhausted the other search options.  

This study contains some relatively negative findings for RF, especially the lack of early up take of RF. 

This may be tempered slightly by the fact that the users did not have the opportunity to explore 

automatic RF, which might have facilitated more interaction with feedback terms. 

 43