and r hilo measures scored lowest in all cases. These results substantiate the relative merit of the 

algorithms derived for AQE when used for IQE (wpq and F4). They also highlight Robertson's original 

concern that functions designed to measure discriminatory power of existing terms (F4) were not 

necessarily the best to use in selecting new terms, as shown by the better performance of wpq over F4. 

5.3 Performance of IQE against AQE 

Harman s original proposal for IQE was that user selection of expansion terms could give better 

performance than automatic expansion by the system. This may be true for a number of reasons. For 

example the system will typically base its estimate of term utility on very little relevance information 

which could lead to a poor set of expansion terms. A user, on the other hand, will be better able to filter 

out poor terms and only use those s/he feels are appropriate.  

Harman, [Har88], demonstrated that selecting terms could improve retrieval effectiveness in a 

simulated case. Magennis and Van Rijsbergen, [MVR97], extended this study in two ways: by studying 

the degree to which IQE can theoretically improve performance over AQE and whether this theoretical 

improvement can be realised with actual users. 

Magennis and Van Rijsbergen's experiments to determine the theoretical performance of IQE are based 

on Harman's [Har88] notion of a perfect user choice. The choice of a different test collection (the 

larger Wall Street Journal (WSJ) collection) necessitated repeating some of Harman's work. In 

particular they investigated how many terms to add

29

. They found that the range of terms, to 

automatically add to the query, to achieve optimal performance is closer to 0 10 for the WSJ than 

Harman's 20 40 terms for the Cranfield 1400. This shows the difficulty of predicting good estimates of 

numbers of expansion terms, in particular for different collections and different query sets. 

Magennis and Van Rijsbergen repeated Harman's simulation experiment, which expanded the query 

using terms chosen from the relevant documents in the top 20 retrieved documents. They ranked the top 

20 terms chosen from the relevant documents, and added the top n terms. Terms were weighted 

according to their presence in the unseen, or target, relevant documents as the function of query 

expansion is to select terms that are good at retrieving these new relevant documents. The cut off value, 

n, was treated as an experimental variable with five values: 0 (no expansion) 3, 6, 10, and 20 (no 

selection of expansion terms). For all queries, each combination of cut offs was tried. AQE systems 

will generally expand every query by the same number of expansion terms. As a user may expand each 

query by a different number of expansion terms, combinations of cut offs were used to establish the 

best cut off for each query. For example, expand query one by 0 terms, expand query two by 10 terms, 

query three by six terms, etc. Combinations, therefore, allow the simulation of a user adding a variable 

number of expansion terms. The experiment was run over four iterations of feedback and the best 

retrieval effectiveness was taken as the performance that could be expected by an experienced user.  

The best retrieval effectiveness (precision over 100 documents retrieved) for the AQE case was 

achieved by adding the top 6 expansion terms. This method improved precision over automatic 

expansion by all 20 terms. The experienced user simulation outperformed both automatic expansion by 

the top 6 and by the top 20 terms. Moreover, the simulated experienced user selections improved the 

retrieval effectiveness for more queries: it was a more stable improvement over the AQE methods. 

The experiment also compared the performance of the experienced user against Harman's original 

proposal, [Har88], of adding any term that appeared in a relevant, unseen, document. Harman's 

technique worked well against expansion by the top 20 terms, but only marginally better than automatic 

expansion by the top 6 terms, and less well than Magennis and Van Rijsbergen's approach.  This 

supports Harman's 1992 conclusion, [Har92b], that term weighting (as was done in [MVR97] but not 

[Har88]) is important for query expansion. 

A second experiment was run, using the same queries and same test collection, in which experimental 

subjects were asked to select expansion terms. This was designed to test the actual performance of IQE 

when relatively inexperienced users were making the term selection decisions. The subjects could add 

up to 20 terms, (the default being no expansion) and were allowed four iterations of RF. The searchers 

29

 Using the F4 measure to rank terms. 

 35