For those interested, my Ph.D. thesis is available for download [link at archive.org, the WayBack Machine - thanks guys for storing it].
The problem of information search has been recognised for many years: the main issue is one of being able to ﬁlter out the masses of irrelevant information presented rather than accessing relevant information in the ﬁrst place.
Because machines cannot consistently understand a users’ context, the user remains the ultimate arbiter of what documents are relevant to their concerns. The summaries presented by a Web search engine oﬀer a way to quickly scan large amount of data, and being able to accurately judge the relevance of these summaries is important.
In the ﬁrst two experiments, this thesis tested a custom scale and found it reliable and valid. The second experiment compared four methods of summary construction (initial mention, query-biased, keyword and title only), ﬁnding that the title alone enabled the most accurate discrimination. Experiment four partly replicated this ﬁnding, and models of negative modulation were proposed, tested and found to be present in some if not all summaries.
Experiment ﬁve tested relevance decision accuracy with participant generated materials to deal with the high variance of responses encountered in the earlier experiments, but this experiment needed more power to be reliable.
Experiments six and seven tested misleading and accurate summary elements, and found that they are combined as a modulation, though the user ascribes greater signiﬁcance to the snippet than the title. The title however remained the better element for accurate judgements. These experiments also found evidence to support the idea of negative information scent, an extension to Pirolli and Card’s Information Foraging theory.