Information Retrieval in Large Scale File Systems - Lecture Slides | CSC 6400, Study notes of Computer Science

Download Information Retrieval in Large Scale File Systems - Lecture Slides | CSC 6400 and more Study notes Computer Science in PDF only on Docsity!

2009-4-20 1

Information Retrieval in Large Scale File Systems

CSC 6400: Internet Algorithm

Xin Chen

Introduction

Index Construction and Maintenance

Metadata Searching

Ranking

Online Index Construction and Maintenance

Fast On-Line Index Construction by Geometric Partitioning,

Nicholas Lester, Alistair Moffat, and Justin Zobel

CIKM ’05: Proceedings of the 14th ACM international conference on Information and knowledge management.

Efficient On-line Index Maintenance for Dynamic Text Collections

by Using Dynamic Balancing Tree, Ruijie Guo, Xueqi Cheng,

Hongbo Xu, and Bin Wang.

CIKM ’07: Proceedings of the sixteenth ACM conference on Conference on information and knowledge management.

Problems

In a truly dynamic search environment, documents may be

added to and removed from the collection at any point in

time.

How to build an inverted index when the underlying

data still must be continuously queryable?

Eg. News search, products search, and file system search.

Solutions

 A geometric partitioning mechanism that offers a range

of tradeoffs between costs, and can be adapted to

different balances of insertion and querying operations,

is proposed by Lester et al.

 A dynamic balancing tree (DBT) is proposed by Guo et

al. to achieve a good indexing and query processing

performance by always merging indices with similar

sizes.

Off-line Index construction and Maintenance Techniques

 In-place

 Contiguous on-disk posting lists maximize query processing performance, but require frequent relocations of most lists in the index  Non-contiguous on-disk posting lists increase index maintenance performance, but decrease query processing performance

 Re-merge

This strategy maximizes the query processing performance, however for every merge event the entire index must be processed.

 The key issue:

How best to manage the sequence of mergings so as to minimize the total merging cost, without allowing the number of partitions to grow excessively.

 Requirements:

 If a bufferload contains b pointers, the first partial index cannot exceed (r-1)b pointers, where k is a parameter. In general, the kth partial index is no more than (r-1)rk-1b pointers.  At level k, the partition is either empty, or contains at least rk-1b pointers_._

The merging pattern established when r = 3. After nine bufferloads have been generated by

the in-memory part of the index process, the first index is placed into partition 3. All numbers

listed represent multiples of b , the size of each bufferload.

Fix the number of partitions p , and determine r accordingly.

Querying performance vs. construction performance.

Evaluation – Construction Performance

Dynamic Balancing Tree

 DBT not only deals with document insertion but also

takes care of document deletion.

 The impact of document deletion:

 A large number of posting lists of term have to be decompressed and

re-constructed.

 The complexity of index maintenance is largely increased.

A DBT is an m-way tree,

 Each node of which is a sub-index;  The nodes of the tree is divided into H layers;  At layer k, the number of nodes is either zero, or is less than m (m

= 2 );  The node in layer k+1 is roughly c times bigger than the node in layer k;  Any two nodes in each layer are not significantly in size.  If a DBT is balanced and sub-index merge operation is only performed on one layer.  When a tree is unbalanced, a node is pushed down to a lower layer to make the tree to be balanced.

 Garbage Collection: a bitmap is used to identify the deleted documents and filter out those deleted documents at query processing time. A threshold p is checked to determine whether a garbage collection is integrated into the merging process or not.

Evaluation – Index maintenance Performance

Evaluation– Indexing performance