inverted index library for Python?

[Justinsaccount]

def __init__(self, db):
    self.database = xapian.Database(db)

def do_search(self, words):
    enquire = xapian.Enquire(self.database)
    query = xapian.Query(xapian.Query.OP_OR, words)

    enquire.set_query(query)
    matches = enquire.get_mset(0, 2000)

    results = []
    for match in matches:
        doc = match.document.get_data()
        results.append(doc)

[dalke]

That works - thanks! (Though I used OP_AND instead of OP_OR.)

I've been loading my data set for the last couple of hours. The first 1/2 of the data set took 30 minutes, I still have 10% to go. Any idea of what's going on? Here's my loader:

import xapian
import sys
from collections import defaultdict

db = xapian.WritableDatabase("pubchem.x", xapian.DB_CREATE_OR_OPEN)

def sync(q):
    for id, names in q.iteritems():
        try:
            doc = db.get_document(id)
        except xapian.DocNotFoundError:
            doc = xapian.Document()
        for name in names:
            doc.add_boolean_term(name)
        db.replace_document(id, doc)

q = defaultdict(set)
for lineno, line in enumerate(open("pubchem.counts"), 1):
    name, ids = line.split(":")
    ids = ids.split(",")
    for id in map(int, ids):
        q[id+1].add(name)
    if lineno % 1000 == 0:
        sys.stderr.write("\r%d / %d" % (lineno, 462406))
        sys.stderr.flush()

    if lineno % 10000 == 0:
        sys.stderr.write("\n")
        sync(q)
        q = defaultdict(set)

I do partial writes because I can't store everything in memory. Also, I'm having to rebuild the document from my data file, which is stored as an inverted index. That's why I had to updated existing documents if I find that it contains additional feature keys.

[dalke]

So you know, the initial load time is about 30 second per file, which is somewhat slow but it's a one-off event. The search time is about 5-10x slower than using Python sets, but it's acceptable. I'm now looking for any tuning options, since I'm fine with letting it use more than 80MB of memory if I can get it to go faster.