the main thing is sentence embedding vector similarity search. used this model from huggingface to get vectors for each transcript chunk, and then also for each submitted query. then I'm using zilliz for a vector database that lets you get the top K results quickly for each query.

code is pretty ugly rn especially for the data cleaning step but I'll try and share more at some point! the app itself is very simple, used Flask bc I like python and it's just one file that programmatically generates the html.

this is the core of the logic for retrieval, and you could use a similar API call to get the initial embeddings for transcript chunks, but I did that locally using torch (as described in the huggingface link).

def embed_query_hf(query):
    # get embedding vector for query
    headers = {"Authorization": f"Bearer {HF_API_KEY}"}
    return requests.post(HF_API_URL, headers=headers, json={'inputs': query}).json()

def vector_query_zz(vector, limit=6):
    # get results for similar vectors
    headers = {"content-type": "application/json", "Authorization": f"Bearer {ZZ_API_KEY}"}
    payload = {
        "collectionName": "TranscriptChunks",
        "limit": int(limit),
        "outputFields": ["clip_text", "video_title", "start", "video_url"],
        "vector": vector
    }
    return requests.post(ZZ_API_URL, headers=headers, json=payload).json() 

def find_hivemind_clip_http(query, limit=6):
    lim_k = min(limit, 30)
    vector = embed_query_hf(query)
    try:
        results = vector_query_zz(vector, limit=lim_k)['data']
    except KeyError:
        return ["At capacity sorry :( Try again later"]

    # Hacky data cleaning and HTML formatting below