It also assures that there is a subset of domains thats actually possible to crawl in sensible time frame. I then extra in some logic to rank matches in domains/urls and titles increased than content material, penalize scaled-down matching paperwork and reward for a longer period types (to offset the bias thats in BM25). I didn’t want to overthink this so I executed BM25 ranking for the principal ranking calculation.