Building an RFP/RFI tender tracker with clustered search

Sales teams that sell into procurement-driven markets share a quiet problem: the best opportunities are published as RFPs and RFIs on hundreds of scattered portals, newsrooms and tender aggregators, and by the time someone stumbles across one the deadline is often days away. Manual searching does not scale, and generic alerts drown you in noise. A tender tracker solves this by turning the search itself into structured, repeatable code, then filtering the results down to the handful that are genuinely worth a bid.

This guide describes the architecture of such a system in Python: how to design query clusters that cover a domain, how to run them against a search API, how to deduplicate and score the results, and how to schedule the whole thing so a fresh, ranked pipeline lands in your inbox every morning.

Model the domain as query clusters, not a single search

The first instinct is to write one clever search and be done. That fails, because a single query can only express one angle of a broad domain. The better model is a set of query clusters, where each cluster targets one sub-theme with its own vocabulary. One cluster hunts for the core product category, another for adjacent capabilities, another for the regulatory or compliance framing that buyers use, and another for the segment or buyer type.

Each cluster is a small template with a slot for the year, so the same structure sweeps forward each period without rewriting. Expressed as data rather than code, the clusters stay easy to review and extend.

QUERY_CLUSTERS = [
    {"tag": "core", "q": '(intitle:RFP OR intitle:RFI OR intitle:tender) '
                         '("core product term" OR "synonym") '
                         '(buyer OR "buyer type") __YEAR__'},
    {"tag": "adjacent", "q": '("Request for Proposal" OR "invitation to tender") '
                             '("adjacent capability" OR "related module") __YEAR__'},
    {"tag": "compliance", "q": '(RFP OR RFI OR tender OR procurement) '
                               '("regulatory frame" OR "standard") (buyer) __YEAR__'},
]

Advanced operators do most of the precision work. intitle: restricts to documents that announce a tender in their title, quoted phrases lock exact terminology, and OR groups let one cluster cover a family of synonyms. Spending time on operators is what separates a signal-rich tracker from an alert firehose.

Run the clusters against a search API

Search engines do not want to be scraped, so the sustainable path is a search API that returns structured results. Whatever provider you use, the wrapper is the same: expand the year, send the query, and collect the organic results. Centralise the key and a small delay between calls so you stay within rate limits and behave politely.

import os, time, requests

def search(query, num=20):
    resp = requests.post(
        "https://search-api.example/search",
        headers={"X-API-KEY": os.environ["SEARCH_KEY"]},
        json={"q": query, "num": num},
        timeout=15)
    resp.raise_for_status()
    return resp.json().get("organic", [])

def run_clusters(clusters, year):
    hits = []
    for c in clusters:
        q = c["q"].replace("__YEAR__", str(year))
        for r in search(q):
            hits.append({**r, "cluster": c["tag"]})
        time.sleep(1)                      # be polite to the API
    return hits

Tag every result with the cluster that found it. That tag is useful later both for scoring and for understanding which angle of the domain is producing the most live opportunities.

Deduplicate on a normalised URL

Because clusters overlap by design, the same tender will surface several times. Deduplicate on a normalised URL: strip the scheme, drop tracking parameters and trailing slashes, and lowercase the host. Keep the first occurrence but remember every cluster that found it, since a tender matched by three clusters is usually more relevant than one matched by a single narrow query.

from urllib.parse import urlparse, parse_qsl, urlencode

def norm_url(u):
    p = urlparse(u)
    keep = [(k, v) for k, v in parse_qsl(p.query)
            if not k.startswith("utm_")]
    return f"{p.netloc.lower()}{p.path.rstrip('/')}" + \
           (f"?{urlencode(sorted(keep))}" if keep else "")

def dedupe(hits):
    seen = {}
    for h in hits:
        key = norm_url(h["link"])
        if key in seen:
            seen[key]["clusters"].add(h["cluster"])
        else:
            seen[key] = {**h, "clusters": {h["cluster"]}}
    return list(seen.values())

Score relevance so a human reads ten, not a thousand

Deduplication removes noise; scoring surfaces signal. A transparent, rule-based score beats a black box here, because a salesperson needs to trust the ranking. Reward the strong signals and penalise the weak ones, then sort. A workable scheme rewards tender keywords in the title, a recent publication date, and matches across multiple clusters, while penalising sources that historically produce noise.

import re
from datetime import datetime

STRONG = re.compile(r"\b(rfp|rfi|tender|request for proposal|"
                    r"invitation to tender|solicitation)\b", re.I)

def score(item):
    s = 0
    title = item.get("title", "")
    if STRONG.search(title):
        s += 40
    s += 15 * (len(item["clusters"]) - 1)      # cross-cluster agreement
    if any(x in item.get("link", "") for x in ("tender", "procure", "eprocure")):
        s += 20
    if "deadline" in (item.get("snippet", "") or "").lower():
        s += 10
    return s

def rank(items):
    return sorted(items, key=score, reverse=True)

The exact weights matter less than the principle: make the score explainable, tune it against a week of real results, and let the person who acts on the pipeline adjust it. A ranking someone understands is a ranking they will use.

Enrich, then decide what a human sees

The top of the ranked list deserves a little enrichment before it reaches a person. Fetch each candidate page, extract a clean title and any visible deadline, and detect the buying organisation where possible. Keep this step lightweight and defensive, because tender portals vary wildly and a parser that assumes structure will break constantly. Anything that fails enrichment still passes through with its raw fields; enrichment improves a result, it never gates it.

• Pull the page title and meta description for a cleaner summary.
• Regex for date patterns near words like “deadline” or “closing”.
• Flag documents behind logins so nobody wastes time clicking them.

Schedule it and deliver a digest

A tracker is only valuable if it runs without anyone remembering to trigger it. Schedule the full sweep once a day, early, so the ranked pipeline is waiting before the team starts work. Persist the results so you can suppress opportunities already seen and highlight only what is new since yesterday. Then deliver a short digest: the new, high-scoring tenders with a title, a score, a deadline and a link, capped at a length a person will actually read.

def daily(clusters, year, store):
    fresh = rank(dedupe(run_clusters(clusters, year)))
    new = [x for x in fresh if norm_url(x["link"]) not in store]
    for x in new:
        store.add(norm_url(x["link"]))
    return new[:15]                        # top new opportunities

Persisting seen URLs is what turns a noisy daily scrape into a calm stream of genuinely new opportunities. Without it, the digest repeats yesterday’s list and people stop reading.

Go beyond web search for coverage

Clustered web search is the workhorse, but the best trackers blend several source types, because no single channel sees every tender. Many public bodies and large enterprises publish opportunities on dedicated procurement portals that never rank well in general search, so it pays to poll their listings directly where they offer a feed or a predictable listing page. Industry newsrooms and association bulletins often announce major solicitations before they appear anywhere else, and many of them syndicate through RSS, which is trivial to poll on a schedule. Aggregators that collect tenders across a region can cover the long tail that clusters miss, at the cost of more noise to filter.

The architecture absorbs all of these without changing shape. Each source is just another producer of candidate items that flow into the same deduplication, scoring and digest pipeline. A portal poller yields items, an RSS reader yields items, the search clusters yield items, and downstream nothing needs to know where a candidate came from. This separation is what lets the system grow coverage over time: when you notice a source that consistently publishes relevant tenders, you write a small adapter for it and plug it in, and every existing filter and score applies automatically.

Manage false positives deliberately

The enemy of a tender tracker is not missing an opportunity; it is crying wolf so often that people stop trusting the digest. A tracker that surfaces ten irrelevant results for every good one trains its users to ignore it, which is worse than having no tracker at all. Managing false positives is therefore an ongoing discipline, not a one-time tuning.

Two habits keep precision high. The first is a small, evolving list of negative signals: source domains, phrases or document types that have repeatedly produced noise. Penalise or exclude them in the score, and revisit the list whenever a bad result slips through. The second is a weekly review of what the tracker ranked highly versus what the team actually pursued. That gap is the richest tuning signal you have, because it tells you exactly where the score and human judgement diverge. Feed those corrections back into the weights, and precision climbs steadily.

It also helps to be honest about recall versus precision. A tracker tuned for maximum recall catches everything and buries the good in the noise; one tuned for precision shows only the obvious and misses the interesting edge cases. The right balance is usually precision-leaning with a clearly separated “long tail” section for lower-confidence matches, so a curious salesperson can dig deeper without the daily digest being cluttered by maybes.

Where this fits, and where it stops

A clustered-search tracker is deliberately a discovery tool, not a system of record. It finds opportunities early and ranks them; the qualification, the bid decision and the relationship still belong to people. Treated that way it is transformative, because it converts the scattered, time-sensitive world of public procurement into a structured feed that a small team can actually work. Treated as an oracle it disappoints, because no scoring rule replaces a salesperson’s judgement about whether a bid is winnable.

Key takeaways

1. Model the domain as query clusters, each targeting one sub-theme with precise operators.
2. Run clusters through a search API, tagging every result with its cluster.
3. Deduplicate on a normalised URL and keep the set of clusters that matched.
4. Score with a transparent, tunable rule set so a human trusts the ranking.
5. Enrich defensively, and never let a failed parse drop a result.
6. Schedule daily, persist seen URLs, and deliver a short digest of only what is new.

There is a compounding benefit worth naming. Because the whole system is code and data rather than manual searching, it improves every time you touch it. A better operator here, a new source adapter there, a tuned weight after a weekly review, and the signal quality climbs while the effort to run it stays flat. Manual prospecting gets harder as a market grows; a well-built tracker gets sharper, because each improvement is permanent and cumulative rather than a one-off search that has to be repeated by hand tomorrow.

Build it this way and the best opportunities stop slipping past unnoticed. The tracker does the tireless searching; your team spends its judgement where it counts, on the bids worth winning.