Monitor Your Backlinks With a 60-Line Python Script

If you have ever paid for a backlink, you have probably also discovered, months later, that it quietly disappeared. The page was redesigned, the post was deleted, the link was switched to nofollow, or the whole site went down. Nobody emails you when that happens.

Checking by hand does not scale past about twenty links. So here is a small Python script that checks a list of backlinks and tells you, for each one, whether the page still loads, whether your link is still on it, and whether it is still followed.

What we actually need to check

For each link there are four separate questions, and they fail independently:

1. Does the page still return HTTP 200?
2. Is there still an <a> tag pointing at your domain?
3. What is its rel attribute now?
4. Is the page still indexable, or did someone add noindex?

A link can pass the first three and fail the fourth, which is the case people miss most often. A page with <meta name="robots" content="noindex"> passes no value no matter how perfect the link is.

The script

It needs requests and beautifulsoup4:

pip install requests beautifulsoup4

The input is a CSV with two columns, page_url and target_domain:

page_url,target_domain
https://example.com/blog/some-post,yoursite.com
https://another.com/resources,yoursite.com

And the checker:

import csv
import time
import requests
from bs4 import BeautifulSoup
from urllib.parse import urlparse

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; LinkChecker/1.0)"}
TIMEOUT = 20


def check(page_url, target_domain):
    result = {
        "page_url": page_url,
        "status": None,
        "found": False,
        "rel": "",
        "anchor": "",
        "indexable": None,
        "note": "",
    }

    try:
        r = requests.get(page_url, headers=HEADERS, timeout=TIMEOUT)
    except requests.RequestException as e:
        result["note"] = f"request failed: {type(e).__name__}"
        return result

    result["status"] = r.status_code
    if r.status_code != 200:
        result["note"] = "page does not return 200"
        return result

    soup = BeautifulSoup(r.text, "html.parser")

    robots = soup.find("meta", attrs={"name": "robots"})
    content = (robots.get("content") or "").lower() if robots else ""
    result["indexable"] = "noindex" not in content

    for a in soup.find_all("a", href=True):
        host = urlparse(a["href"]).netloc.lower()
        if host.endswith(target_domain.lower()):
            rel = " ".join(a.get("rel") or []).lower()
            result["found"] = True
            result["rel"] = rel or "(none)"
            result["anchor"] = a.get_text(strip=True)[:60]
            if "nofollow" in rel or "ugc" in rel or "sponsored" in rel:
                result["note"] = "link is not followed"
            elif not result["indexable"]:
                result["note"] = "followed link on a noindex page"
            else:
                result["note"] = "ok"
            break

    if not result["found"]:
        result["note"] = "link not found on page"

    return result


def main(infile="links.csv", outfile="report.csv"):
    with open(infile, newline="", encoding="utf-8") as f:
        rows = list(csv.DictReader(f))

    results = []
    for row in rows:
        res = check(row["page_url"], row["target_domain"])
        print(f"{res['note']:35} {res['page_url']}")
        results.append(res)
        time.sleep(1)

    with open(outfile, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=list(results[0].keys()))
        writer.writeheader()
        writer.writerows(results)


if __name__ == "__main__":
    main()

Run it and you get a report.csv you can sort by the note column. Everything that does not say ok needs a look.

Reading the results

link not found on page usually means the post was edited or the link was removed outright. Before you email the publisher, open the page yourself, because some sites render links with JavaScript and the script only sees the server HTML. If that is the case for a particular site, check it by hand or render it with a headless browser.

link is not followed means someone added rel="nofollow", ugc or sponsored. On a paid placement this is a perfectly legitimate thing for a publisher to do, and Google's own guidance asks for it. What matters is whether it matches what you agreed. If the agreement said followed and the live link says otherwise, that is a conversation worth having.

followed link on a noindex page is the quiet one. The link looks healthy in every tool that only checks the anchor, but the page is excluded from the index, so nothing flows through it. This happens a lot with author profile pages, tag archives and community platforms.

page does not return 200 is a 404, a 301 to somewhere unrelated, or a dead site. A 301 is not automatically bad, so follow it and see where it lands before you write the link off.

Things this script deliberately does not do

It does not render JavaScript. It does not check whether the page is in Google's index, only whether it allows indexing, and those are different things. It does not respect robots.txt, so keep the one-second delay and do not point it at a thousand URLs on the same host. And it has no retry logic, so a single timeout shows up as a failure even when the page is fine.

Each of those is a reasonable thing to add later. The version above is deliberately small enough that you can read all of it in one sitting.

Running it on a schedule

A weekly run is enough for most people. On Linux:

0 7 * * 1 cd /home/you/linkcheck && /usr/bin/python3 check.py >> log.txt 2>&1

Then diff this week's report.csv against last week's and only read the rows that changed. That turns a long list into a two-minute job, which is the difference between actually doing it and not.

Where the links come from in the first place

Monitoring tells you what you already have; it does not get you anything new. If you are building links manually, the slow part is finding sites worth approaching, and that is its own routine: traffic trend, topical fit, real readers, indexation. We built LinkStone around that problem, and even with a marketplace in front of you, the checks above are still worth running on every link after it goes live.

Summary

Four checks, one small script, one weekly cron entry. The point is not the code, which you will rewrite to fit your own setup. The point is that links decay, nobody tells you when, and the only way to know is to look on a schedule.