Loading
Search Engine Optimization • 4 min read

Build Your Own Python SEO Site Crawler: Audit Broken Links & Title Tags with Zero Setup

By Ken M • Published on August 03, 2026
CyberOak Deployment Blueprint Ready

You can load this entire automation pipeline script directly into your runtime console environment.

Deploy to Workspace

Build Your Own Python SEO Site Crawler with Python (Zero Setup Required)

Search engine optimization begins with understanding your website. Before improving rankings, you need to know whether your pages are accessible, whether links are broken, and whether every page has a meaningful title.

Commercial SEO crawlers are powerful, but they can become expensive for freelancers, agencies, and small businesses. Fortunately, Python makes it easy to build a lightweight SEO crawler that performs many of the same core auditing tasks.

In this tutorial, you'll learn how to build a simple website crawler that runs directly on CyberOak. Since CyberOak already includes the required Python libraries, you can upload the script and execute it immediately—no virtual environments or package installation necessary.

What This Automation Does

The crawler starts from a single URL and automatically explores internal pages across your website.

For every page it visits, it records:

  • URL
  • HTTP status code
  • Page title
  • Internal links discovered

The final output is exported as a CSV file that can be opened in Excel or imported into your favorite reporting tool.

Why Build Your Own SEO Crawler?

Many website audits only require a few essential checks.

This script helps identify issues such as:

  • Broken pages (404 errors)
  • Server errors (500 responses)
  • Missing HTML title tags
  • Website structure
  • Internal linking opportunities
  • Redirect chains (with small modifications)

For many technical SEO tasks, a lightweight crawler is more than enough.

Why Run It on CyberOak?

CyberOak removes the friction of running Python automation.

Instead of configuring environments, installing dependencies, or maintaining servers, you simply:

  • Upload the Python script
  • Configure the starting URL
  • Execute the automation
  • Download the generated CSV report

Because CyberOak includes the required libraries out of the box, everything works immediately.

This automation uses:

  • requests
  • beautifulsoup4
  • pandas

No pip install commands required.

How the Crawler Works

The crawler begins with a single URL.

For each page:

  1. Downloads the HTML.
  2. Reads the page title.
  3. Records the HTTP status code.
  4. Finds every internal hyperlink.
  5. Adds newly discovered pages to the crawl queue.
  6. Repeats until the page limit is reached.

To prevent infinite crawling, visited pages are tracked and ignored if encountered again.

Output Example

The generated CSV contains records similar to:

| URL | Status Code | Title | |------|------------:|-------| | https://example.com | 200 | Home | | https://example.com/about | 200 | About Us | | https://example.com/blog | 200 | Blog | | https://example.com/contact | 404 | Missing Title |

This makes it easy to filter broken pages or identify pages with missing titles.

Customizing the Crawl

The script is intentionally simple, making it easy to extend.

You could add support for:

  • Meta descriptions
  • Canonical tags
  • Robots meta directives
  • H1 tag detection
  • Image ALT text auditing
  • Duplicate titles
  • Duplicate content detection
  • XML sitemap comparison
  • Redirect reporting
  • Page response times

CyberOak already provides many of the libraries required for these enhancements.

Scheduling SEO Audits

One of the biggest advantages of running this script on CyberOak is automation.

Instead of remembering to audit your website manually, schedule the crawler to run:

  • Daily
  • Weekly
  • Monthly
  • Before every deployment

Historical crawl reports help identify SEO issues before they impact search rankings.

Who Can Use This?

This automation is ideal for:

  • SEO specialists
  • Digital marketing agencies
  • Website owners
  • Developers
  • Content teams
  • Freelance consultants

Whether you're auditing a small business website or monitoring hundreds of pages, this crawler provides a fast starting point.

Conclusion

A technical SEO audit doesn't have to require expensive software.

Using Python, Requests, BeautifulSoup, and Pandas, you can build an efficient website crawler that checks page availability, discovers internal links, and audits title tags.

Upload the script to CyberOak, run it whenever you need, and generate SEO reports in minutes without worrying about Python environments or dependency management.

preview.py 
from urllib.parse import urljoin, urlparse

import pandas as pd
import requests
from bs4 import BeautifulSoup

# Configure the website to crawl.
# In CyberOak, create Key-Value Parameters:
# START_URL = https://example.com
# MAX_PAGES = 20

START_URL = START_URL
MAX_PAGES = int(MAX_PAGES)


class SEOCrawler:

    def __init__(self, base_url, max_pages=50):
        self.base_url = base_url.rstrip("/")
        self.domain = urlparse(base_url).netloc
        self.max_pages = max_pages
        self.visited = set()
        self.to_visit = [base_url]
        self.results = []

    def is_internal(self, url):
        parsed = urlparse(url)
        return parsed.netloc == self.domain or parsed.netloc == ""

    def crawl(self):
        while self.to_visit and len(self.visited) < self.max_pages:
            url = self.to_visit.pop(0)

            if url in self.visited:
                continue

            self.visited.add(url)

            try:
                response = requests.get(
                    url,
                    timeout=10,
                    headers={
                        "User-Agent": "CyberOak SEO Crawler/1.0"
                    },
                )

                status = response.status_code
                title = "N/A"

                if "text/html" in response.headers.get(
                    "Content-Type", ""
                ):
                    soup = BeautifulSoup(
                        response.content,
                        "html.parser",
                    )

                    title_tag = soup.find("title")
                    title = (
                        title_tag.get_text(strip=True)
                        if title_tag
                        else "Missing Title"
                    )

                    for link in soup.find_all(
                        "a",
                        href=True,
                    ):
                        href = urljoin(url, link["href"])
                        href = href.split("#")[0]

                        if (
                            self.is_internal(href)
                            and href not in self.visited
                            and href not in self.to_visit
                        ):
                            self.to_visit.append(href)

                self.results.append(
                    {
                        "url": url,
                        "status_code": status,
                        "title": title,
                    }
                )

                print(f"[{status}] Crawled {url}")

            except Exception as exc:
                self.results.append(
                    {
                        "url": url,
                        "status_code": "Error",
                        "title": str(exc),
                    }
                )

        return pd.DataFrame(self.results)


if __name__ == "__main__":
    crawler = SEOCrawler(
        START_URL,
        max_pages=MAX_PAGES,
    )

    results = crawler.crawl()

    output_file = "seo_site_audit.csv"
    results.to_csv(output_file, index=False)

    print(
        f"\nSEO crawl complete. "
        f"Scanned {len(results)} pages."
    )
    print(f"Report saved as {output_file}")
Return to Automation Hub
Related Articles

No other articles are categorized under this module layout yet.

Need Custom Orchestration?
Explore Runtime Documentation