Web Link Extractor: Quick Guide to Scrape URLs from Any Page

Choose based on performance needs, complexity of target pages, and how much control you need over requests and concurrency.

Core Features Your Extractor Should Support

• URL normalization (resolve relative URLs to absolute).
• Filtering by domain or path (include/exclude rules).
• Deduplication and canonicalization (remove fragments, sort query params if needed).
• Respect robots.txt and rate limits / politeness delays.
• Retry and error handling for transient network errors.
• Optionally render JavaScript for dynamic pages.
• Output formats: CSV, JSON, or direct insertion into a database or queue.
• Logging and metrics for monitoring extraction runs.

Practical Example: Python Script with Requests + BeautifulSoup

Below is a practical, extendable Python script that extracts links from a list of pages, normalizes them, filters by domain, and writes results to CSV. It does not render JavaScript—use Playwright or Selenium for that case.

#!/usr/bin/env python3 """ Simple web link extractor: - reads URLs from input.txt (one URL per line) - fetches each page - extracts <a href=""> links - normalizes & filters links (same domain optional) - writes results to urls.csv """ import csv import sys import time from urllib.parse import urljoin, urlparse, urldefrag import requests from bs4 import BeautifulSoup INPUT_FILE = "input.txt" OUTPUT_FILE = "urls.csv" USER_AGENT = "web-link-extractor/1.0 (+https://example.com)" REQUEST_TIMEOUT = 10 DELAY_BETWEEN_REQUESTS = 1.0  # seconds HEADERS = {"User-Agent": USER_AGENT} MAX_RETRIES = 2 def normalize_url(base, link):     if not link:         return None     link = link.strip()     # skip javascript:, mailto:, tel:, and fragments-only     if link.startswith(("javascript:", "mailto:", "tel:", "data:")):         return None     # resolve relative URLs     abs_url = urljoin(base, link)     # remove fragment     abs_url, _ = urldefrag(abs_url)     parsed = urlparse(abs_url)     # keep only http/https     if parsed.scheme not in ("http", "https"):         return None     # normalize: lowercase scheme & host     normalized = parsed._replace(scheme=parsed.scheme.lower(), netloc=parsed.netloc.lower()).geturl()     return normalized def fetch(url):     tries = 0     while tries <= MAX_RETRIES:         try:             r = requests.get(url, headers=HEADERS, timeout=REQUEST_TIMEOUT)             r.raise_for_status()             return r.text         except requests.RequestException as e:             tries += 1             if tries > MAX_RETRIES:                 print(f"ERROR: Failed to fetch {url}: {e}", file=sys.stderr)                 return None             time.sleep(1) def extract_links(base_url, html):     soup = BeautifulSoup(html, "html.parser")     links = set()     for a in soup.find_all("a", href=True):         norm = normalize_url(base_url, a["href"])         if norm:             links.add(norm)     return links def main(only_same_domain=True):     with open(INPUT_FILE, "r", encoding="utf-8") as f:         seeds = [line.strip() for line in f if line.strip()]     rows = []     seen = set()     for seed in seeds:         print(f"Fetching: {seed}")         html = fetch(seed)         if not html:             continue         links = extract_links(seed, html)         seed_domain = urlparse(seed).netloc.lower()         for link in links:             if only_same_domain and urlparse(link).netloc.lower() != seed_domain:                 continue             if link in seen:                 continue             seen.add(link)             rows.append({"source": seed, "link": link})         time.sleep(DELAY_BETWEEN_REQUESTS)     # write CSV     with open(OUTPUT_FILE, "w", newline="", encoding="utf-8") as csvf:         writer = csv.DictWriter(csvf, fieldnames=("source", "link"))         writer.writeheader()         writer.writerows(rows)     print(f"Wrote {len(rows)} links to {OUTPUT_FILE}") if __name__ == "__main__":     main() 

Handling JavaScript-Rendered Pages

If target sites render links via JavaScript (SPAs, infinite scroll, client-side routing), use Playwright or Puppeteer to load pages and extract DOM after rendering. Example approach:

• Start a headless browser session.
• Navigate to the URL and wait for network idle or specific selectors.
• Optionally simulate scrolling to load lazy content.
• Extract link hrefs from the rendered DOM.

Playwright has Python and Node clients and is much faster and more modern than Selenium for many tasks.

Scaling: From Script to Pipeline

For larger workloads:

• Use a job queue (Redis + RQ, Celery, or cloud queues) to schedule fetch tasks.
• Run concurrent workers with controlled concurrency and per-domain throttling.
• Store results in a database (Postgres, Elasticsearch) for search and deduplication.
• Add incremental runs by tracking seen URLs and last-crawled timestamps.
• Instrument with Prometheus/Grafana or cloud monitoring for throughput/error rates.

Ethical and Legal Considerations

• Respect robots.txt and site terms of service. Not all sites allow automated scraping.
• Rate-limit requests and avoid overloading servers. Use sensible delays and concurrency caps.
• Identify your bot with a clear User-Agent and contact info when appropriate.
• Avoid collecting or exposing private or personal data without consent.

Troubleshooting Common Issues

• Missing links: site uses JavaScript → use headless rendering.
• Duplicate or noisy URLs: implement normalization and canonical checks.
• Slow runs: increase concurrency carefully, enable HTTP keep-alive, or use a CDN/proxy.
• IP blocks: use exponential backoff, rotate proxies responsibly, or contact site owners for bulk access.

Conclusion

Automating URL collection with a web link extractor script streamlines many web tasks from SEO audits to data collection. Start with a simple parser for static pages, switch to headless browsers for dynamic sites, and scale with queuing and storage systems as needs grow. Always follow ethical guidelines, respect site policies, and monitor extraction jobs to keep the pipeline reliable.