Auto-Ping Search Engines to Re-Crawl Your Sitemap

Schedule

Setup

1. 1

   Locate your sitemap URL

   Confirm your sitemap is publicly accessible. It is typically at https://yourdomain.com/sitemap.xml or listed in your robots.txt under the Sitemap: directive. Open the URL in a browser to verify it returns valid XML before proceeding.

2. 2

   Create a heartbeat monitor in CronJobPro

   In your CronJobPro dashboard, create a new Heartbeat monitor with a period of 7 days and a grace period of 12 hours to match the weekly schedule. Copy the unique ping URL you receive, which will look like https://cronjobpro.com/ping/<token>. You will embed this URL in the script.

3. 3

   Deploy the ping script to your server

   Save the script below to your server, for example at /usr/local/bin/ping-sitemaps.sh. Replace SITEMAP_URL and HEARTBEAT_URL with your actual values. Make it executable with: chmod +x /usr/local/bin/ping-sitemaps.sh

4. 4

   Register the cron job

   Open your crontab with crontab -e and add the schedule line shown below. This runs the script every Monday at 6 AM in the server timezone. Ensure the server timezone matches your expectations by checking with timedatectl or date.

5. 5

   Verify the first run

   After the first scheduled execution, check the CronJobPro heartbeat monitor dashboard to confirm the ping arrived. Also inspect /tmp/ping-sitemaps.log on your server for the HTTP response codes from Google and Bing, which should be 200 or 202 for a successful submission.

The script

bash

#!/usr/bin/env bash
# ping-sitemaps.sh
# Notifies Google and Bing that your sitemap should be re-crawled.
# Dependencies: curl (pre-installed on most Linux systems)

set -euo pipefail

SITEMAP_URL="https://yourdomain.com/sitemap.xml"
HEARTBEAT_URL="https://cronjobpro.com/ping/YOUR_TOKEN_HERE"
LOG_FILE="/tmp/ping-sitemaps.log"

log() {
  echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] $*" | tee -a "$LOG_FILE"
}

ping_engine() {
  local name="$1"
  local endpoint="$2"
  local full_url="${endpoint}$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1],safe=''))"
"$SITEMAP_URL")"
  local http_code
  http_code=$(curl --silent --output /dev/null --write-out "%{http_code}" \
    --max-time 30 "$full_url")
  log "$name ping returned HTTP $http_code"
  if [[ "$http_code" != "2"* ]]; then
    log "WARNING: $name returned non-2xx status $http_code"
    return 1
  fi
}

main() {
  log "Starting sitemap ping for: $SITEMAP_URL"

  # Google: public ping endpoint (no authentication required)
  ping_engine "Google" "https://www.google.com/ping?sitemap="

  # Bing (also covers Yahoo, DuckDuckGo via Bing index)
  ping_engine "Bing" "https://www.bing.com/ping?sitemap="

  log "All pings completed successfully."

  # Signal success to CronJobPro heartbeat monitor
  curl --silent --output /dev/null --max-time 10 "$HEARTBEAT_URL"
  log "Heartbeat pinged: $HEARTBEAT_URL"
}

main

Frequently asked questions

More recipes