Sync Data Between Two APIs on a Schedule

Schedule

Setup

1. 1

   Create a heartbeat monitor in CronJobPro

   In your CronJobPro dashboard, add a new Heartbeat monitor. Set the period to 1 hour and the grace period to 5 minutes. Copy the unique ping URL you receive, for example https://cronjobpro.com/ping/abc123. This URL is what your script will call on a successful run.

2. 2

   Store credentials and the ping URL as environment variables

   On your server, add SOURCE_API_KEY, DEST_API_KEY, and HEARTBEAT_URL to a secure environment file such as /etc/etl-sync.env or your application's .env file. Never hardcode secrets in the script itself. Load them with: set -a; source /etc/etl-sync.env; set +a

3. 3

   Save the sync script to your server

   Copy the script below to a path such as /usr/local/bin/api-sync.sh and make it executable with: chmod +x /usr/local/bin/api-sync.sh. Review the SOURCE_URL and DEST_URL variables at the top and replace them with your actual API endpoints.

4. 4

   Add the cron entry

   Open your crontab with crontab -e and add the following line, replacing the path if needed: 0 * * * * /bin/bash -c 'set -a; source /etc/etl-sync.env; set +a; /usr/local/bin/api-sync.sh >> /var/log/api-sync.log 2>&1' This runs the job every hour and appends all output to a log file for debugging.

5. 5

   Test the first run manually

   Run the script once by hand: source /etc/etl-sync.env && /usr/local/bin/api-sync.sh. Check the output for errors and verify that your destination API received the data. Also confirm that a ping arrived in your CronJobPro heartbeat monitor dashboard, showing the monitor as healthy.

The script

bash

#!/usr/bin/env bash
set -euo pipefail

# --- Configuration (loaded from environment) ---
SOURCE_URL="https://source-api.example.com/v1/records"
DEST_URL="https://dest-api.example.com/v1/ingest"
SOURCE_API_KEY="${SOURCE_API_KEY}"
DEST_API_KEY="${DEST_API_KEY}"
HEARTBEAT_URL="${HEARTBEAT_URL}"  # https://cronjobpro.com/ping/<token>

LOG_PREFIX="[api-sync $(date -u '+%Y-%m-%dT%H:%M:%SZ')]"

cleanup() {
  local exit_code=$?
  if [ $exit_code -ne 0 ]; then
    echo "$LOG_PREFIX ERROR: script exited with code $exit_code"
    # Report failure to CronJobPro so the heartbeat is not counted as a miss
    curl -fsS --retry 3 "${HEARTBEAT_URL}/fail" -o /dev/null || true
    # Also report the exact exit code
    curl -fsS --retry 3 "${HEARTBEAT_URL}/exitcode/${exit_code}" -o /dev/null || true
  fi
}
trap cleanup EXIT

echo "$LOG_PREFIX Starting ETL sync"

# --- Step 1: Fetch data from source API ---
echo "$LOG_PREFIX Fetching from source..."
RAW_RESPONSE=$(
  curl -fsS --retry 3 --max-time 30 \
    -H "Authorization: Bearer ${SOURCE_API_KEY}" \
    -H "Accept: application/json" \
    "${SOURCE_URL}"
)

# Validate that we got a non-empty JSON array or object
if [ -z "$RAW_RESPONSE" ]; then
  echo "$LOG_PREFIX ERROR: empty response from source API"
  exit 1
fi

echo "$LOG_PREFIX Received $(echo "$RAW_RESPONSE" | wc -c) bytes from source"

# --- Step 2: Transform (optional — pipe through jq if needed) ---
# Example: extract only active records and rename a field
TRANSFORMED=$(
  echo "$RAW_RESPONSE" | jq '[.[] | select(.status == "active") | {id: .id, name: .full_name, email: .email_address}]'
)

RECORD_COUNT=$(echo "$TRANSFORMED" | jq 'length')
echo "$LOG_PREFIX Transformed ${RECORD_COUNT} records"

if [ "$RECORD_COUNT" -eq 0 ]; then
  echo "$LOG_PREFIX No records to sync, exiting cleanly"
  # Still ping success — zero records is a valid outcome
  curl -fsS --retry 3 "${HEARTBEAT_URL}" -o /dev/null
  exit 0
fi

# --- Step 3: Push data to destination API ---
echo "$LOG_PREFIX Pushing to destination..."
DEST_RESPONSE=$(
  curl -fsS --retry 3 --max-time 60 \
    -X POST \
    -H "Authorization: Bearer ${DEST_API_KEY}" \
    -H "Content-Type: application/json" \
    -d "$TRANSFORMED" \
    "${DEST_URL}"
)

# Check that the destination acknowledged the payload
ACCEPTED=$(echo "$DEST_RESPONSE" | jq -r '.status // "unknown"')
echo "$LOG_PREFIX Destination responded: ${ACCEPTED}"

if [ "$ACCEPTED" != "ok" ] && [ "$ACCEPTED" != "accepted" ]; then
  echo "$LOG_PREFIX WARNING: unexpected destination status '${ACCEPTED}'"
  # Treat unexpected statuses as failure
  exit 2
fi

echo "$LOG_PREFIX ETL sync completed successfully (${RECORD_COUNT} records)"

# --- Step 4: Ping CronJobPro heartbeat on success ---
curl -fsS --retry 3 "${HEARTBEAT_URL}" -o /dev/null
echo "$LOG_PREFIX Heartbeat pinged"

Frequently asked questions

More recipes