Cloudflare Tunnel Setup & Error 1016 Recovery Runbook

1. What Error 1016 Means

Cloudflare Error 1016 is an origin DNS error. It occurs when:

• The Cloudflare Tunnel daemon (cloudflared) on the Oracle server is not running or crashed.
• The CNAME record pointing your proxy hostname to the tunnel was deleted or never created.
• The tunnel was deleted from the Cloudflare dashboard while the connection was still live.
• The Oracle server cannot reach Cloudflare's edge (egress firewall blocks outbound 443).

2. Quick Diagnostic Checks

Run these from any machine with internet access to confirm the failure mode before touching the Oracle server.

# 1. Confirm you're getting Error 1016 (not a different error)
curl -sv https://YOUR-HOSTNAME.tunevault.app/health 2>&1 | head -40

# Expected when tunnel is working:
# {"status":"ok","proxy":"oracle-proxy","version":"..."}

# 2. Check DNS — CNAME should point to a .cfargotunnel.com address
dig YOUR-HOSTNAME.tunevault.app CNAME

# Good: CNAME → xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.cfargotunnel.com
# Bad:  NXDOMAIN or no CNAME record (tunnel was deleted or not created)

# 3. Resolve the cfargotunnel.com endpoint to confirm Cloudflare has the tunnel
dig xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.cfargotunnel.com A

3. Verify Tunnel Daemon on the Oracle Server

SSH to the Oracle server where the proxy agent is installed. All commands below run as the user that installed the agent (usually oracle or root).

# Check if cloudflared service is running
sudo systemctl status cloudflared

# If inactive or failed — restart it
sudo systemctl restart cloudflared

# Check logs for connection errors
sudo journalctl -u cloudflared -n 50 --no-pager

# Check if oracle-proxy.py service is also running
sudo systemctl status oracle-proxy

# If proxy service is down, restart it too
sudo systemctl restart oracle-proxy

# Verify proxy is listening on its local port (default 8080)
ss -tlnp | grep 8080
# or:
netstat -tlnp | grep 8080

After restarting both services, wait 30 seconds and re-test the /health endpoint. Cloudflare reconnection typically takes 5–15 seconds.

Common daemon log errors and their causes

• failed to connect to tunnel: websocket: bad handshake — outbound port 443 is blocked. Go to Step 5.
• certificate verify failed — system time is wrong. Run timedatectl and sync NTP.
• error authenticating tunnel: Invalid token — tunnel credentials were revoked. Go to Step 6.
• connection refused 127.0.0.1:8080 — oracle-proxy.py is not running. Check systemctl status oracle-proxy.

# Check system time (clock skew breaks TLS)
timedatectl status

# Sync if needed (RHEL/OL)
sudo chronyc makestep

# Sync if needed (older systems)
sudo ntpdate -u pool.ntp.org

4. DNS and CNAME Verification

If the daemon is running but you still get 1016, the CNAME record may be stale or missing.

# From any machine — verify CNAME points to cfargotunnel.com
dig +short YOUR-HOSTNAME.tunevault.app CNAME
# Expected: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.cfargotunnel.com.

# If CNAME is wrong or missing, get the correct tunnel UUID
# On the Oracle server:
cloudflared tunnel list
# Note the ID column — this is the UUID that should appear in the CNAME

# Verify the tunnel is connected (shows active connections)
cloudflared tunnel info YOUR-TUNNEL-NAME

5. Firewall and Outbound Connectivity

Cloudflare tunnels require outbound HTTPS (port 443) from the Oracle server to Cloudflare's edge. No inbound port opening is needed — this is the security advantage of tunnel mode.

# Test outbound connectivity from the Oracle server
curl -sv https://cloudflare-quic.com/ 2>&1 | head -10
# Should connect. Failure = firewall blocking outbound 443.

# Alternative test: connect to Cloudflare tunnel endpoint
curl -sv https://region1.argotunnel.com/ 2>&1 | head -10

# Check local iptables for outbound blocks
sudo iptables -L OUTPUT -n | grep -E "DROP|REJECT"

# On RHEL/OL with firewalld
sudo firewall-cmd --list-all

If outbound 443 is blocked by a corporate firewall, contact your network team. The Oracle server needs outbound HTTPS to *.argotunnel.com and *.cloudflare.com.

6. Re-install or Recreate the Tunnel

If the tunnel was deleted from Cloudflare's dashboard, or the credentials file is missing, you must re-provision.

# Option A: Re-run TuneVault's one-line installer
# Get a fresh install token from TuneVault Settings → Connections → Install Agent
curl -sSL https://tunevault.app/install.sh | bash -s -- --token YOUR_INSTALL_TOKEN

# Option B: Manual tunnel recreation
# 1. Log in to Cloudflare
cloudflared tunnel login

# 2. Delete the old broken tunnel (get UUID from 'cloudflared tunnel list')
cloudflared tunnel delete OLD-TUNNEL-UUID

# 3. Create a new tunnel
cloudflared tunnel create oracle-proxy-YOURSERVER

# 4. Note the new tunnel UUID from the output
# 5. Update CNAME in Cloudflare DNS to point to NEW-UUID.cfargotunnel.com
# 6. Start the tunnel
cloudflared tunnel run oracle-proxy-YOURSERVER

7. Fallback: Switch to Direct TCP Mode

If the Cloudflare tunnel cannot be repaired (network policy blocks outbound, Cloudflare API issues, etc.), switch the connection to Direct TCP. This requires the Oracle listener port to be reachable from TuneVault's servers.

1. Go to Settings → Connections in TuneVault.
2. Edit the affected connection.
3. Change Connection Type from Proxy Agent to Direct TCP.
4. Enter the Oracle server's public IP (or VPN-accessible private IP) and listener port (default 1521).
5. Save and run a new health check to verify connectivity.

8. Prevention

# Enable autostart for both services
sudo systemctl enable cloudflared
sudo systemctl enable oracle-proxy

# Verify both are set to start on boot
sudo systemctl is-enabled cloudflared oracle-proxy
# Expected: enabled enabled

# Test the health endpoint after any server restart
curl -sf https://YOUR-HOSTNAME.tunevault.app/health | python3 -m json.tool