Troubleshooting

Common issues and solutions for MESH deployments.

Connection issues

Can't Connect to Control plane

Symptoms: - meshcli up fails with connection error - "Unable to reach control plane" message - Timeout errors

Solutions:

1. Verify control plane URL

   curl https://mesh.yourdomain.com/health
   # Should return: {"status":"ok"}
   

2. Check DNS resolution

   nslookup mesh.yourdomain.com
   dig mesh.yourdomain.com
   

3. Check firewall

   # Test HTTPS connectivity
   telnet mesh.yourdomain.com 443
   
   # Check if port is open
   nc -zv mesh.yourdomain.com 443
   

4. Verify control plane is running

   docker ps | grep headscale
   docker logs headscale
   

5. Check reverse proxy

   # Nginx
   sudo nginx -t
   sudo systemctl status nginx
   
   # Caddy
   sudo systemctl status caddy
   

6. Try different network

7. Switch from WiFi to mobile data (or vice versa)
8. Some networks block VPN traffic

Daemon Won't Start

Symptoms: - tailscaled-amnezia fails to start - Socket errors - Permission denied errors

Solutions:

1. Check if already running

   ps aux | grep tailscaled
   sudo pkill tailscaled-amnezia
   

2. Check socket permissions

   sudo rm -rf /var/run/mesh/tailscaled.sock
   sudo mkdir -p /var/run/mesh
   sudo chmod 755 /var/run/mesh
   

3. Check state directory

   sudo mkdir -p /var/lib/mesh
   sudo chmod 755 /var/lib/mesh
   

4. Run with verbose logging

   sudo tailscaled-amnezia \
     --socket=/var/run/mesh/tailscaled.sock \
     --state=/var/lib/mesh/tailscaled.state \
     --statedir=/var/lib/mesh \
     --verbose=2
   

5. Check systemd service (if using)

   sudo systemctl status mesh
   sudo journalctl -u mesh -f
   

Authentication Fails

Symptoms: - "Invalid auth key" error - "Auth key expired" error - Login prompt doesn't appear

Solutions:

1. Verify pre-auth key

   # On control plane
   docker exec headscale headscale preauthkeys list
   

2. Generate new key

   docker exec headscale headscale preauthkeys create --expiration 24h --reusable
   

3. Check key expiration

4. Pre-auth keys expire after the specified time

5. Generate a new key if expired

6. Try interactive authentication

   # Without --authkey
   sudo meshcli up --login-server=https://mesh.yourdomain.com
   # Follow the URL to authenticate
   

No Mesh IP Assigned

Symptoms: - meshcli status doesn't show a mesh IP address - Node appears offline - Can't communicate with other nodes

Solutions:

1. Verify pre-auth key is valid

   # On control plane, check if key is valid and not expired
   docker exec headscale headscale preauthkeys list
   

2. Check control plane logs

   docker logs headscale
   # Look for registration errors
   

3. Ensure user exists

   # List users
   docker exec headscale headscale users list
   
   # Create user if missing
   docker exec headscale headscale users create default
   

4. Try reconnecting

   sudo ./meshcli down
   sudo ./meshcli up --login-server=https://mesh.yourdomain.com --authkey=YOUR_KEY
   

5. Check node registration

   # On control plane
   docker exec headscale headscale nodes list
   # Verify your node appears in the list
   

Connectivity issues

No Peer-to-Peer Connection (Using DERP Relay)

Symptoms: - meshcli status shows DERP address in CurAddr - Slow connection speeds - High latency

Solutions:

1. Check NAT type

   sudo meshcli netcheck
   

2. Check UDP connectivity

   # Test if UDP is blocked
   nc -u -v mesh.yourdomain.com 3478
   

3. Check firewall rules

   # Linux
   sudo iptables -L -n -v
   sudo ufw status
   
   # Allow WireGuard port (usually random high port)
   sudo ufw allow 41641/udp
   

4. Enable UPnP/NAT-PMP on router

5. Check router settings

6. Enable UPnP or NAT-PMP for automatic port forwarding

7. Try different network

8. Some networks (corporate, public WiFi) block UDP

9. Try mobile data or different WiFi network

10. Force direct connection

    export TS_DEBUG_ALWAYS_USE_DERP=false
    sudo -E meshcli up --login-server=https://mesh.yourdomain.com
    

Peers Not Visible

Symptoms: - meshcli status --peers shows no peers - Devices don't appear in mesh - Can't ping peers

Solutions:

1. Verify both devices are connected

   sudo meshcli status
   # Should show "Running"
   

2. Check ACLs

   # On control plane
   docker exec headscale headscale nodes list
   cat config/acl.yaml
   

3. Verify same control plane

4. Ensure all devices connect to the same control plane URL

5. Check user assignment

   docker exec headscale headscale nodes list
   # Verify nodes are in correct user/group
   

6. Restart daemon

   sudo systemctl restart mesh
   # or
   sudo pkill tailscaled-amnezia
   sudo tailscaled-amnezia ...
   

Connection Drops Frequently

Symptoms: - Intermittent connectivity - Frequent reconnections - "Peer offline" messages

Solutions:

1. Check network stability

   ping -c 100 8.8.8.8
   # Look for packet loss
   

2. Disable power saving (mobile)

3. Android: Settings > Battery > Optimize battery usage > MESH > Don't optimize

4. Disable "Adaptive battery" for MESH app

5. Increase keepalive

   # Edit WireGuard config to add:
   PersistentKeepalive = 25
   

6. Check DERP relay stability

   sudo meshcli netcheck
   # Check DERP latencies
   

7. Monitor logs

   sudo journalctl -u mesh -f
   # Look for errors or warnings
   

ADB Issues

ADB Connection Fails

Symptoms: - adb connect fails - "Connection refused" error - "Unable to connect" error

Solutions:

1. Verify device is online

   sudo meshcli status --peers
   ping 100.64.X.X
   

2. Check ADB port

   # On Android device (via USB)
   adb shell getprop service.adb.tcp.port
   # Should return 5555
   

3. Enable ADB-over-WiFi

   # Via USB
   adb tcpip 5555
   adb connect 100.64.X.X:5555
   

4. Restart ADB

   adb kill-server
   adb start-server
   adb connect 100.64.X.X:5555
   

5. Check firewall on Android

6. Some security apps block ADB

7. Temporarily disable and test

8. Verify ADB is enabled in MESH app

9. Open MESH app
10. Settings > ADB-over-WiFi > Enabled

ADB Disconnects Frequently

Symptoms: - ADB connection drops during operations - "device offline" errors - Need to reconnect frequently

Solutions:

1. Disable power saving

2. Settings > Battery > MESH > Don't optimize

3. Keep MESH app in foreground

4. Don't minimize the app during ADB operations

5. Use persistent connection

   # Keep pinging to maintain connection
   ping 100.64.X.X &
   adb connect 100.64.X.X:5555
   

6. Check network stability

7. Ensure stable WiFi/mobile connection
8. Avoid switching networks during operation

Performance issues

Slow Transfer Speeds

Symptoms: - Slow file transfers - High latency - Timeouts during large operations

Solutions:

1. Check connection type

   sudo meshcli status --json | jq '.Peer[] | {name: .HostName, addr: .CurAddr}'
   # Direct connection is faster than DERP relay
   

2. Disable AmneziaWG obfuscation

   # If not needed for censorship resistance
   sudo rm /etc/mesh/amneziawg.conf
   sudo systemctl restart mesh
   

3. Use lighter obfuscation

4. Switch from Heavy to Balanced or Light preset

5. See AmneziaWG Configuration

6. Check network bandwidth

   # Test bandwidth
   iperf3 -s  # On one peer
   iperf3 -c 100.64.X.X  # On other peer
   

7. Compress data

   # Compress before transfer
   adb shell tar -czf /sdcard/data.tar.gz /data/local/tmp/
   adb pull /sdcard/data.tar.gz
   

High CPU Usage

Symptoms: - High CPU usage by tailscaled-amnezia - Device heating up - Battery drain (mobile)

Solutions:

1. Disable obfuscation
2. AmneziaWG adds CPU overhead

3. Disable if not needed

4. Reduce connection checks

   # Increase check interval
   sudo meshcli set --netcheck-interval=10m
   

5. Limit peer count

6. Use ACLs to limit visible peers

7. Only connect to necessary devices

8. Check for loops

   # Look for routing loops
   sudo meshcli routes
   

Control plane Issues

Headscale Won't Start

Symptoms: - Docker container exits immediately - "Address already in use" errors - Database errors

Solutions:

1. Check logs

   docker logs headscale
   

2. Check port conflicts

   sudo lsof -i :8080
   sudo lsof -i :9090
   sudo lsof -i :3478
   

3. Verify config syntax

   cat config/config.yaml
   # Check for YAML syntax errors
   

4. Check database

   ls -la data/db.sqlite
   sudo chown -R 1000:1000 data/
   

5. Reset database (WARNING: deletes all data)

   docker-compose down
   rm data/db.sqlite
   docker-compose up -d
   

Nodes Not Registering

Symptoms: - Nodes don't appear in headscale nodes list - Authentication succeeds but node not visible - "Node not found" errors

Solutions:

1. Check node list

   docker exec headscale headscale nodes list
   

2. Check user exists

   docker exec headscale headscale users list
   

3. Create user if missing

   docker exec headscale headscale users create analyst1
   

4. Check ACLs

   cat config/acl.yaml
   # Verify ACLs allow the connection
   

5. Register manually

   # Get node key from client logs
   docker exec headscale headscale nodes register --user analyst1 --key nodekey:abc123
   

DERP Relay Not Working

Symptoms: - Peers can't connect even via relay - "No DERP home" errors - All connections fail

Solutions:

1. Check STUN port

   sudo netstat -ulnp | grep 3478
   docker logs headscale | grep STUN
   

2. Test STUN

   sudo apt install stun-client
   stun mesh.yourdomain.com 3478
   

3. Check DERP config

   cat config/config.yaml | grep -A 20 "derp:"
   

4. Verify DERP is enabled

   derp:
     server:
       enabled: true
       stun_listen_addr: "0.0.0.0:3478"
   

5. Check firewall

   sudo ufw allow 3478/udp
   

Android App Issues

App Crashes

Symptoms: - App closes unexpectedly - "MESH has stopped" error - Can't open app

Solutions:

1. Clear app data
2. Settings > Apps > MESH > Storage > Clear Data

3. Reconnect to mesh

4. Reinstall app

   adb uninstall com.barghest.mesh
   adb install mesh-android.apk
   

5. Check logs

   adb logcat | grep MESH
   adb logcat | grep AndroidRuntime
   

6. Check Android version

7. MESH requires Android 8.0+
8. Update Android if possible

VPN Permission Denied

Symptoms: - "VPN permission required" error - Can't connect to mesh - Permission dialogue doesn't appear

Solutions:

1. Grant permission manually
2. Settings > Apps > MESH > Permissions

3. Enable VPN permission

4. Check for conflicting VPN

5. Disconnect other VPN apps

6. Only one VPN can be active at a time

7. Reinstall app

8. Uninstall and reinstall to reset permissions

Diagnostic commands

Collect Logs

# Analyst client logs
sudo journalctl -u mesh --since "1 hour ago" > mesh-logs.txt

# Control plane logs
docker logs headscale > headscale-logs.txt

# Android logs
adb logcat -d > android-logs.txt

Generate Bug Report

# Analyst client
sudo meshcli bugreport --diagnose > mesh-bugreport.txt

# Android
adb bugreport bugreport.zip

Network Diagnostics

# Check NAT type and DERP latencies
sudo meshcli netcheck

# Trace route to peer
sudo meshcli ping 100.64.X.X --verbose

# Check WireGuard interface
ip addr show tun0
sudo wg show

Connection Diagnostics

# Detailed status
sudo meshcli status --json | jq

# Check peer connectivity
for peer in $(sudo meshcli status --json | jq -r '.Peer[].TailscaleIPs[0]'); do
    echo "Pinging $peer..."
    sudo meshcli ping $peer --c 3
done

# Monitor connection changes
watch -n 1 'sudo meshcli status --peers'

Getting Help

If you can't resolve your issue:

1. Check GitHub Issues: github.com/BARGHEST-ngo/mesh/issues
2. Create Bug Report: Include logs, config, and steps to reproduce
3. Contact BARGHEST: barghest.asia

Next steps

• User guide - Learn forensic workflows
• CLI reference - Complete command documentation
• Architecture - Understand how MESH works