# Troubleshooting

All commands are for Linux. Run `docker compose` commands from the `deploy/` directory.

## Reading the logs

```bash
docker compose ps                         # are all containers up?
docker compose logs -f --tail=200 api     # or watcher, sweeper, payouts, caddy
docker compose logs --since=1h watcher | grep -i -E 'warn|error'
```

Each log line is JSON with a `msg` and a `level`. Internal API errors are logged with the request path and a
`request_id`; the client only sees `internal error`.

## Install and Docker

- **Containers can't resolve DNS, or `npm ci` hangs during `docker build`.** This is common with Tailscale or a
  local DNS stub (`nameserver 100.100.100.100` or `127.0.0.53`), which containers can't reach. Give Docker
  real resolvers:

  ```bash
  echo '{ "dns": ["1.1.1.1", "9.9.9.9"] }' | sudo tee /etc/docker/daemon.json
  sudo systemctl restart docker
  ```

  For a one-off build, pass `--network host` to `docker build` instead.
- **npm hangs on hosts with broken IPv6:** `export NODE_OPTIONS="--dns-result-order=ipv4first --no-network-family-autoselection"`,
  or pass it to `docker build` as `--build-arg NODE_OPTIONS=...`.
- **`go mod download` fails with permission errors under `~/go/pkg/mod`:** point the module cache somewhere
  writable, e.g. `go env -w GOMODCACHE=/var/tmp/$USER-gomodcache`.
- **Published ports don't answer, but `docker compose exec caddy wget -qO- http://localhost/healthz` works:**
  the firewall is blocking host-to-container forwarding. Check `sudo iptables -L DOCKER-USER` and whether ufw
  or Tailscale rules drop forwarded traffic.
- **`permission denied` on the Docker socket:** run `sudo usermod -aG docker $USER`, then log out and back in.
- **Caddy can't get a certificate:** the DNS name must point at this server, and ports 80 and 443 must be
  reachable from the internet. Check `docker compose logs caddy`.

## Sign-in

- **"Too many attempts":** login is limited to 10 attempts a minute per IP and per email. Wait a minute.
- **Two-factor codes rejected:** check the server clock (`timedatectl`; it should say "System clock
  synchronized: yes"). Each code works once, so wait for the next code. After five wrong codes, checks are
  locked for 15 minutes.
- **Lost two-factor device:** there is no reset button, on purpose. Someone with server access can clear it
  from the database, after which the user signs in with their password and enrols again:

  ```bash
  docker compose exec postgres psql -U coldtill coldtill -c \
    "UPDATE users SET totp_enabled = false, totp_secret = NULL WHERE email = 'user@example.com'"
  ```

## Deposits

- **A deposit isn't showing up:**
  1. Check the network under **Nodes & networks**: the last processed block should be close to the chain head.
     If not, the nodes are failing or rate limiting you. Add your own node.
  2. Check that the token is enabled on that network, and that the customer used the right network.
  3. Look for `watcher` warnings mentioning the address or transaction.
- **A deposit is stuck "confirming":** it needs the network's confirmation count. On Bitcoin a block can take
  an hour or more.
- **"evm watcher catching up after downtime" warning:** the watcher was down for more than 10,000 blocks. It catches up
  block by block; that can take a while, but nothing is skipped.
- **A Bitcoin deposit was marked orphaned:** every node reported the transaction gone (replaced or dropped).
  The customer should pay again.

## Sweeps

- **Nothing is being swept:** sweeps run when an address holds more than the activation minimum, then on the
  amount, address-count or time triggers. Check the hot wallet has gas, and look for `sweeper` errors.
- **"insufficient funds" in the sweeper log:** top up the EVM or Solana hot wallet with native coin.
- **A wallet is left unswept after a sweep:** the token's transfer failed inside the contract (for example a
  paused or blocklisted token). The funds stay at the deposit address, still locked to your cold wallet.
- **A Solana sweep failed after 10 minutes of backoff:** see the error in the `sweeper` log; usually the
  hot wallet lacks SOL, or the RPC node is rate limiting.

## Payouts

- **Stuck in `pending`:** is the `payouts` container running and did it unlock the hot wallet? Check
  `docker compose logs payouts` for passphrase errors.
- **Stuck in `initiated`:** EVM payouts are re-signed with higher fees automatically. Check the hot wallet has
  enough native coin for the higher fee.
- **`needs-review`:** see [Payouts → Needs review](https://coldtill.com/docs/payouts.md#needs-review).
- **No approval email:** check the `COLDTILL_SMTP_*` settings and the `api` log. On testnet, codes are printed
  to the `api` log instead.

## Webhooks

- **All deliveries fail:** on mainnet the endpoint must be a public `https` URL; private addresses are
  refused. Redirects aren't followed. Use **Send test** to check.
- **Signature mismatch:** verify against the raw body bytes, before JSON parsing. Frameworks that parse the
  body first (for example Express with `express.json()`) change it.
- **Timestamp outside the window:** your server clock is off. Enable time sync with `sudo timedatectl set-ntp true`.

## Auto-convert

- **No swap options at checkout:** auto-convert is off for the project, the install is in testnet mode, or
  the provider couldn't quote that amount (very small amounts often can't be swapped).
- **A conversion shows `mismatch`:** read the reason in **Conversions**. The payment wasn't credited. Check the
  delivery transaction on a block explorer and contact the provider if funds didn't arrive.

## Database

```bash
docker compose exec postgres psql -U coldtill coldtill
```

Migrations run automatically at start-up. If the `api` container exits with a migration error, restore the
last backup before retrying, and report the error.

---

Source: https://coldtill.com/docs/troubleshooting/ · All Coldtill docs: https://coldtill.com/llms.txt