Troubleshooting

Start with the logs:

journalctl -u rchat-server -n 100 --no-pager

For more detail, add --log-level debug to ExecStart temporarily. Debug lines never contain identifiers or, unless you enable log_client_ips, client addresses.

After any deployment change, connect once with the iOS app to check it.

install.sh failed

install.sh stops at the first problem with a message starting install.sh: and one of these exit codes. It never leaves a half-installed binary.

Exit code	Message (start)	Fix
1	unsupported platform: ...	There is no release binary for this system and CPU. See the supported platforms, or build from source.
1	unknown option / --version needs a value / base URL must start with https://	Fix the command line. Pass options after sh -s --.
2	cannot download ... / neither curl nor fetch found / no sha256 tool found / invalid version	Check the host can reach https://assets.r-chat.net, that the version you pinned exists, and that curl (or fetch) and sha256sum, shasum or sha256 are installed.
3	checksum mismatch for ...; nothing was installed	The download was damaged or altered. Try again; if it keeps failing, download by hand and verify it, or build from source.
4	failed: ...	An install step failed: creating a directory, installing the file, or creating the rchat user. Run it as root. On FreeBSD run it as root without sudo.

If install.sh says kept existing ..., a service file was already there and was left alone. That is intended: it never overwrites a config, key or service file.

The server will not start

The relay validates everything before serving and exits with an Error: line naming the problem.

Message (start)	Fix
cannot read config file "..."	The file is missing or unreadable by the rchat user. Install it 0640, group rchat.
invalid config file "..."	TOML syntax error, a wrong type, or an unknown key (unknown field ...). The rest of the message says where. Remove obsolete keys such as randomization_strength or udp_bind.
websocket_path still contains the template placeholder "CHANGE-ME"	Set websocket_path to a long random path, and use the same path in clients’ server URL.
no TLS configured	Add [server.acme] or [server.tls]. See TLS and certificates.
both [server.tls] (static certificate) and [server.acme] (automatic certificates) are configured	Keep exactly one.
[server.tls] `min_version` is not a setting or `tls13_only`	Delete the key. TLS 1.3 is always on.
[server.acme] ... (domains, contact, cache_dir)	See the certificate troubleshooting table.
ACME cache_dir ... is a symlink	Point cache_dir at the real directory.
failed to load TLS certificate chain / failed to load TLS private key / TLS key ... does not match certificate ...	Wrong path, not PEM, unreadable by rchat, or a mismatched pair.
--no-tls with non-loopback bind address	--no-tls is for local development only: bind 127.0.0.1, or configure TLS.
--no-tls requires [server] dev_origin	Add dev_origin = "ws://127.0.0.1:<port>" under [server] (development only).
relay identity key missing at …	Run rchat-server --config … --init-relay-identity once, as the service user. See Relay identity.
A relay identity key owned by another user, or with loose permissions	Create or chown it as the service user, mode 0600.
message_queue_persist_path was removed	Delete the key. Queued messages are memory-only by default; see Offline queue.
trusted_proxy was removed	Delete it. Behind a TCP front end set proxy_protocol = true; see Behind a TCP front end.
proxy_protocol errors	It requires max_connections_per_ip = 0, max_connections_per_ipv6_48 = 0 and a loopback bind_addr.
Offline queue persistence errors (key size, owner, mode, filesystem)	The key must be 32 bytes, mode 0600 or narrower, owned by the service user, on tmpfs, ramfs or mfs, on a different filesystem from persist_path.
connection_timeout_secs must be between 60 and 86400	Fix the value.
Metrics on a non-loopback address refused	Bind metrics to 127.0.0.1, or set [metrics] allow_public = true and firewall it.
max_connections_per_ip (...) must not exceed max_connections (...) and other limit messages	Fix the value. See Resource limits for the allowed ranges.
tls_handshake_timeout_secs must be between 1 and 120	Fix the value.
[server] websocket_path must start with '/' and be plain ASCII ...	Use a path such as /Zk3q... with URL-safe characters only.
[server.decoy] root must be an absolute path / root ... unusable / is not a directory	Fix root, or remove it to use the built-in page.
Permission denied / Address already in use	Use the shipped unit for CAP_NET_BIND_SERVICE; check nothing else listens on the port (ss -ltnp 'sport = :443').

The server runs but clients cannot connect

TLS handshakes fail right after a first ACME start. The log shows ACME: no certificate cached yet. Wait for ACME: certificate issued and deployed; TLS is ready. If ACME: certificate issuance failed, will retry ... repeats instead:

• check that dig +short <domain> returns this host’s address;
• check that port 443 reaches the relay directly, with nothing terminating TLS in front of it;
• read the error text from Let’s Encrypt at the end of the line.

The certificate is not trusted. You are still on staging = true. Set it to false and restart.

The connection opens but goes nowhere, or the client gets an HTML page. The client’s URL does not include the exact websocket_path. Upgrades to any other path get the decoy website. Use wss://<host><websocket_path>.

The app says “This server couldn’t prove its identity”. Something in front of the relay terminates TLS, so the client and the relay see different TLS sessions. Use a TCP passthrough or let the relay own port 443. On the relay, publishes over a different TLS session are refused with “connection proof does not match this TLS session (TLS must terminate at the relay)” (reason="key_auth").

The app says the server’s identity changed. The relay presented an identity key that does not match the one the app pinned, with no valid handover. If you replaced the key with --init-relay-identity after losing it, every client that pinned the old key refuses it. Use --rotate-relay-identity for planned changes. See Relay identity.

The app says “client too old; update rChat”. The app does not speak the relay’s protocol version (2.0.0 speaks protocol v3 only; 1.1.0 speaks v2). Update the relay and the app together.

The client is refused straight away.

• A cap may be reached: max_connections, or max_connections_per_ip (behind a TCP front end both per-address caps must be 0).
• The client must offer TLS 1.3. TLS 1.2 is refused.
• Behind a front end with proxy_protocol = true: every connection closed at once and reason="proxy_header" rising means the front end is not sending PROXY protocol v2 (nginx sends only v1). Both sides must agree.

Check from outside:

openssl s_client -connect chat.example.com:443 -tls1_3 </dev/null 2>&1 | grep -E '^ *Protocol|New, TLS'
curl -s -o /dev/null -w '%{http_code}\n' https://chat.example.com/   # 200: the decoy site answers

Certificate warnings while running

Message	Meaning
TLS certificate expires in N day(s)	With ACME, renewal has been failing; look for earlier ACME: warnings. With a static certificate, renew it and run systemctl reload rchat-server.
TLS certificate being served has EXPIRED	Clients reject it. Same causes as above.
ACME: rate-limited by Let's Encrypt	Wait for the retry time shown. Keep cache_dir persistent so restarts do not order again.
ACME: the cached ACME account key looks corrupt / the cached certificate looks corrupt	Stop, delete the named file in cache_dir, start again.
SIGHUP: TLS certificate reload failed, keeping the old one	The new files are unreadable, malformed or mismatched; the old certificate is still served.
ACME: certificate manager stopped; renewals will not run until the server restarts	Restart the service.

Messages and the offline queue

Symptom	Cause
Sender gets a RateLimited error and no acknowledgement	The recipient’s offline queue is full (1000 messages or 16 MiB), or the whole queue is. Messages are refused, never silently dropped.
Messages queued before a restart are missing	The queue is in memory by default, so a restart loses it. Turn on encrypted persistence if you need it to survive restarts.
Queued messages vanish after three days	Offline messages expire after ttl_secs (72 hours by default).
A contact’s messages after they rotated keys did not arrive	Messages held for their old keys are dropped. The sender’s app shows how many may not have arrived and offers Resend.
Connections drop every few minutes	The client sent nothing for connection_timeout_secs (rchat_rejections_total{reason="idle_timeout"}), did not answer a relay ping within 60 seconds of its being written, or its socket accepted nothing for 30 seconds (reason="writer_stall"). The app keeps the connection alive by itself; check for a front end or firewall that cuts idle TCP connections. See Keepalive and idle connections.

Messages in the app

What you see	Why, and what to do
A message stays Sent	The relay took it, but your contact’s phone has not stored it yet, usually because it is offline. It moves to Delivered when their phone stores it. The relay holds it for up to 72 hours by default.
Not confirmed	The relay’s acknowledgement did not arrive in time. The message may have arrived. Tap it to send it again (it may then arrive twice).
Not delivered with Resend	Your contact’s phone could not read it, usually after a reinstall or a new phone, and rChat could not resend it automatically. Tap Resend. To get automatic recovery next time, scan their code again in person. See Delivery and recovery.
Resent under a message	Your contact’s phone could not read it, and rChat sent it again automatically. Nothing to do.
“Scan their code again to turn on automatic message recovery.”	This contact has no device key pinned. Scan their code in person.
“Scan their code in person to turn on automatic message recovery.”	You pasted a code for a contact you already have. A paste never changes their device key; scan it instead.
“This code is damaged. Ask them to show it again.”	The code’s device-key signature does not check. Nothing was saved.

Startup warnings

Message	Meaning
circuit_ttl_secs is ignored: ...	Remove circuit_ttl_secs. Circuits end when an endpoint disconnects.
websocket_path is the default "/": ...	Set a long random websocket_path.
allow_persistent_key warning	The queue key is not on a memory filesystem, so it survives a reboot.
Metrics endpoint failed on ...	The metrics port is taken. The relay keeps running without metrics.

Metrics

curl -s http://127.0.0.1:9090/metrics on the relay host lists the counters. See Metrics.