Hytale Server Without a Panel: Back to Plain systemd
My game panel died, so my Hytale server runs as a plain systemd service again. Unit file, FIFO console helper, backups and a log-tailing alert bot.
A few months ago I wrote about moving my Hytale server into Docker behind a web panel. Later I switched to another panel, one with a separate daemon on the game box and the web side on a different machine. This week the machine with the web side went away. The daemon on the game box was still running, but it didn't know about any servers anymore. No start button, no console, nothing.
The game server itself was fine. It's a Java process with a world folder. So I took the panel out of the picture and put the server back under plain systemd. It took less time than any of the panel setups did.
TL;DR: One systemd unit, one small start script that opens a FIFO as the server's stdin, one hytale-cmd helper that writes into that FIFO. Backups run from cron, the alert bot tails the server's own log files. For a single server with one or two admins that's all you need, and every part of it is something you can read in a minute.
The unit file
# /etc/systemd/system/hytale.service
[Unit]
Description=Hytale server
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=600
StartLimitBurst=5
[Service]
Type=simple
User=hytale
Group=hytale
RuntimeDirectory=hytale
RuntimeDirectoryMode=0750
ExecStart=/usr/local/bin/hytale-start.sh
ExecStop=/bin/sh -c '/usr/local/bin/hytale-cmd stop; while kill -0 $MAINPID 2>/dev/null; do sleep 1; done'
TimeoutStopSec=120
SuccessExitStatus=143
Restart=on-failure
RestartSec=15
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now hytale
journalctl -u hytale -f
What the lines do:
RuntimeDirectory=hytalecreates/run/hytaleowned by the service user when the service starts and removes it when it stops. The console FIFO lives in there, so it only exists while the server runs. That matters later.ExecStopsendsstopto the server console and then waits until the Java process is actually gone. IfExecStoponly sent the command and returned, systemd would send SIGTERM right after, and the server wouldn't get time to save.TimeoutStopSec=120is the safety net. If the server hangs on shutdown, systemd gives up waiting and kills it. I've had that: once, after removing a world withworld remove, the server hung after the plugin shutdown onstopand onlykill -9got rid of it. With the timeout, systemd does that for me.SuccessExitStatus=143because a JVM that gets SIGTERM exits with 143, and that shouldn't count as a failure.Restart=on-failurerestarts after a crash, but not after a cleanstopyou typed in the console on purpose.StartLimitBurststops a boot loop from running forever. More on that in the gotcha below.
If unit files are new to you, I went through all the sections in systemd service unit files.
The start script with the FIFO
The console problem is the same one I had with Docker. Hytale wants an interactive console for things like auth login device, op add or stop. Under systemd, stdin is /dev/null by default, so you can't type anything. The fix is a named pipe as stdin:
#!/bin/bash
# /usr/local/bin/hytale-start.sh
set -euo pipefail
SERVER_DIR=/srv/hytale
FIFO=/run/hytale/console
cd "$SERVER_DIR"
[ -p "$FIFO" ] || mkfifo -m 0660 "$FIFO"
# <> opens the FIFO read-write. The server holds a writer itself,
# so it never sees EOF when a command writer closes the pipe.
exec java -Xms2G -Xmx5G -jar Server/HytaleServer.jar \
--assets Assets.zip \
--allow-op \
--accept-early-plugins \
--bind 0.0.0.0:5520 \
<>"$FIFO"
The important part is <>. If you redirect a FIFO with a plain <, two things go wrong. The open blocks until someone writes to the pipe, so the server doesn't even start. And as soon as the first writer closes, the reader gets EOF, and a lot of console readers treat EOF on stdin as "console closed". Opening it read-write means there's always a writer (the server itself), so it never blocks and never hits EOF.
exec replaces the shell with Java, so the PID systemd watches is the actual server. $MAINPID in the unit points at the right process, and signals go straight to the JVM.
hytale-cmd
#!/bin/sh
# /usr/local/bin/hytale-cmd
FIFO=/run/hytale/console
if [ ! -p "$FIFO" ]; then
echo "hytale is not running (no $FIFO)" >&2
exit 1
fi
printf '%s\n' "$*" | timeout 5 tee "$FIFO" >/dev/null
hytale-cmd say Restart in 5 minutes
hytale-cmd op add alice
hytale-cmd auth login device
The output of the command shows up in the journal and in the server log, so I keep a second terminal with journalctl -u hytale -f open. Input and output being separate sounds annoying, but it's actually what you want. You can script the input and nobody hits Ctrl+C in an attached console and kills the server.
The timeout is there because writing into a FIFO with no reader blocks forever. Because of RuntimeDirectory=, the FIFO is gone when the service is stopped, so the -p check usually catches it. The timeout covers the window where the service is starting or stopping.
Root can write into the pipe anyway. If other admins should be able to use it without sudo, put them in the hytale group. The pipe is 0660 and the directory 0750, so group members can reach it.
While the panel daemon was still running the server, my console trick was writing straight into the stdin of the Java process through /proc/<pid>/fd/0. That works when stdin is a pipe, but you depend on whatever the daemon set up. Having my own FIFO is cleaner.
Gotcha: the assets path
My first start went into a boot loop. The server complained about missing cosmetics files, exited, systemd restarted it, same thing again. The reason: without --assets, the server looks for the assets in a default path one directory up, not for the Assets.zip next to it. The panel had always passed the path for me, so I never noticed. --assets Assets.zip fixed it.
That's also why StartLimitBurst is in the unit. Without it, a broken start just loops every 15 seconds until you notice. With it, systemd gives up after 5 tries in 10 minutes and the unit shows up as failed in systemctl --failed.
Gotcha: the old daemon still owns the port
The panel daemon was a Docker container that listened on the game port range. Before starting the server under systemd I stopped that container and set its restart policy to no, so it doesn't come back after the next reboot and grab the port before my service does:
docker stop paneldaemon
docker update --restart=no paneldaemon
I didn't delete it. If I ever want to go back, it's still there. Same idea as keeping the old systemd unit disabled when I moved to the panel.
Auth survived
I expected to redo the device login. I didn't have to. The server reads its saved credentials from auth.enc in its working directory, and since the server stayed on the same box with the same working directory, it picked them up on the first start. If you move the server to another machine, plan for hytale-cmd auth login device and a browser.
Backups
Backups don't need the panel either. A cron job at 04:00 UTC runs a script that packs two archives with tar and zstd: one with the worlds, prefabs and configs (about 1.3 GB right now), one with everything else except the assets (about 4.6 GB). It checks the SHA256 after upload, pushes both to a storage box over SSH, and a second script deletes anything older than 30 days.
# /etc/cron.d/hytale-backup
0 4 * * * root (/root/backup-to-box.sh && /root/backup-rotate.sh) >> /var/log/hytale-backup.log 2>&1
It runs while the server is live. That's a trade-off. A copy of a world that is being written can be inconsistent, and the clean way is stop, archive, start, like I described in Hytale backups and auto-restart with systemd. With the FIFO you can also send a save command first, if your build has one. For now I take the live backups plus 30 days of history. Whatever you pick, unpack one of those archives on a test box now and then, a backup you never restored is just a hope.
The alert bot
My Discord alert bot was built for Docker. It ran docker logs -f and docker events to see crashes. When I moved to the panel I gave it a file mode: point it at the server's log directory, it tails the newest server log file, switches to the new one when the server restarts, and checks for the Java process via /proc instead of asking Docker.
That paid off this week. The server writes its own log files into logs/ no matter who starts it, so after the switch to systemd the bot just kept working. I didn't change anything. The rules are the same as in Discord alerts from server logs: only fatal patterns alert, kicks with odd reasons post without a ping, and a UDP probe catches freezes where the process is up but nobody can connect.
Why fewer parts wins here
A panel is a permissions tool. It's worth it when several people need start, stop and console and you don't want to hand out SSH. I said that in the Docker post and I still think so.
But look at what it cost me. The panel's web side lived on another machine. When that machine went away, the daemon was still running and knew nothing. The server itself never had a problem.
The systemd setup has four pieces: a unit, a start script, a 10-line helper and a cron file. All of them are plain text on the box itself. If something breaks, systemctl status hytale and the journal tell me why.
For one server on one box, that's the setup I keep. If more people need access again, I'd rather give them a restricted SSH user that can only run systemctl restart hytale and hytale-cmd than another daemon with access to the Docker socket.