On HCMatrix, employees’ clock-ins and clock-outs were being recorded against the wrong times. For an HR platform that feeds attendance data into real decisions, that’s a serious bug.
The natural place to look was the attendance logic. The cause wasn’t there.
Finding it
A single user action on HCMatrix can touch several of its eleven services, and there were three of us on the backend. An issue you can’t follow across services is one you can’t realistically find, so structured logging with request correlation was the first step. It gave us a way to trace what happened instead of guessing.
The cause was the server’s own time settings, interacting with the cron job that drove the scheduled attendance work. Change the server’s timezone below and watch the record move.
Wrong. The record inherited whatever the server was set to.
The fix
Correcting the server’s settings would only move the problem to the next server or deployment. So I stopped depending on server local time at all.
Timestamps are stored in UTC and converted at the boundary, so the cron job and the attendance records no longer inherit whatever the host happens to be set to.
The lesson
The code that looks wrong isn’t always where the bug is. Here the attendance logic was fine, and the problem was relying on whatever time the server happened to be set to.