Issue Summary: Following a recent backend infrastructure change, some hosted clients experienced increased latency and degraded performance.
What Happened: As part of our ongoing infrastructure improvements, we migrated our hosted servers to a new storage system. This change was designed to be seamless with no expected client impact. However, shortly after the migration, we began receiving reports of performance issues.
Root Cause: Our investigation revealed that a duplicate IP address had been unintentionally included in the configuration during the migration. This caused unexpected increase in load on the affected server, leading to performance degradation for some hosted clients.
Resolution: The duplicate IP entry was identified and removed from the configuration. After this change, system performance returned to normal levels.
Next Steps: We are reviewing our change validation processes and implementing additional safeguards to help prevent similar issues in the future.
We sincerely apologize for any disruption this may have caused and appreciate your continued trust.
Posted Jun 30, 2025 - 17:18 EDT
Monitoring
Our IT team believes they have identified the source of the latency affecting some hosted clients and has implemented changes to address the issue. We are now actively monitoring performance to confirm that the changes have resolved the problem. We’ll provide another update once monitoring is complete or if anything changes.
Thank you for your patience.
Posted Jun 30, 2025 - 13:41 EDT
Update
We are still investigating the issue. Our team made some changes Friday evening to try and alleviate the latency issues. Because of the lower volume of users over the weekend, the issue seemed to have been resolved until this morning when we have our normal volume of users logged in.
Posted Jun 30, 2025 - 11:40 EDT
Investigating
We are currently investigating database performance issues affecting a subset of our hosted clients. The errors appear to be related to increased latency when accessing the database.
Posted Jun 27, 2025 - 11:04 EDT
This incident affected: Occupational Health/Employee Health.