We would like to apologize for the recent incident that affected our Applicant Tracking System (ATS) services, particularly impacting our mailing services. We are committed to maintaining high standards of reliability and transparency, and it is important to us to provide a clear understanding of the events that occurred, the actions taken to resolve the incident, and our plans to prevent such issues in the future. 

Summary of the Incident:

On the morning of May 1st, an issue in our job queue management resulted in a delay in sending out messages and notifications. This was caused by a new update implemented to optimize our billing process, which inadvertently prioritized new job queues over some existing legacy queues. Consequently, from 3 AM until approximately 9:10 AM PT, certain jobs, including mailings, were delayed.

Technical Overview:

The problem began when our billing jobs, which were intended to be processed faster due to recent changes made on April 29th, overwhelmed the system. This resulted in lower-priority legacy job queues, including those responsible for mailing services, being neglected. Once the issue was detected, our engineering team swiftly responded by reallocating resources and adjusting the prioritization of job queues to mitigate the delay.

Steps Taken:

1. Immediate scaling up of utility nodes to enhance processing capacity.
2. Dedicated utility nodes to handle both fast and legacy queues more effectively.
3. Added additional database readers to manage the increased load and prevent future bottlenecks.
   

Ongoing and Future Remediation Plans:

To prevent a recurrence of this issue, we have implemented several strategic changes:

• Preemptive Resource Scaling: We have scheduled an automatic scaling of utility nodes from the 25th of each month to the 2nd of the following month to accommodate increased job loads, especially during billing periods.
• Moratorium on Changes: A policy has been established to halt changes to background job configurations after the 26th of each month, ensuring stability during critical processing times.
• Monitoring and Adjustment: Continuous monitoring and dynamic adjustments will be made to our job queue system to maintain efficiency and responsiveness.

We understand the critical role our services play in your daily operations, and we deeply regret any disruptions this incident may have caused. Our team is dedicated to ensuring the reliability of our system and preventing such issues from occurring in the future.

Thank you for your understanding and continued support. Should you have any questions or require further information, please do not hesitate to contact the Support Team.