After thorough investigation, we determined that we hit some unforeseen I/O limits with our database. The combination of a cache flush due to a restart, recalculation of some statistics, and peak load resulted in brownouts and occasional errors in our application.
To remediate this type of resource exhaustion, we've purchased additional I/O capacity from AWS. As always, we'll continue to monitor and tune our database configuration in response to the large amounts of data that we process.
There was no impact to customer data during this period. The application is accessible and operating as expected at this time.
Posted Apr 24, 2018 - 09:13 PDT
Monitoring
The LiquidPlanner application is working as expected following the update. We are continuing to monitor the systems closely.
Posted Apr 23, 2018 - 19:04 PDT
Update
6pm - 7pm Pacific Daylight time, LiquidPlanner is undergoing maintenance to improve performance. The application is available, but you may experience delays in scheduling and page loads while changes roll out across the system.
Posted Apr 23, 2018 - 17:57 PDT
Update
The LiquidPlanner application is accessible. Some users may find it slow to reschedule or load pages. We'll address these issues in a release going out at 6pm Pacific Daylight time today, April 23, 2018. Thank you for your continued patience.
Posted Apr 23, 2018 - 15:16 PDT
Update
To minimize further disruption, we moved our planned update outside of US business hours. Changes will release at 6pm Pacific Daylight time. We apologize for the performance issues today and thank you for your patience while we determine the best course of action to rectify the situation.
Posted Apr 23, 2018 - 13:33 PDT
Update
At 2pm Pacific Daylight time our team will release an update to address performance issues. There may be a degradation in service while changes roll out until the system stabilizes.
Posted Apr 23, 2018 - 12:25 PDT
Update
We've identified the source of the performance issues, and will make changes later this afternoon (Pacific time) to correct the problem. There may be intermittent performance issues in the meantime (slow page loads and scheduling). We're currently analyzing site traffic history to determine the optimum time to make updates. We'll post again prior to updating to make you aware of the time we expect to make changes. The app will be accessible during the maintenance period.
Posted Apr 23, 2018 - 11:35 PDT
Update
We continue to see improved performance although some monitors still show slower response times than normal. All areas of the application are accessible. We will update this page until resolved.
Posted Apr 23, 2018 - 10:13 PDT
Update
Engineers have made changes to reduce load, and application performance is back to normal levels. We're still monitoring the situation, and will provide a more thorough update soon. We apologize for the inconvenience and appreciate your patience.
Posted Apr 23, 2018 - 09:04 PDT
Identified
Some customers are seeing a message that the website is under heavy load. We're tracking down the cause.
Posted Apr 23, 2018 - 08:16 PDT
Update
We are experience slow load times. The app is accessible and performance is improving. Our team is monitoring continuously and working to resolve.
Posted Apr 23, 2018 - 07:22 PDT
Investigating
We are experiencing a service disruption. Our team was notified and they are investigating the issue. This page will be updated again as soon as we have more information about impact and scope.