After thorough investigation, we determined that we hit some unforeseen I/O limits with our database. The combination of a cache flush due to a restart, recalculation of some statistics, and peak load resulted in brownouts and occasional errors in our application.
To remediate this type of resource exhaustion, we've purchased additional I/O capacity from AWS. As always, we'll continue to monitor and tune our database configuration in response to the large amounts of data that we process.
There was no impact to customer data during this period. The application is accessible and operating as expected at this time.
Posted 29 days ago. Apr 24, 2018 - 09:13 PDT
The LiquidPlanner application is working as expected following the update. We are continuing to monitor the systems closely.
Posted 30 days ago. Apr 23, 2018 - 19:04 PDT
6pm - 7pm Pacific Daylight time, LiquidPlanner is undergoing maintenance to improve performance. The application is available, but you may experience delays in scheduling and page loads while changes roll out across the system.
Posted 30 days ago. Apr 23, 2018 - 17:57 PDT
The LiquidPlanner application is accessible. Some users may find it slow to reschedule or load pages. We'll address these issues in a release going out at 6pm Pacific Daylight time today, April 23, 2018. Thank you for your continued patience.
Posted 30 days ago. Apr 23, 2018 - 15:16 PDT
To minimize further disruption, we moved our planned update outside of US business hours. Changes will release at 6pm Pacific Daylight time. We apologize for the performance issues today and thank you for your patience while we determine the best course of action to rectify the situation.
Posted 30 days ago. Apr 23, 2018 - 13:33 PDT
At 2pm Pacific Daylight time our team will release an update to address performance issues. There may be a degradation in service while changes roll out until the system stabilizes.
We've identified the source of the performance issues, and will make changes later this afternoon (Pacific time) to correct the problem. There may be intermittent performance issues in the meantime (slow page loads and scheduling). We're currently analyzing site traffic history to determine the optimum time to make updates. We'll post again prior to updating to make you aware of the time we expect to make changes. The app will be accessible during the maintenance period.
Engineers have made changes to reduce load, and application performance is back to normal levels. We're still monitoring the situation, and will provide a more thorough update soon. We apologize for the inconvenience and appreciate your patience.