Simply Testable Updates #108: Job Performance++, System Stability++
September 24, 2014
|This is the online archive for the Simply Testable weekly behind-the-scenes newsletter.
Subscribe to get weekly updates on the latest changes and the newest planned features.
This is the 108th of weekly progress updates on the development of Simply Testable, your professional automated web frontend testing service providing one-click testing for your entire site.
This week I've been deploying changes to our background task processing system and have realised some performance improvements as a result.
I mentioned in last week's newsletter how I was switching over to a new background task processing system. Well, more specifically, I was moving to a different implementation of the background task processing system and moving to a different way of keeping the system working.
Changes to the test workers were deployed last Thursday. I tweaked the configuration a bit both to fully understand what the options did and to observe the benefits, or drawbacks, of configuration changes.
It turns out there were indeed benefits. I'll start out with a bit of background so as to explain the benefits.
We have a collection of worker applications. Each worker instance has a background task queue and each task queue has a collection of processes. Each process takes an item from the queue and performs a testing task.
Number of maximum concurrent tests = number of worker applications * number of processes per task queue
The previous background task processing system implementation could handle only 2 processes per task queue. I could configure the system to start up any number of processes per task queue but the most that it would actually start up was 2 per task queue. No idea why it did this but it did.
With the number of processes per task queue stuck at 2, I deployed 5 worker application instances to support the running of 10 concurrent test tasks. Not a huge amount but it worked at the time and seemed better than nothing.
The new background task processsing system implementation handles any number of processes per task queue.
There are currently 5 processes per task queue and with 5 worker applications this allows for the running of 25 concurrent test tasks. This is much better than before.
I'm currently only seeing up to 25 concurrently-running test tasks when there are at least 3 test jobs running due to other limitations that I am planning to address. Once the other limitations have been addressed, the maximum number of concurrently-running test tasks will be limited only by the hardware. I'll have fun seeing how far that can be stretched.
Under the previous background task processing implementation, the whole system could, and sometimes did, grind to a halt. A messy halt.
The previous background queue manager didn't handle some failures particularly well.
If the process spawned by a background queue process crashed, the background queue manager would run around in circles, throw its hands in the air, and scream like a madman. Or, more technically, the background queue manager would unendingly dump to a log file some intelligible but useless information.
In such cases I knew that something was up but couldn't tell what. The relevant log files would grow by tens of megabytes per second which would put a bit of a strain on the IO system. I had little option than to restart the background queue manager and clear out the abundant log files. What I couldn't do in such cases was perform any sort of investigation that would give me a hint as to what was going wrong.
Thankfully the new background queue manager doesn't behave in the same way. In the above failure case, the background queue manager would dump to a log file a large amount of useful information. It will also rotate log files that increase beyond a certain size. This gave me the information I needed to resolve what was causing some spawned processes to crash. Yay!
Next Monday sees the start of ads airing on podcasts. I'll be keeping a close eye on things when that happens. I'll also be finishing to move to the new background task processing system.
As always, if you'd like to see web testing you find boring handled automatically for you, add a suggestion or vote up those that interest you. This really helps.
Feedback, thoughts or ideas: email email@example.com, follow @simplytestable or keep an eye on the Simply Testable blog.