NCO is currently experiencing major supercomputer issues that are causing failures of some of our models. NCO is working swiftly to resolve these issues but we expect a delay to some of our model output on our web services.
[i]NCO is taking action to switch production to our backup supercomputer. This is an emergency switch and will mean major impacts to possibly all 12Z model output arrival times on our web services, including the GFS.
This may also impact other cycles of the hourly models. We plan to send another update with more information as we know it.[/i]
I don’t know if it is related, but WXSIM is locking up for me and not able to create any forecasts.
Whatever the problem is my wxsimate is crashing on every run and no GFS data appears in the output, although no error trying to obtain it. Switching to Bohler and now it works, although yesterday Bohler data was causing it to crash as well and the latest data in wxsimlite was from midday and this was at 10pm!
Stuart
The web server problem I’m trying to resolve will cause problems with picking GFS data up because that’s done by HTTP. The server seems to have been stable overnight although I’m also restarting it from time to time to apply changed settings as my investigations proceed.
That’s what I thought Chris. I’ll stick with Bohler data for a while until things settle down here.
Stuart
Apache web server seems more stable now and I’ve found a few things that could be affecting performance and put fixes in place. I’ve also been slowly tweaking the web server performance related settings to get the best out of it and that seems to be going in the right direction. Hopefully the GFS issue that was reported has worked its way through the system now so my data should be good and accessible again.